Title: Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training
Venue: NeurIPS 2025
Accepted: yes
Keywords: AI4PDE

Neural Operator

Pre-Training
Abstract: Pre-training has proven effective in addressing data scarcity and performance limitations in solving PDE problems with neural operators.
However, challenges remain due to the heterogeneity of PDE datasets in equation types, which leads to high errors in mixed training. 
Additionally, dense pre-training models that scale parameters by increasing network width or depth incur significant inference costs.
To tackle these challenges, we propose a novel **M**ixture-**o**f-**E**xperts **P**re-training **O**perator **T**ransformer (**MoE-POT**), a sparse-activated architecture that scales parameters efficiently while controlling inference costs.
Specifically, our model adopts a layer-wise router-gating network to dynamically select 4 routed experts from 16 expert networks during inference, enabling the model to focus on equation-specific features. 
Meanwhile, we also integrate 2 shared experts, aiming to capture common properties of PDE and reduce redundancy among routed experts. 
The final output is computed as the weighted average of the results from all activated experts.
We pre-train models with parameters from 30M to 0.5B on 6 public PDE datasets.
Our model with 90M activated parameters achieves up to a 40\% reduction in zero-shot error compared with existing models with 120M activated parameters.
Additionally, we conduct interpretability analysis, showing that dataset types can be inferred from router-gating network decisions, which validates the rationality and effectiveness of the MoE architecture.

Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training

Abstract

Pre-training has proven effective in addressing data scarcity and performance limitations in solving PDE problems with neural operators. However, challenges remain due to the heterogeneity of PDE datasets in equation types, which leads to high errors in mixed training. Additionally, dense pre-training models that scale parameters by increasing network width or depth incur significant inference costs. To tackle these challenges, we propose a novel M ixtureo fE xperts P re-training O perator T ransformer ( MoE-POT ), a sparse-activated architecture that scales parameters efficiently while controlling inference costs. Specifically, our model adopts a layer-wise router-gating network to dynamically select 4 routed experts from 16 expert networks during inference, enabling the model to focus on equationspecific features. Meanwhile, we also integrate 2 shared experts, aiming to capture common properties of PDE and reduce redundancy among routed experts. The final output is computed as the weighted average of the results from all activated experts. We pre-train models with parameters from 30M to 0.5B on 6 public PDE datasets. Our model with 90M activated parameters achieves up to a 40% reduction in zero-shot error compared with existing models with 120M activated parameters. Additionally, we conduct interpretability analysis, showing that dataset types can be inferred from router-gating network decisions, which validates the rationality and effectiveness of the MoE architecture 1 .

1 Introduction

Learning solution operators for partial differential equations (PDEs) has emerged as a fundamental paradigm in scientific machine learning, enabling data-driven modeling of complex physical systems through neural operators [62, 22, 28, 12]. These operators learn mappings between infinitedimensional function spaces, offering surrogate models that can outperform traditional numerical solvers by orders of magnitude in speed [45, 2]. To address the scarcity of PDE data and further enhance the performance of neural operators, recent studies have introduced pre-training techniques into neural operator frameworks [15]. However, their application to PDE learning remains nascent due to unique challenges in operator learning.

First, ( challenge 1 ) PDE datasets demonstrate substantial variations across equation types, boundary conditions, and spatiotemporal resolutions. This diversity causes conflicting knowledge patterns when







Multiple PDE Datasets

Pre-Training

Fine-Tuning

Router

Expert 2

Expert 1

Expert 3

Expert 4

Neural Network























×



































×















merging different PDE types during training-direct data mixing frequently results in detrimental interference that restricts knowledge acquisition, rather than enhancing model generalization.

Second, ( challenge 2 ) existing approaches to scaling model capacity through dense architectural expansion (increasing width/depth) incur prohibitive inference costs, making pre-training impractical for real-world applications.

To address these challenges, we propose the M ixtureo fE xperts P re-training O perator T ransformer ( MoE-POT ), a novel sparse architecture. Our key insight is to decouple capacity expansion from computational cost through dynamic expert activation.

Specifically, MoE-POT employs a learnable router-gating network that automatically selects 4 routed experts based on each layer's input data, working with 2 fixed shared experts. The outputs from these 6 experts are then weighted and aggregated to produce the final result. The router-gating network dynamically selects routed experts that specialize in learning distinctive features of the current PDE category, effectively isolating interference from significantly different PDE data types. Meanwhile, the shared experts act as fixed computational modules for all data. They ensure consistent learning of fundamental dynamic evolution laws.

The key contributions and advantages of MoE-POT are summarized as follows:

Novel Architecture : Weintroduce MoE-POT, a novel sparse architecture for neural operator pre-training. MoE-POT employs two types of expert networks: routed experts and shared experts, enabling a balance between generalization and specialization.

Empirical Validation : As shown in Figure 1 (right), we pre-train models on 6 public PDE datasets and design multiple versions of MoE-POT with total parameter scales ranging from 30M to 0.5B. Our model with 90M activated parameters achieves up to a 40% reduction in zero-shot error compared to existing models with 120M activated parameters.

Interpretable : We observe that the trained router-gating network can infer the PDE type of input data with 98% accuracy, showing MoE-POT's ability to effectively handle diverse PDE datasets. This result validates both the rationale and effectiveness of the MoE architecture.

















































































 

































 































 



































 

















2 Preliminaries

2.1 Time-dependent PDE Problem

We consider a general form of parameterized time-dependent PDEs characterized by variables u ( x, t ) ∈ R m . These equations satisfy the following conditions:

Here, F [ u ; θ ]( x, t ) = F ( t, x, u , ∂ x u , ∂ xx u , . . . ; θ ) represents a differential operator involving spatial derivatives, while θ ∈ Θ denotes unknown parameters that define the type and coefficients of the PDE. The initial condition is given by u 0 ( x ) , and B [ u ]( x, t ) specifies the boundary conditions. This general formulation encompasses a variety of fundamental PDEs.

In practical scenarios, datasets are often collected from multiple PDEs, represented as D = ∪ K k =1 D k , where D k = { u i } 1 ≤ i ≤ N k . Each solution function u i ∈ D is discretized on spatiotemporal meshes, expressed as u i = ( u 1 i , . . . , u T i ) , with u t i = { ( x j , u t j ) : x j ∈ X i } for 1 ≤ t ≤ T . The spatial meshes X i may consist of regular grids or irregular point clouds, depending on the geometry of the domain. The parameters θ govern the type and specific characteristics of the PDE. However, in many real-world applications, such as climate modeling, only observational trajectories of data are available, while the detailed parameters θ remain inaccessible. To predict future timesteps, it is essential to infer the most likely θ implicitly from the observed sequence of T frames ( u 1 i , . . . , u T i ) .

2.2 Auto-regressive Denoising Pre-training

To effectively learn from temporal PDE datasets, we propose a neural operator G w ( u t<T ) , parameterized by weights w , which auto-regressively takes T frames as input and predicts the next frame based on the previous frames:

By predicting the next frame, the model learns an internal representation of the underlying PDE dynamics. However, directly supervising the one-step loss has been shown to be suboptimal [3]. Following DPOT [15], we inject small-scale noise into the input frames. For ∀ t ≤ T , let u <t denote ( u 0 , . . . , u t -1 ) , and the noise ε is sampled as ε ∼ N (0 , ϵ || u <t || I ) . Noise injection improves robustness and reduces the discrepancy between training and inference.

We adopt the experimental setup of DPOT [15] and FNO [28], which focuses on the challenging scenario where models must infer system dynamics solely from solution trajectories, without access to the governing PDE parameters. In contrast to parameter-informed approaches, our core contribution is the integration of a MoE architecture into this auto-regressive paradigm.

3 Motivation

3.1 Challenges in Dense Neural Operator Pre-training

Current dense neural operator pre-training approaches face two challenges, which severely limit the model's ability to generalize across PDE tasks and hinder performance improvement.

(C1) Performance degradation due to dataset heterogeneity : As shown in Figure 2 (left), our preliminary experiments reveal that the heterogeneity of PDE datasets significantly impacts pretraining effectiveness. For instance, when training on different parameter configurations within the same equation family (e.g., fluid simulation data with varying Reynolds numbers), the average test error increases by only 80% compared to training on individual datasets. However, when mixing three entirely different equation types for training, the error increases by up to 5000%.

The properties of different PDE types exhibit substantial variations, yet dense models enforce parameter sharing across all inputs, making it difficult to efficiently absorb diverse PDE knowledge within a unified architecture. For example, when simultaneously learning PDEBench-SWE and PDEBench-DR, the model must encode two fundamentally different differential operators within the same parameter space, leading to negative transfer or inter-task interference [6].

(C2) Scaling bottlenecks in model capacity and performance : To better capture the diverse properties of PDEs, increasing model parameters is often necessary to enhance the expressive power of neural operators. However, this approach introduces significant computational cost. As shown in Figure 1 (right), experiments demonstrate that models follow a diminishing returns scaling law: improving model performance requires a substantial increase in parameters. Since all parameters are activated during inference, dense models generate high inference costs, further exacerbating computational challenges.

3.2 Sparse Pre-training with Mixture-of-Experts

To address these challenges, we propose a PDE pre-training model based on the MoE architecture.

Efficient scaling with sparse activation : As illustrated in Figure 3, the MoE architecture decomposes the fully connected computation of traditional dense layers into a collaborative mechanism involving parallel expert networks and gated routing. Each Transformer layer consists of 16 routed experts and 2 shared experts. For a given input sample, the router-gating network activates only 4 routed experts, resulting in an actual computational cost equivalent to only 33% of the total parameters. This mechanism effectively scales model capacity while controlling inference cost ( C2 ).

Physics-driven gated routing : The dynamic selection capability of the router-gating network provides a natural solution for integrating heterogeneous PDE knowledge ( C1 ). 1. Shared experts: Through cross-task learning, the 2 shared experts are constrained to capture universal physical principles (e.g., conservation laws, symmetry). 2. Routed experts: The remaining 16 experts autonomously develop distinct functional roles to learn the unique characteristics of different PDEs.

As shown in Figure 2 (right), after training, the router-gating network decisions vary significantly across different PDE datasets. For example, NS(1e-5) and NS(1e-3), which are closely related datasets, exhibit similar gating patterns. In contrast, PDEBench-SWE and PDEBench-DR, which differ substantially, show distinct gating behaviors. More importantly, our interpretability analysis (see Experiment 5.4) reveals that gating weights can serve as identifiers for PDE types. The expert selection for a given input can be used to determine its dataset type with an accuracy of 98%. This demonstrates that the MoE architecture not only improves performance but also enables implicit equation type recognition for neural operators, paving the way for building interpretable foundational models for PDEs.

4 Method

Overview . Our proposed model architecture is illustrated in Figure 3. It begins by processing raw data through a patchification layer and a temporal aggregation layer [15], which reduces spatial-temporal resolution and extracts dynamic structures inherent to PDEs. The processed features are then passed through N blocks, each of which contains a Fourier layer [13] and a MoE layer, thereby achieving efficient representation and specialization for diverse PDE tasks.

z

Sampling

Padding

Noise

Injection

Pre-trained

PDE Model

Patch/Positional Embedding

Temporal Aggregation Layer

Output Projection Layer

Weighted Temporal MLP

(                 )

Block

Mixture of Experts Layer

Router

1

Output Layer

2

1

···

···

···

Routed Expert

Shared Expert

Input Encoding and Temporal Aggregation . The input u <T ∈ R H × W × T × C represents a spatiotemporal signal with C channels. To encode spatial features, we apply a patchification layer with positional embeddings inspired by vision transformers [10]:

where P is a convolutional layer, and p t i,j = W p ( x i , y j , t ) denotes learnable positional encodings. The output Z t p ∈ R H/p × W/p × C captures spatial features, with W p ∈ R n × 3 , where n is the feature dimension of the positional encoding (e.g. n = C ). To capture temporal dynamics, we employ a temporal aggregation layer that extracts information across adjacent time steps. For each local node feature z t p ∈ R C in Z t p , we apply a learnable MLP transformation W t combined with Fourier feature constant γ ∈ R C :

This aggregation enables the model to implicitly infer the underlying PDE governing parameters.

Fourier Layer . Using a multi-head architecture, the Fourier layer is designed to learn complex kernel-based integral transformations that approximate PDE solutions [13, 15]. Let z l ( x ) denote the feature at spatial location x in the l -th block, and Z l its discretized representation. We apply a kernel integral operator K ϕ parameterized by a neural network:

where κ ( x, y ; ϕ ) is a learnable kernel function. To reduce computational complexity, we constrain the kernel to be translation-invariant: κ ( x, y ; ϕ ) = κ ( x -y ; ϕ ) . This reformulation allows efficient implementation in the Fourier domain:

Here, z l ( x ) ∈ R d z , R ϕ ( k ) ∈ C d z × d z is a frequency-dependent learnable transformation, and F / F -1 denote the Fourier transform and its inverse. To ensure memory efficiency and jointly attend to information from different representation subspaces, we first divide spatial features z l ( x ) into h groups. The grouping is performed on the channel dimension, where h is the number of heads, i.e., l l l l l dz l

h

, . . . z

)

, where

z

i

(

k

)

∈

R

h

. we approximate

(

K

ϕ

z

)(

x

)

using

h

smaller MLPs:

= Concat(

z

1

, z

2

where W l 1 ,i , W l 2 ,i ∈ R d z /h × d z /h and b l 1 ,i , b l 2 ,i ∈ R d z /h are learnable parameters, and σ ( · ) is an activation function. We set z l 0 = Concat( z l 01 , z l 02 , . . . z l 0 h ) , that passed to the MoE layer.

Mixture of Experts Layer . To facilitate sparse activation and enable expert specialization, we integrate a MoE layer specifically designed for PDE inputs. Both expert networks and routergating networks are implemented using convolutional neural networks (CNNs) to preserve spatial information. For the reason behind the structural design, please see Appendix B.2. These features are then passed to a router-gating network G l ( z l 0 ( x )) , which computes a vector of routing logits s l ( z l 0 ( x )) ∈ R N r , where N r is the number of routed experts (e.g., N r = 16 ). The gating weights are computed via a softmax function:

To maintain sparsity, only the TopK entries in w l ( z l 0 ( x )) are retained (e.g., K = 4 ), and the rest are masked to zero:

where i k is the index of the k -th selected routed expert and w l k ( x ) is the normalized routing weight.

Let E l s = { E l ( s ) 1 , . . . , E l ( s ) N s } denote the set of shared experts, which are always activated for every input. Let E l r = { E l ( r ) 1 , . . . , E l ( r ) N r } denote the set of routed experts, from which the topK are selected dynamically per input. Each expert E l ( s ) i or E l ( r ) j is a convolutional subnetwork that takes z l 0 ( x ) as input and maps it to an output feature map of the same shape. Specifically, the final output of the MoE layer is computed as [8, 51]:

Load Balancing Objective. To encourage uniform utilization of all routed experts and avoid routing collapse [51], we introduce a load balancing loss during the training phase. Following prior work [11, 8], we define the importance of each expert over a batch B ( |B| = B ) as the sum of its routing weights:

We compute the coefficient of variation (CV) across all N r routed experts, and define the loss as:

where w bal is a tunable scaling factor (e.g., w bal = 0 . 1 ). This auxiliary loss regularizes the routing distribution to maintain a balanced expert load and improves overall training stability.

Loss Function. The primary objective of the model is to predict the one-step transition between samples from different datasets. The loss function is defined as:

where G w represents the model's prediction function, u <t denotes the input from previous timesteps, and ε is a perturbation term. By predicting the next timestep data from previous frames, the model learns to implicitly infer the PDE's governing dynamics and propagate the solution forward in time.

5 Experiments

We conducted comprehensive experiments to evaluate the performance of MoE-POT. This section is organized as follows: 1. Comparison with various small and pre-trained models on 6 PDE datasets.

Testing knowledge transfer capabilities on downstream tasks. 3. Investigating scaling laws to understand performance trends. 4. Interpretable analysis of the router-gating network selection.

Analyzing model inference time 6. Ablation studies to assess the impact of hyperparameters.

Datasets. For pre-training, we utilize 6 datasets sourced from 3 benchmark collections: FNO [28], PDEBench [53], and CFDBench [38]. These datasets encompass a wide range of PDE types and parameters. The mathematical formulations of these PDEs are provided in Appendix B.5. To ensure consistency and compatibility across datasets, we applied preprocessing techniques such as padding and masking. Detailed descriptions of preprocessing steps can be found in Appendix B.1.

Training and Evaluation. The model configurations for different scales are detailed in Appendix B.3. Across all model sizes, we employed the Adam optimizer with a learning rate of 1 × 10 -3 and trained the models for 1000 epochs. Training was conducted on servers equipped with 8 RTX 4090 GPUs, each with 24 GB of memory. We use the l 2 relative error (L2RE) as the primary metric to evaluate prediction quality, following the standard practice outlined in [28].

Baseline. We selected the following influential methods as baselines for comparison, categorized into two groups: 1. Small models : This group includes FNO (along with Geo-FNO for irregular datasets) [28, 26], UNet [50], FFNO [55], GK-Transformer [5], OFormer [27], and GNOT [16]. These models are trained and evaluated individually on each dataset.

2. Pre-trained models : FNO-(T/S/M): Larger-scale variants of the original FNO model, with expanded parameter sizes for comparison; DPOT-(T/S/M): The state-of-the-art pre-trained neural operator model [15]. Both pre-trained baselines and MoE-POT-(T/S/M) are first pre-trained on six datasets and then evaluated for performance.

5.1 Main Experiments

Table 1 summarizes the results of our main experiments. The parameter in the second row corresponds to the PDE dataset configuration, such as 1 e -5 for viscosity in the FNO NS dataset [28]. The activation parameter counts for MoE-POT-(T/S/M) are 17M, 90M, and 188M, respectively, with total parameter counts of 30M, 166M, and 489M.

The second part of the Table 1 evaluates pre-trained models, including FNO variants, DPOT, and MoE-POT. Our model achieves the best zero-shot performance on 5 out of 6 datasets, with significant improvements in L2RE compared to DPOT and FNO-M. For example, on the PDEBench-CNS(0.1, 0.01), Ours-S (with 90M activation parameters) reduces L2RE by 57% compared to DPOT-M (with 122M activation parameters). The FNO architecture, not specifically designed for pre-training, struggles to optimize datasets with highly diverse properties due to its dense network structure. This leads to instability during pre-training, resulting in large errors or training collapse on certain datasets. For instance, FNO-M fails to produce valid results on PDEBench-CNS(0.1, 0.01) and CFDBench due to excessively high L2RE. The performance improvements of our model are primarily attributed to the expert network design within the MoE architecture, which effectively captures intrinsic features across datasets with differing properties without mutual interference. This demonstrates the effectiveness of the MoE architecture in pre-training scenarios for PDEs.

The last part of the Table 1 presents the results of fine-tuning pre-trained models on each subset for 200 epochs. Fine-tuning consistently improves performance across all datasets, with larger models yielding better results. For example, MoE-POT-M achieves the best fine-tuning results on 5 out of 6 datasets, reducing L2RE by over 50% compared to the zero-shot model on PDEBench-DR. Compared to DPOT, our MoE-based model achieves significant performance gains. These improvements stem from the ability of the MoE architecture to substantially expand the total model parameters while keeping activation parameters relatively constant, thereby enhancing model performance without increasing inference costs. Additionally, after fine-tuning, our model outperforms all small models on most datasets, achieving state-of-the-art results on 4 out of 6 datasets. This demonstrates that our model successfully learns from multiple PDE datasets simultaneously through pre-training.

In summary, fine-tuning significantly enhances performance, suggesting that pre-training on largescale PDE datasets is a promising and scalable approach for improving operator learning tasks. The results highlight the advantages of our MoE architecture in handling complex and heterogeneous PDE data. Furthermore, for additional comparative experiments with large-scale models, including DPOT-L and Poseidon, please refer to Appendix C.2.

5.2 Downstream Tasks Experiments

To evaluate the effectiveness of our pre-trained model in enhancing performance across diverse PDE downstream tasks, we conducted experiments to test its broader applicability. We selected three downstream tasks. NS ( 1 e -4 ), closely related to the pre-training datasets NS ( 1 e -3 ), and PDEArena, which represents equations with entirely different mathematical structures. All models were trained or fine-tuned for 500 epochs, and the results are summarized in Table 2.

Firstly, in all experiments, our model trained from scratch ('w/o Pre-train') consistently outperforms smaller models, demonstrating the effectiveness of the MoE-POT architecture for operator learning. Secondly, across all tasks, both DPOT and MoE-POT models show significant performance improvements after pre-training and fine-tuning, far surpassing the performance of small models





















×









































×

































































trained from scratch. This indicates that pre-training enables the model to learn more effective and transferable representations. These results highlight the remarkable versatility of our model, which can be seamlessly extended to a wide range of downstream tasks.

5.3 Scaling Experiments

The relationship between performance and increasing model size is a critical property of pre-trained models. In this study, we conduct scaling experiments to evaluate the scalability of our model. The results are presented in Figure 4 (a). We observe that as the model size increases, the zero-shot test error consistently decreases, approximately following a scaling law. Furthermore, fine-tuning the model on specific datasets leads to improved performance. As shown in Figure 4 (b), while all models exhibit scaling properties, our model demonstrates better performance for a given number of activated parameters. This advantage primarily stems from the MoE architecture, which significantly increases the total number of parameters without proportionally expanding the number of activated parameters. For example, MoE-POT-T with a total of 30M parameters, requires only 57% of the activated parameters (17M) compared to existing models with similar performance. Furthermore, further analytical experiments are presented in Appendices C.3, C.4, and C.5, which include an analysis of error accumulation over rollout steps, a study on the relationship between fine-tuning data size and performance, and an investigation into the impact of pre-training data heterogeneity.

5.4 Interpretable Analysis

We aim to determine which dataset a given data point belongs to by analyzing the expert selection in the MoE router-gating network. For a specific block, we first compute the average expert selection values Y i for each dataset (forming a vector), where i = 1 , . . . , 6 represents the i -th dataset. Then, for the expert selection vector I 0 of input, we calculate its distance to each Y i . The Y i 0 with the smallest distance to it is obtained, indicating that the input belongs to the i 0 -th dataset. Detailed procedures can be found in Appendix B.4.

As shown in Figure 4 (c), the router-gating network in Block 2 achieves an accuracy of 97.7% in classifying the input dataset. Similar results are observed in other blocks. This strongly demonstrates that the MoE architecture effectively learns the differences between PDE datasets and uses this information for classification. Furthermore, further interpretability analysis, including the emergence of classification ability and its generalization to out-of-distribution (OOD) data, is available in Appendix C.6.

5.5 Inference Time Analysis

We evaluated the inference time of various models. As shown in Table 3, under the same total number of parameters, our model demonstrates significantly lower inference time compared to DPOT. For example, MoE-POT-M has a total parameter count of 489M, comparable to DPOT-L, yet its inference time is only 68% of the latter, making it equivalent to DPOT-M with just 158M parameters. This efficiency is primarily attributed to the MoE structure, which activates far fewer parameters than the total parameter count. For certain PDE tasks, a single computation may require 10 3 to 10 5 inference steps. The MoE structure effectively reduces inference time while preserving model performance.

5.6 Ablation Experiments

We conduct ablation studies by training MoE-POT-T on 6 pre-trained datasets and compare the averaged zero-shot performance on the corresponding test datasets. As shown in Table 4, the error remains stable when the number of routed experts N r is sufficiently large. However, increasing N r leads to higher computational costs during training. Thus, we select N r = 16 as a balance between performance and efficiency. Increasing the number of expert selections (TopK ) reduces error, as more experts contribute to inference, thereby increasing the activated parameters. However, this improvement exhibits diminishing returns, so we set TopK = 4 as an optimal trade-off. Additionally, we analyzed the impact of the number of heads h and patch sizes on the model's performance. Please refer to Appendix C.1 for details.

6 Limitations and Conclusions

This paper introduces MoE-POT, a sparse architecture designed for PDE pre-training. By dynamically activating routed experts and leveraging fixed shared experts, MoE-POT achieves state-of-the-art performance under the same activated parameters. Additionally, we observe that the router-gating network can effectively distinguish features of different PDEs and classify data, further illustrating the rationality of the MoE structure.

However, we have yet to analyze the mathematical essence of this classification mechanism. Exploring how PDE classification can guide the construction of more effective pre-training datasets remains an important direction for future work.

References

Yannick Augenstein, Taavi Repan, and Carsten Rockstuhl. Neural operator-based surrogate solver for free-form electromagnetic inverse design. ACS Photonics , 2023.

Kamyar Azzizadenesheli, Nikola Kovachki, Zongyi Li, Miguel Liu-Schiaffini, Jean Kossaifi, and Anima Anandkumar. Neural operators for accelerating scientific simulations and design. arXiv preprint arXiv:2309.15325 , 2023.

Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural pde solvers. arXiv preprint arXiv:2202.03376 , 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Shuhao Cao. Choose a transformer: Fourier or galerkin. Advances in neural information processing systems , 34:24924-24940, 2021.

Rich Caruana. Multitask learning. Machine learning , 28(1):41-75, 1997.

Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics-informed neural networks: Where we are and what's next. Journal of Scientific Computing , 92(3):88, 2022.

Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066 , 2024.

Huanshuo Dong, Hong Wang, Haoyang Liu, Jian Luo, and Jie Wang. Accelerating pde data generation via differential operator action in solution space. arXiv preprint arXiv:2402.05957 , 2024.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research , 23(120):1-39, 2022.

Wenhan Gao, Ruichen Xu, Yuefan Deng, and Yi Liu. Discretization-invariance? on the discretization mismatch errors in neural operators. The Thirteenth International Conference on Learning Representations , 2025.

John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587 , 2021.

Jayesh K Gupta and Johannes Brandstetter. Towards multi-spatiotemporal-scale generalized pde modeling. arXiv preprint arXiv:2209.15616 , 2022.

Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, and Jun Zhu. Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training. In International Conference on Machine Learning , pages 17616-17635. PMLR, 2024.

Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, and Jun Zhu. Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learning , pages 12556-12569. PMLR, 2023.

Liu Haoyang, Wang Jie, Cai Yuyang, Han Xiongwei, Kuang Yufei, and HAO Jianye. Optitree: Hierarchical thoughts generation with tree search for LLM optimization modeling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems , 2025.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

Maximilian Herde, Bogdan Raoni´ c, Tobias Rohner, Roger Käppeli, Roberto Molinaro, Emmanuel de Bézenac, and Siddhartha Mishra. Poseidon: Efficient foundation models for pdes. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems , volume 37, pages 72525-72624. Curran Associates, Inc., 2024.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics , 3(6):422-440, 2021.

Ehsan Kharazmi, Zhongqiang Zhang, and George Em Karniadakis. Variational physics-informed neural networks for solving partial differential equations. arXiv preprint arXiv:1912.00873 , 2019.

Yufei Kuang, Jie Wang, Haoyang Liu, Fangzhou Zhu, Xijun Li, Jia Zeng, HAO Jianye, Bin Li, and Feng Wu. Rethinking branching on exact combinatorial optimization solver: The first deep symbolic discovery framework. In The Twelfth International Conference on Learning Representations , 2024.

Xijun Li, Fangzhou Zhu, Hui-Ling Zhen, Weilin Luo, Meng Lu, Yimin Huang, Zhenan Fan, Zirui Zhou, Yufei Kuang, Zhihai Wang, et al. Machine learning insides optverse ai solver: Design principles and applications. arXiv preprint arXiv:2401.05960 , 2024.

Zhijie Li, Wenhui Peng, Zelong Yuan, and Jianchun Wang. Fourier neural operator approach to large eddy simulation of three-dimensional turbulence. Theoretical and Applied Mechanics Letters , 12(6):100389, 2022.

Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. arXiv preprint arXiv:2205.13671 , 2022.

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 , 2020.

Zongyi Li, Nikola Borislavov Kovachki, Chris Choy, Boyi Li, Jean Kossaifi, Shourya Prakash Otta, Mohammad Amin Nabian, Maximilian Stadler, Christian Hundt, Kamyar Azizzadenesheli, et al. Geometry-informed neural operator for large-scale 3d pdes. arXiv preprint arXiv:2309.00583 , 2023.

Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794 , 2021.

Haoyang Liu, Yufei Kuang, Jie Wang, Xijun Li, Yongdong Zhang, and Feng Wu. Promoting generalization for exact solvers via adversarial instance augmentation, 2023.

Haoyang Liu, Jie Wang, Zijie Geng, Xijun Li, Yuxuan Zong, Fangzhou Zhu, Jianye HAO, and Feng Wu. Apollo-MILP: An alternating prediction-correction neural solving framework for mixed-integer linear programming. In The Thirteenth International Conference on Learning Representations , 2025.

Haoyang Liu, Jie Wang, Wanbo Zhang, Zijie Geng, Yufei Kuang, Xijun Li, Bin Li, Yongdong Zhang, and Feng Wu. MILP-studio: MILP instance generation via block structure decomposition. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Hongyu Liu, Haoyang Liu, Yufei Kuang, Jie Wang, and Bin Li. Deep symbolic optimization for combinatorial optimization: Accelerating node selection by discovering potential heuristics. In Proceedings of the Genetic and Evolutionary Computation Conference Companion , pages 2067-2075, 2024.

Songming Liu, Zhongkai Hao, Chengyang Ying, Hang Su, Ze Cheng, and Jun Zhu. Nuno: A general framework for learning parametric pdes with non-uniform data. arXiv preprint arXiv:2305.18694 , 2023.

Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence , 3(3):218-229, 2021.

Jian Luo, Jie Wang, Hong Wang, Zijie Geng, Hanzhu Chen, Yufei Kuang, et al. Neural krylov iteration for accelerating linear system solving. Advances in Neural Information Processing Systems , 37:128636-128667, 2024.

Yining Luo, Yingfa Chen, and Zhen Zhang. Cfdbench: A comprehensive benchmark for machine learning methods in fluid dynamics. arXiv preprint arXiv:2310.05963 , 2023.

Qitan Lv, Tianyu Liu, and Hong Wang. Exploiting edited large language models as general scientific optimizers. arXiv preprint arXiv:2503.09620 , 2025.

Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, et al. Multiple physics pretraining for physical surrogate models. arXiv preprint arXiv:2310.02994 , 2023.

Katarzyna Michałowska, Somdatta Goswami, George Em Karniadakis, and Signe RiemerSørensen. Neural operator learning for long-time integration in dynamical systems with recurrent neural networks. arXiv preprint arXiv:2303.02243 , 2023.

S Karthik Mukkavilli, Daniel Salles Civitarese, Johannes Schmude, Johannes Jakubik, Anne Jones, Nam Nguyen, Christopher Phillips, Sujit Roy, Shraddha Singh, Campbell Watson, et al. Ai foundation models for weather and climate: Applications, design, and implementation. arXiv preprint arXiv:2309.10808 , 2023.

Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foundation model for weather and climate. arXiv preprint arXiv:2301.10343 , 2023.

Oded Ovadia, Adar Kahana, Panos Stinis, Eli Turkel, and George Em Karniadakis. Vito: Vision transformer-operator. arXiv preprint arXiv:2303.08891 , 2023.

Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.11214 , 2022.

Tianle Pu, Zijie Geng, Haoyang Liu, Shixuan Liu, Jie Wang, Li Zeng, Chao Chen, and Changjun Fan. RoME: Domain-robust mixture-of-experts for MILP solution prediction across domains. In The Thirty-ninth Annual Conference on Neural Information Processing Systems , 2025.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: Adeep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 378:686-707, 2019.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 , 2017.

Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. arXiv preprint arXiv:2306.00258 , 2023.

Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems , 35:1596-1611, 2022.

Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlpmixer: An all-mlp architecture for vision. Advances in neural information processing systems , 34:24261-24272, 2021.

Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. arXiv preprint arXiv:2111.13802 , 2021.

Hong Wang, Zhongkai Hao, Jie Wang, Zijie Geng, Zhen Wang, Bin Li, and Feng Wu. Accelerating data generation for neural operators via krylov subspace recycling. arXiv preprint arXiv:2401.09516 , 2024.

Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing , 43(5):A3055-A3081, 2021.

Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science advances , 7(40):eabi8605, 2021.

Haixu Wu, Tengge Hu, Huakun Luo, Jianmin Wang, and Mingsheng Long. Solving highdimensional pdes with latent spectral models. arXiv preprint arXiv:2301.12664 , 2023.

Liu Yang, Siting Liu, Tingwei Meng, and Stanley J Osher. In-context operator learning with data prompts for differential equation problems. Proceedings of the National Academy of Sciences , 120(39):e2310142120, 2023.

Zherui Yang, Zhehao Li, Kangbo Lyu, Yixuan Li, Tao Du, and Ligang Liu. Learning sparse approximate inverse preconditioners for conjugate gradient solvers on gpus. In Advances in Neural Information Processing Systems , 2025.

Eleftherios C Zachmanoglou and Dale W Thoe. Introduction to partial differential equations with applications . Courier Corporation, 1986.

Xiaoyu Zhao, Xiaoqian Chen, Zhiqiang Gong, Weien Zhou, Wen Yao, and Yunyang Zhang. Recfno: a resolution-invariant flow and heat field reconstruction method from sparse observations via fourier neural operator. International Journal of Thermal Sciences , 195:108619, 2024.

Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: a universal 3d molecular representation learning framework. 2023.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS Paper Checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims are put in the abstract and Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: For further details, please refer to the "Limitations and Conclusions" section, found in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: For detailed information, please refer to the "Method" section in Section 4. Specific experimental parameters are discussed in Section 5, Appendix B.3.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the instruction and code in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Specific experimental parameters are discussed in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For a detailed comparison, please refer to the experimental sections in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Specific experimental environments can be found in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have checked the NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The work presented in this paper aims to develop a sparse architecture designed for PDE pre-training, focused solely on the field of scientific computing. It does not have any direct positive or negative social impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We have checked this and confirmed the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The original papers or URLs of the codes, models, and data sets used in this article have been cited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the instruction in supplemental material about our code and data.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: This paper only utilizes LLMs for writing and editing.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Related Work

A.1 Neural Operators

Neural operators have emerged as powerful tools for learning solution operators of partial differential equations (PDEs) directly from data, demonstrating immense potential across diverse fields such as fluid dynamics [28], thermodynamics [63], electromagnetism [1], and climate forecasting [45]. The wide-ranging mathematical properties and data formats associated with PDEs have driven substantial research into designing effective neural operator architectures.

For instance, DeepONet [36] employs a dual-network structure consisting of a branch and trunk network, while the Fourier neural operator (FNO) [28] introduces a frequency-domain approach to efficiently learn solution mappings. Building on FNO, extensions such as Geo-FNO [26], NUNO [35], and GINO [29] adapt the method to handle complex geometries. Other works [3, 41] focus on time-dependent PDEs, addressing next-time prediction challenges with specialized training strategies. Hybrid approaches like PINO [30] and PI-DeepONet [58] integrate operator learning with physics-informed neural networks (PINNs) [49, 23, 57, 7], leveraging physical constraints to enhance generalization and reduce reliance on large datasets.

Further advancements have been achieved through transformer-based architectures [5, 27, 16, 59, 44], which incorporate techniques like patchification and linear attention mechanisms. For example, the GK-Transformer [5], OFormer [27], and GNOT [16] demonstrate strong performance on problems involving irregular geometries. AFNO [13] combines the efficiency of Fourier transforms with attention mechanisms, inspired by FNO, to achieve low memory and computational costs akin to MLP-Mixer [54]. This approach has been further adapted for large-scale applications such as climate forecasting [45, 61].

Despite these advancements, existing neural operator methods often require task-specific training and large amounts of domain-specific data, underscoring the need for more data-efficient approaches to broaden their applicability and scalability.

A.2 Pre-training in Scientific Machine Learning

Pre-training has emerged as a highly effective paradigm for enhancing downstream tasks by training models in a (self-)supervised manner on large-scale datasets. This approach has achieved remarkable success in traditional domains such as natural language processing [47, 48, 4] and computer vision [19, 18], and is increasingly showing promise in scientific machine learning applications, including protein modeling [21], molecular representation learning [64], and climate and weather modeling [43, 42, 45, 31, 32, 33, 46, 24, 34, 25, 17, 39].

In the context of learning PDE data, initial efforts have been made to explore pre-training across various physical systems [56, 9, 37]. For instance, [52] designs a relatively universal PDE model to collectively train data from multiple steady-state PDEs. [60] utilizes the MathGPT architecture to investigate in-context learning capabilities for PDE data. Additionally, MPP [40] introduces an auto-regressive approach for pre-training on time-dependent PDE datasets. [15] proposes an autoregressive denoising pre-training strategy combined with a scalable Fourier-based model architecture, enabling efficient large-scale pre-training on PDE data.

Additionally, works such as Poseidon [20], which are based on a multiscale operator transformer, have achieved excellent pre-training effectiveness and generalization by employing a novel training strategy that leverages the semi-group property of time-dependent PDEs. These approaches primarily focus on mapping parameters to PDE solutions, which contrasts with the auto-regressive solution method discussed in this paper, giving each methodology a distinct scope of application.

However, these works primarily rely on dense neural network architectures and do not explicitly consider the relationships between different PDE datasets, which could significantly impact model performance. There remains substantial room for exploration in pre-training models for more complex scenarios and larger parameter spaces.

B Details of Experiment Settings

B.1 Data Preprocessing and Sampling

We adopt the data preprocessing strategy proposed in DPOT [15], with modifications to ensure compatibility across diverse PDE datasets.

Data Padding and Masking. To standardize spatial resolution, we fix the resolution at H = 128 , which aligns with a significant portion of the datasets. For datasets with lower resolutions, we upscale them to H using interpolation. For datasets with higher resolutions, we downscale them to H using random sampling or interpolation.

To unify the number of variables (i.e., channels) across different PDEs, we pad all datasets along the channel dimension to match the dataset with the maximum number of channels, filling unused entries with a constant value (e.g., 1). For datasets with irregular geometric shapes, we use an additional mask channel that encodes the geometric configuration of each PDE instance. This ensures consistent representation across datasets while preserving unique structural information.

Noise inserting. During the training process, we add noise to improve the stability of the training. We only insert noise during the pre training process, and do not insert noise in fine-tuning or downstream tasks. The insertion method of noise is as follows. For ∀ t ≤ T , denote u <t as ( u 0 , . . . , u t -1 ) and the noise as ε ∼ N (0 , ϵ || u <t || I ) . Then the input is u <t + ε .

Balanced Data Sampling. To balance the contribution of datasets with varying sizes, we assign an importance weight w k to each dataset. Let |D k | denote the number of data points in the k -th dataset, where 1 ⩽ k ⩽ K . The probability of sampling a data point from the k -th dataset is computed as:

This sampling strategy ensures that datasets with fewer samples or higher importance scores are appropriately represented during training.

Patchification Layer. We follow the patch-based tokenization strategy used in Vision Transformers [10]. Given a spatiotemporal input tensor u <T ∈ R H × W × T × C , we apply a convolutional embedding layer with kernel size P × P and stride P . This partitions the spatial domain into non-overlapping patches of size P × P . Each patch is mapped to a d -dimensional embedding vector via a shared linear projection implemented as a convolution:

This process generates a sequence of patch tokens for each timestep, which are subsequently fed into the downstream attention layers. By reducing spatial resolution while preserving local structure, this patchification approach enables efficient global modeling in the transformer blocks.

B.2 Design of the MoE Structure

As detailed in the Section 4, our MoE architecture is built upon a CNN framework and utilizes both shared and routed experts in parallel. The design of the current MoE-POT architecture is the culmination of extensive experimentation. We explored numerous configurations, including combinations of MoE with other neural operators (such as MPP, FNO, and FFNO) and alternative MoE structural designs (e.g., omitting shared experts or employing MLP-based experts instead of CNNs). Many of these alternative designs proved unstable, yielding models with substantial errors on certain datasets that could not be corrected through fine-tuning. The final MoE-POT architecture presented in this work is the robust and effective design that emerged from this rigorous development process.

B.3 Model sizes and training details

Pre-training. We selected 3 models of varying sizes, i.e., MoE-POT-Tiny, MoE-POT-Small, and MoE-POT-Medium. The specific parameters of these models are shown in Table 5. For the pretraining stage, we set the learning rate to 1 × 10 -3 and used a One-cycle learning rate schedule over 1000 epochs, with the first 200 epochs as the warm-up phase. The Adam optimizer was employed with a weight decay of 1 × 10 -6 and momentum parameters ( β 1 , β 2 ) = (0 . 9 , 0 . 9) . Training was

conducted using 8 RTX 4090 GPUs, with a total batch size of 20. The patch size was set to 8. We set the weight w = 1 for all datasets. For our training process, we selected T = 10 timesteps to predict the next frame, aligning with the original settings of most datasets.

Fine-tuning. Our model supports fine-tuning across various downstream datasets while retaining the generalization capability learned during pretraining. Specifically, we freeze the parameters of the router-gating network during fine-tuning to preserve the expert assignment strategy obtained from the joint training stage. This strategy allows the model to reuse the learned routing behavior, enabling different experts to specialize in different data distributions. Only the expert networks are updated to adapt to the target dataset, while the router-gating network continues to provide consistent and stable expert selection. This separation of routing and expert adaptation ensures more stable and efficient fine-tuning, particularly when transferring to tasks with limited data.For the fine-tuning stage, we set the learning rate to 1 × 10 -3 and used a one-cycle learning rate schedule over 200 epochs, with the first 40 epochs as the warm-up phase. And for the downstream tasks, we set the learning rate to 1 × 10 -3 and used a one-cycle learning rate schedule over 500 epochs, with the first 100 epochs as the warm-up phase.

be noted that NS (1e-4) and CNS (1,0.01) have similar datasets in the pre-training dataset, while pdearea differs significantly from the pre-training dataset.

Details of inference. To learn from temporal PDE datasets, our network G w ( u t<T ) parameterized by weights w that auto-regressively takes T frames as input and decodes the next frame from previous frames,

By predicting the next frame, we can infer the numerical solution of the final time step based on auto-regression. For example, if we take the first 10 steps as our input, we can predict the solution x pred for the next 10 steps. And the ground truth is x gt , then the loss is

B.4 Interpretable Analysis Algorithms

To further investigate the router-gating network within the MoE structure, we designed the following experiment. Our goal is to leverage the section of the router-gating network to determine which dataset the input data belongs to.

Specifically, given an input sample X , the gating network outputs a probability vector Y ∈ R 16 representing the likelihood of selecting each of the 16 experts. Although only the top-4 experts are used during inference, the full softmax output encodes meaningful distributional information about expert preferences.

For a specific block, we compute the average expert selection distribution Y i = 1 N i ∑ N i j =1 Y ij , and Y i , Y ij ∈ R 16 , where N i is the number of samples from i -th dataset, i = 1 , ..., 6 . Y ij is the routergating network output for the j -th sample in i -th dataset. Then, for any new input X , we compare its expert distribution I 0 = ( I 0 , 1 , ..., I 0 , 16 ) to all Y i = ( Y i, 1 , ...., Y i, 16 ) using the cross-entropy loss function:

Suppose i 0 represents the nearest dataset.

In this case, we classify the input data X as belonging to the i 0 -th dataset.

B.5 Mathematical Forms of Datasets

Here, we list the PDEs of the datasets we used for pre-training.

FNOν [28]: The quantity of interest (QoI) is the vorticity w ( x, t ) , ( x, t ) ∈ [0 , 1] 2 × [0 , T ] and it satisfies, ν represents the viscosity coefficient. In paper are FNO (1e-3), FNO (1e-4) and FNO (1e-5).

PDEBench-CNS( η, ζ ) [53]: We need to predict the velocity, pressure, and density fields u ( x, t ) , p ( x, t ) , ρ ( x, t ) where ( x, t ) ∈ [0 , 1] 2 × [0 , 1] . The PDEs are as follows . η is dynamic shear viscosity and ζ is bulk viscosity.In paper are CNS (0.1,0.01), CNS (1,0.01).

PDEBench-SWE [53]: We need to predict water depth h ( x, t ) where the domain is [ -1 , 1] 2 × [0 , 5] . The PDEs are as follows,In paper is SWE.

PDEBench-DR [53]: We need to predict the density fields u ( x, t ) . The domain is [ -2 . 5 , 2 . 5] 2 × [0 , 1] and the PDEs are as follows,in paper is DR

PDEArena-NS1/2 [14]: We need to predict the velocity, pressure, and density fields u ( x, t ) , p ( x, t ) , ρ ( x, t ) where ( x, t ) ∈ [0 , 32] 2 × [0 , 24] . The PDEs are as follows,in paper is PDEArena.

CFDBench [38]: We need to predict the velocity and pressure fields u ( x, t ) , p ( x, t ) . The domains are different as this is a dataset with irregular geometries. The PDEs are as follows, in paper is CFDBench.

C Experimental Data and Supplementary Experiments

C.1 Partial Hyperparameter Ablation Experiment

Table 8: Results of ablation experiments on the influences of the number of heads h (left part) and patch sizes P (right part). L2RE is used as the evaluation metric.

Table 8 demonstrates that the number of heads h has minimal impact on error but affects computational cost. Accordingly, we choose h = 4 for efficiency. Finally, medium patch sizes ( P = 4 or 8 ) help reduce error, leading us to select P = 8 for optimal performance.

C.2 More Comparative Experiments

We selected DPOT [15] as our primary multi-physics baseline for the following reasons:

Clear Attribution of Gains: Our MoE-POT architecture is a direct modification of the DPOT model, where we replace the dense feed-forward network with our proposed sparse MoE layer. This controlled comparison allows us to cleanly attribute any performance improvements directly to the MoE architecture, providing a clear and rigorous validation of our core contribution.

Divergent Experimental Paradigms: Our work, following DPOT, operates under an auto-regressive paradigm, predicting future states based solely on a sequence of previous solution frames. In contrast, models like Poseidon [20] and MPP [40] are designed for a parameter-informed setting, where they take explicit problem parameters (e.g., coefficients, boundary conditions) as input to predict a future state. The public benchmark datasets used in our primary experiments (from FNO, PDEBench, and CFDBench) do not provide these explicit PDE parameters, making a direct comparison with parameter-informed models infeasible under our main experimental protocol.

This section provides additional experimental results and analysis to supplement the main paper. We compare our MoE-POT architecture with the larger DPOT-L model and the Poseidon.

C.2.1 Experimental Results with DPOT-L

To provide a more comprehensive comparison against large-scale dense models, we evaluated the performance of DPOT-L (493M parameters). The results, alongside our MoE-POT models and smaller DPOT variants, are presented in Table 9.

The results in Table 9 lead to two key observations: 1. Diminishing returns for dense models: The performance of the dense DPOT architecture shows diminishing returns with scale. The improvement from DPOT-M (122M) to DPOT-L (493M)-a 4 × increase in parameters-is marginal on several datasets (e.g., NS(1e-3)) and modest on others. 2. Competitive performance with higher efficiency: When comparing MoE-POT-M (288M activated) with DPOT-L (493M activated), our model achieves competitive, and in some cases superior, performance. MoE-POT-M outperforms DPOT-L on three of the six datasets (NS(1e-3), CNS, CFDBench), while DPOT-L holds a slight advantage on the other three.

C.2.2 Experimental Results with Poseidon

To address the interest in comparing with the latest models, we conducted supplementary fine-tuning experiments on two challenging downstream tasks from the Poseidon paper [20]: Wave-Layer and Wave-Gauss. We evaluated both MoE-POT and Poseidon under two distinct settings to fairly assess their capabilities.

Setting 1: Auto-regressive (Our Native Setting) In this setting, models predict future states using only previous solution trajectories, without access to explicit PDE parameters. The results are shown in Table 10.

The results from these supplementary experiments indicate that each model excels in its native operational setting. In the auto-regressive setting (Table 10), where PDE parameters are unknown, MoE-POT significantly outperforms Poseidon. This highlights our model's strength in implicitly learning system dynamics from solution trajectories alone.

Conversely, in the parameter-informed setting (Table 11), Poseidon generally demonstrates superior performance, showcasing its effectiveness when explicit physical knowledge is available. These findings suggest that MoE-POT and Poseidon have different primary application scopes rather than one being definitively superior across all scenarios.

C.3 Rollout Error at Different Timesteps

Akey challenge in auto-regressive prediction is the accumulation of errors. Even a small improvement in single-step prediction accuracy can lead to a substantial reduction in the cumulative error over a long rollout, as prediction inaccuracies propagate through the sequence.

As demonstrated in Table 12, the error for both models increases over the rollout period, confirming the effect of error accumulation. More importantly, the performance advantage of MoE-POT-S grows significantly over time. The relative error reduction compared to DPOT-S is approximately 23% at frame 50, but this gap widens to over 31% by frame 100. This super-linear divergence underscores the critical impact of achieving lower single-step prediction error, as its benefits are amplified during long-term, multi-step rollouts.

C.4 Analysis of Fine-Tuning Sample Efficiency

This section investigates the relationship between the number of fine-tuning samples and model performance, thereby analyzing the data efficiency of the MoE-POT architecture. We conducted fewshot fine-tuning experiments on both an in-distribution and an out-of-distribution task to demonstrate how performance scales with data availability for both MoE-POT and its dense counterpart, DPOT.

We compare the performance of MoE-POT-S and DPOT-S on two fine-tuning tasks: 1. In-Distribution Task: The NS (1e-4) dataset, which is closely related to the data used during pre-training. 2. Out-ofDistribution Task: The Wave-Layer dataset from the Poseidon [20], which represents a novel physical system.

For each task, we fine-tuned both models for 500 epochs while varying the number of available training samples, and we report the final L2 Relative Error. The results for the in-distribution and out-of-distribution tasks are presented in Table 13 and Table 14, respectively.

The results demonstrate a clear trend: while both models improve with more data, MoE-POT-S consistently outperforms DPOT-S across all sample sizes on both tasks. This highlights the superior data efficiency of the MoE-POT architecture. The larger capacity and specialized experts of the pre-trained MoE-POT model enable it to generalize more effectively from limited data. This means it can either achieve a target performance level with significantly fewer fine-tuning examples or deliver superior accuracy given the same amount of data.

C.5 Performance with Increasing Dataset

A core motivation for our work is the challenge of negative transfer [6] in dense neural operators when pre-trained on a mixture of heterogeneous PDE datasets. A single, dense network struggles to learn conflicting physical laws, which can degrade performance as more diverse data is added. This section presents an experiment designed to test this hypothesis and demonstrate the robustness of the MoE-POT architecture in mitigating this issue.

To investigate the impact of increasing data heterogeneity, we pre-trained both the dense DPOT-S model and our sparse MoE-POT-S model on progressively larger and more diverse mixtures of datasets. For each experiment, the models were trained from scratch on the specified data mixture. We evaluated their zero-shot performance on the original six pre-training datasets to measure how well they retained knowledge.

The dataset mixtures were constructed as follows:

6 Datasets: The standard pre-training set used in our main experiments: NS(1e-5), NS(1e-3), CNS(0.1, 0.01), SWE, DR, and CFDBench.

10 Datasets: The base set plus four additional datasets from the DPOT paper [15]: NS(1e-4), CNS(1, 0.1), and two Navier-Stokes tasks from PDEArena.

12 Datasets: The 10-dataset mix plus two additional CNS variants: CNS(1, 0.01) and CNS(0.1, 0.1).

The zero-shot L2 Relative Error (L2RE) for both models across the different pre-training configurations is presented in Table 15.

DPOT-S (Dense Model) The performance of DPOT-S is unstable and often degrades as more heterogeneous data is introduced. While adding four datasets (from 6 to 10) yields minor improvements on some tasks, it causes significant performance degradation on others (e.g., DR and CFDBench). Expanding to 12 datasets results in a notable performance collapse on several tasks (e.g., NS(1e-3), SWE, DR) compared to the original 6-dataset training. This confirms that the dense architecture suffers from negative transfer, where the model's capacity is overwhelmed by conflicting information from diverse physical systems.

MoE-POT-S (Sparse Model) In stark contrast, MoE-POT-S demonstrates remarkable robustness. As the number of pre-training datasets increases from 6 to 12, its performance remains stable or even improves on most tasks (e.g., CNS and CFDBench). The slight variations in error are minor compared to the drastic fluctuations observed with DPOT-S. This stability indicates that the MoE architecture effectively mitigates negative transfer by allowing different experts to specialize in distinct physical dynamics, thereby preventing knowledge conflict.

C.6 Extended Interpretability Analysis: Emergence and Generalization of Router Specialization

This appendix expands on the interpretability analysis in Section 5.4, which demonstrated the routergating network's ability to classify input data by its source PDE. This capability is not explicitly

programmed; the router only processes tokenized inputs, and the model's loss function is based on prediction error, not a classification objective. Here, we investigate two key questions: (1) How does this specialization emerge during pre-training? (2) Does this learned capability generalize to entirely new, OOD datasets?

Emergence of Specialization During Training We first examine how the router's classification ability develops. We tracked the dataset classification accuracy at different stages of pre-training, using the method described in Section 5.4. The results, shown in Table 16, confirm that this is an emergent property. Initially (Epoch 50), the accuracy is low, but it rapidly improves and reaches 100% by Epoch 250. This demonstrates that the router learns to distinguish between data distributions as part of the end-to-end optimization process.

Table 16: Evolution of the router's dataset classification accuracy over training epochs. The accuracy steadily improves, eventually reaching 100% as the experts and router co-specialize.

Generalization to Out-of-Distribution Datasets To test if this specialization generalizes beyond the pre-training distributions, we evaluated the router's classification performance on two OOD tasks from the Poseidon benchmark [20]: Wave-Layer and Wave-Gauss. These datasets represent novel physical systems unseen during pre-training.

As shown in Table 17, the router achieves 100% classification accuracy on both unseen tasks. For instance, when processing the Wave-Layer dataset, the router in Block-2 consistently activated a sparse subset of experts (e.g., Expert 11 at 100% usage, Expert 1 at 79%), with other experts receiving minimal or zero activation.

Taken together, these results provide strong evidence that the router's ability to identify PDE types is an emergent property of joint optimization. To minimize the global prediction loss across heterogeneous datasets, the model learns to partition its knowledge, routing inputs with similar dynamics to specialized experts. This process effectively mitigates the negative transfer that hinders dense architectures. Crucially, the perfect classification accuracy on OOD data demonstrates that the router is not merely memorizing training distributions. Instead, it learns to recognize fundamental properties of the underlying physics from the tokenized solution data, a capability that generalizes to novel systems.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nContext: PDEs are very important as mathematical models in physics and engineering and given the slow speed of physics-based PDE solvers, the design of neural PDE surrogates (also termed Operator Learning) is very important. Most of the research in operator learning as been in the task specific context. However, it is well known that the number of training examples need per task can be too high to be practically useful. Hence, the design of Foundation models has been pursued recently as data across PDE operators might be leveraged to  reduce sample complexity per task through finetuning. Some recently developed PDE Foundation models are MPP, DPOT and Poseidon (which the authors strangely omit from their literature review). However, there is certainly enormous scope for further development of Foundation models for PDEs. \n\nGoal: The article proposes a PDE Foundation Model, called MoE-POT, aiming to reduce model size and improve scaling. \n\nMethod: The basic goal is operator learning for time-dependent PDEs. The task is \"given the PDE solution at a set of time steps, predict the solution at the next time step\". To do so, MoE-POT reuses most elements of DPOT -- patchification, temporal aggregation, noise addition, Fourier layers etc. The main new element is a mixture-of-experts (MoE) Module. At each processor layer, the outputs of the Fourier layer are fed into a set of shared and routing experts. The shared experts are always active and the routing experts are activated through a gating network. The expert and gating network are CNNs. During training, a standard load balance loss is added to promote balance between the experts and prevent routing collapse. \n\nResults: The pretraining dataset are 6 operators (Incompressible Navier-Stokes with 2 different viscosities from the original FNO paper, A compressible Navier-Stokes, Shallow-water and a Reaction-Diffusion dataset from PDEbench and another forced Incompressible Navier-Stokes example from CFD bench. All are 2-D fairly academic problems. Most of the evaluation is on the Pretraining dataset itself, either in the form of zero-shot test loss or with additional fine-tuning. 3 Downstream tasks are also considered: Incompressible Navier-Stokes with a different viscosity from the FNO paper, Compressible Navier-Stokes with different parameters from PDEbench and a Navier-Stokes dataset from PDEarena. MoEPOT is shown to be either competitive or better than baselines (small neural operators and DPOT). Additional interpretability is shown through the importance values of gating weights.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n--- The problem being addressed is of great importance in Scientific machine learning. \n--- First direct application of MoE in a PDE Foundation Model setting\n--- MoEPOT outperforms DPOT at least marginally.\n--- It is always an engineering challenge to train PDE foundation models and the authors have done so.\n--- The classification of datasets using the routing networks (Fig 4(c)) is a very interesting observation.  \n\nWeaknesses: \n\n--- Limited Novelty: The only novel model contribution is the use of MoE in this setting as all the other choices in architecture and dataset selection are based on DPOT. Nevertheless, MoEs have been widely used in Foundation models for text and vision. \nAlso, MoE have been widely used in scientific machine learning:\nRafael Bischof and Michael A Kraus. Mixture-of-experts-ensemble meta-learning for physics-informed neural networks. In Proceedings of 33. Forum Bauinformatik, 2022, \nZheyuan Hu, Ameya D Jagtap, George Em Karniadakis, and Kenji Kawaguchi. Augmented\n360 physics-informed neural networks (apinns): A gating network-based soft domain decomposition, Engineering Applications of Artificial Intelligence, 126:107183, 2023, \nand see in particular arXiv:2404.09101 for even a theoretical analysis of MoEs for operator learning. \nGiven this context and the absence of theory, the reviewer has to judge this paper based on the empirical results. \n\n--- Incomplete literature review and context: The authors do not cite or contextualize let alone compare with Poseidon (arXiv:2405.19101) published in Neurips last year, which in the opinion of the reviewer, is the current state-of-the-art PDE foundation model (it is compared and shown to be superior to DPOT in Sec D.5 of arXiv:2405.19101 at least on a wide range of downstream tasks. \n\n--- Lack of Clarity: This paper is badly written. The narrative does not flow smoothly. For instance, Sec 3.2 comes in abruptly with a lot of jargon that is totally unhelpful for the reader. Just look at the number of questions that this reviewer poses below to see why the paper lacks clarity. \n\n---- Incorrect Operator learning task: Given a time-dependent PDE such as Eq. [1], the solution operator of a PDE is very well defined as mapping from the initial conditions $u^0$, PDE parameters $\\theta$ and boundary conditions and to the solution $u(t)$ of (1) for any time t >0. This operator is well-posed and is exactly the mapping that is learned by a numerical Physics-based PDE solver. \n\nOn the other hand, the authors misinterpret this learning task to: Given the trajectory $\\{ u(\\tau) \\}$ for $0 \\leq \\tau \\leq t$, map this input into the PDE solution $u(t+dt)$ for some lead time $dt$. This is different from the PDE solution operator and it is over-determined and may not be mathematically correct for many PDEs. From a practical perspective, unless $t=0$, it will require the end user to first run a numerical solution of the underlying PDE (access to knowledge about all the PDE parameters) for some time $t$ and then use your neural operator to update the solution. However, such a solver may not be available for inference. Moreover, the upfront cost of setting up such a solver if available (for instance in mesh generation) can be very significant, vastly limiting utility of the PDE surrogate.\n\nAlthough, the authors are not the first to setup this operator learning task, this incorrect or limited utility task has crept into the SciML community from surrogates for weather forecasting, where it is totally justified as weather forecasting is an online streaming task whereas solving PDEs is rarely so. \n\nIn particular, it does not really matter what the authors do at Pretraining as they have all the trajectory data but for downstream tasks, you have to run your model such that it can learn from the initial data to the solution trajectory. See arXiv:2405.19101 OLT on Page 3 for more context. \n\n---- Very Limited Evaluation: As common in text/vision, evaluating on pretraining dataset is not of much importance for a Foundation model, except perhaps for model and dataset size scaling. In fact, (V)-LLMs do not even have the same pretraining objective as that in the downstream tasks (for instance masked modeling). The whole point of a foundation model is how it performs on downstream tasks and NOT on the pretraining dataset as an end-user will have very different inference tasks (PDE types or IC/BC) than the Foundation model designers. This issue was carefully analyzed in arXiv:2405.19101 and its authors clearly compared model performance on downstream tasks in terms of two reasonable metrics i) what is the gain in terms of how few samples are needed to finetune a Foundation model on a downstream task to obtain the same error as that of a task-specific baseline with a given number of samples (see E_G in Eqn (11) of arXiv:2405.19101). This tells the user how few samples that they can get away with to obtain reasonable errors and ii) for a given number of downstream task-specific examples, the gain in accuracy over the baseline model (A_G in Eqn (11) of arXiv:2405.19101). \n\nThus, all the evaluation of a Foundation Model should be at the level of downstream tasks. This paper does exactly the opposite -- bulk of its results are on the pretraining dataset (in-distribution) and there is very limited evaluation on the downstream tasks. Thus, the whole point of the model and its purported gains in performance are very incomplete. \n\n--- Poor Choice of Downstream tasks: Given the above point, any Foundation Model has to be evaluated on a challenging set of downstream tasks. The authors make a very poor selection of such tasks. The first downstream task is just the same NS example from the original FNO paper but with a different choice of viscosity, which conveniently lies between the viscosity coefficients considered in the pretraining dataset. The second downstream is just CNS with a different (10 times larger) coefficient from the CNS dataset considered in the pretraining data. Again, this is very related and even simpler than the one similar task in the pretraining data. The final downstream task is again Incompressible forced Navier-Stokes (already seen twice in the pretraining dataset) with perhaps different initial conditions. This reviewer can buy this example as a genuine downstream task but here, MoEPOT has exactly the same error as DPOT (6.21 % vs 6.18 %). None of the downstream tasks represent physics unseen during the pretraining, just contrast this with the Poseidon evaluation where 9 of the 15 downstream tasks were for unseen physics. After all, no Foundation model can be pretrained on all the physics in the universe and needs to be tested on unseen physics. \n\n--- Questionable choice of Pretraining dataset:  Out of the 6 operators in the pretraining dataset, 4 are very simplistic as even small models (less than 2M params) are able to attain less than 1% relative error. Only the CNS and FNO task with very low viscosity are relatively challenging. It is unlikely that a Foundation model trained on such simple operators can be expected to generalize well to complicated physics. \n\n--- Underwhelming results: Given the above critique on zero-shot test losses, even here, on the 4 easy datasets, very small models are as good as MoEPOT and on the difficult ones, the gains wrt DPOT are marginal (on FNO low-viscosity: MoEPOT-M 5.28% vs. DPOT-M 5.69% and on CNS: MoE-POT-M 1% approx vs. DPOT 2% approx) are hardly worth writing about. The authors claim that their model scales better than DPOT but this is not true on the evidence of Fig. 4(b) where DPOT has a sharper gradient than MoEPOT and is already catching up for moderate model sizes. \n\n--- Lack of comparison with Poseidon: It would certainly be worthwhile to compare your model with DPOT and Poseidon on a neutral downstream task, something not seen in either of the pretraining datasets -- a very good choice for that would be the wave-layer and wave-gauss datasets of arXiv:2405.19101, which are publicly available and where both Poseidon and DPOT have already been tested. \n\n--- Lack of data scaling: The whole point of a Foundation model is to require less task-specific examples downstream. Please provide such results for MoEPOT -- see arXiv:2405.19101 Figs. 7-21 for how to perform these scaling tasks. \n\n--- Lack of visualization: The authors do not provide any examples of how exactly their model performs in terms of sample visualization.\n\nQUESTIONS:\n1. Fig 2 (Left): Why would the performance of FNO deteriorate so sharply on a mixed dataset ? If so, why does DPOT work in the same setting as it only time aggregates and patches, when compared to FNO. Anyway, there is too little information about this figure in the caption or the text. \n\n2. L79-81: Where exactly is the noise injected ? Is it uniform over time steps ? \n\n3. L90-91: See relation to Q1.\n\n4. L96: Explain feature competition and knowledge illusion ? or else drop jargon !\n\n5. L102: Long Inference time is a misnomer: DPOT-L takes 25 milliseconds to infer compared to 17 milliseconds for MoEPOT-L. This has be compared to the min-to-hour time scale for PDE solvers. Inference is not an argument for MoEs in this setting. \n\n6. Sec 3.2 should be moved ? It does not fit here ?\n\n7. L108: Explain parallel expert networks and gated routing, else drop jargon till you explain it in the next section. \n\n8. L136: Should the spatial dimensions of Z^t_p be HxW ? Isn't patchification supposed to reduce spatial size ? \n\n9. L135: The notation W_p(x_i,y_j,t) is unclear, I am assume that W_p is just a fixed matrix or is it a MLP ? \n\n10. L138: What is W_t ? Is it a MLP ? \n\n11. L139: What are Fourier features \\gamma -- are they learnable or not ? \n\n12. L141: Why do you call Eq (7) as a Fourier Attention layer ? It is just  Fourier layer from FNO and no attention mechanism is used here.\n\n13. L151: Is the group and concatanation over channels or over spatial points, it is unclear from Eq (7) as the weight matrices have dimensions of the space with patches ? Is h the \"patch size\" ? \n\n14. L156-157: Why choose CNNs here -- shouldn't FNO be a natural choice ? \n\n15. L171: Is the Batching over samples only or over spatial points -- I ask as you use x (spatial coordinate) in Eqn (11) below. \n\n16. L172: What is importance in Eq (11) and what is e_i ? Aren't they the same ? \n\n17. L175: The phrase \"one step transition between samples from different datasets\" is misleading here -- you just want to predict the next state, given a sequence of states. \n\n18. L187: The claim on \"wide range of PDEs\" is disputable as you only consider fluid equations with 1 Reaction-Diffusion equation -- PDEs have a much wider variety. \n\n19. L194: L2RE but for what time ? Is it average one-step prediction error ? or final time evaluation error ? \n\n20. L201-202: The claim \"SOTA\" pre-trained neural operator (DPOT) can be contested, see arXiv:2405.19101 Sec D.5. \n\n21. Table 1: What does 0.1 and 0.01 in CNS stand for ? \n\n22. L221: Why do you need to fine-tune your pretrained model on the same tasks as in the pretraining dataset. Is it taking the model checkpoint and then task-specific training ? What is the fine-tuning procedure ? \n\n23. L238-239: \"represents entirely different equations\" is simply wrong - it is just forced incompressible Navier-Stokes for which you already have 2 datasets in the pretraining data. \n\n25. L254: Claim on superior performance is untrue as DPOT has a sharper gradient in Fig. 4 (b). \n\n26. Fig 4 a and b: please provide averages over all the tasks or provide all the tasks in the Supplement. \n\n27. L262 and L263: Y and I are introduced here out of thin air -- what are they ? At least provide link to Supplement \n\n28. L543 -- what is T ?\n\n29. L546: What are the constants -- see Q 21 ?\n\n30. L550: Define what u, D and R are ?\n\n31. L551-552: What are the initial conditions ? \n\n32. L554: What are the geometric shapes ? How do you carry geometric information ? \n\n33. Please state how you normalize different spatial and temporal domains in your datasets ?\n\n34. How do you set the number of input time steps $t$ in your operator learning task ? Is it the same for all the datasets. \n\n35. To reach a certain time from the initial data (and the initial time step sequence), do you use an autoregressive rollout ? If so, how does the error accumulate over time ? Please present a plot to show this. \n\n36. Fig 4 (c): Please provide average classification accuracy over all blocks ? There is presumably nothing special about block 2. \n\n37. It would be great if Fig 4 (c) can be shown for each of the downstream tasks, by comparing with the pretraining dataset -- it make sense that the cross-entropy loss for FNO is close to the FNO datasets, Cross-entropy loss for CNS is close to the CNS pretraining datasets and the one for PDEarena is also close to the FNO dataset. The exact values of the cross-entropy losses should be compared with the ones obtained for Fig 4 (c) and I am speculating that the results will be consistent with my intuition that the downstream tasks are not really that out-of-distribution with respect to the pretraining dataset -- this will be a very interesting test of the authors' method.\n\n38. Although Fig 4 (c) shows that the expert networks are able to separate the pretraining datasets -- the authors do not explain why this is the case ? After all, the experts have only access to processed tokens and not to the datasets directly and there is nothing in either the model or in the loss function so that the experts can distinguish between different datasets ? Is this an emergent property ? If so, can the authors indicate if this arises after a certain model size, certain number of training epochs or certain size of the dataset ? \n\n39. What is the size of the total pretraining dataset in terms of the number of training samples ? \n\n40. What is the size of the dataset for each downstream task ?\n\nLIMITATIONS:\nLimited Novelty: See Weaknesses.\n\nLimited scope of operator learning task: Next time-step prediction from a sequence, unclear how to use parameters explicitly, unclear how to directly predict from initial data, unclear how to do steady state predictions and large time predictions. Error accumulation is possible due to long-time autoregressive rollouts. Restricted to 2D, mostly cartesian. \n\nLimited Evaluation: particularly at the level of downstream tasks, unclear if there is any gain with respect to DPOT on genuine downstream tasks. No comparison to Poseidon and MPP. \n\nEvaluation: Although very natural in this context, given its widespread use in LLMs, the authors are the first to use a standard MoE module in a PDE foundation model. However, they have not demonstrated that it leads to any benefits, particularly on downstream tasks involving unseen physics. hence, this reviewer cannot recommend acceptance in current form but I am willing to modify my score if the authors can rebut the criticism, particularly if they provide evidence for their model directly predicting solution trajectories from initial data, at least at the level of downstream tasks, they test on unseen downstream tasks such as the Wave-Layer and Wave-Gauss experiments of arXiv:2405.19101 (datasets are publicly available) and benchmark performance over DPOT and Poseidon, provide data scaling plots to show that MoEPOT is competitive in the low task-specific sample regime  and answer the minor questions above satisfactorily.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a mixture of experts module at the end of a SOTA neural operator (DPOT) to deal with two challenges of building large SciML foundation models: (a) performance degradation due to heterogeneity of PDE datasets and (b) too many parameters necessary for dense models to alleviate the previous challenge. The authors pre-train the DPOT with MOE on 6 PDE datasets and demonstrate zero-shot and fine-tuned performance on several downstream tasks. They show that the MOE addition uniformly helps increase the accuracy of DPOT. Further, they show that their MOE activations correspond to learning the underlying PDE task distribution, motivating its creation. Some ablations on number of experts demonstrate that a balance is being struck w.r.t accuracy vs speed. Finally, the authors show the practical utility of their model by comparing inference times - the setting where these models will be used in production.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n* The paper is well-motivated, the topic is of high interest to the NeurIPS audience\n* Performance of the MOE model is better than DPOT (a SOTA pre-training architecture for SciML)\n* Experiments demonstrate that the experts learn the PDE data in-distribution structure by looking at the MoE activation distributions\n\nWeaknesses:\n* The paper's clarity could be improved (see comments). While the motivation is clear, the results section could benefit greatly with more clarifications\n* The improvement from DPOT seems marginal and limited to one-step improvement. It's unclear how much this translates to rollout improvements\n* There is very little discussion on out-of-distribution generalization properties of the MOE model (or others)\n* The interpretability analysis (while appreciated) could be made more extensive to stress-test the motivation behind the MOE (see comments)\n\nQUESTIONS:\nClarity:\n* The results are unclear in some places. It would be useful to demonstrate clearly what is the train/val/test split for each dataset used, which datasets are used for pre-training vs zero-shot vs finetuning, size of the datasets for each case (pre-training/finetuning), degree of difficulty for the downstream tasks w.r.t out-of-distribution generalization (this can be as simple as quantifying the effects of the physics coefficients on the difficulty of the tasks -  some tasks such as diffusion would be easier to generalize rather than more hyperbolic systems and hence the error metrics are not sufficient to gauge the improvements). \n* Tab1. would be more beneficial as some sort of figure (error vs fine-tuning dataset size) - it's then clear what value the MOE adds in addition to the dataset volume. \n* Fig 2 (right) would be beneficial to see it as one figure (with different color bars) so it's easier to compare the MOE activations\n\nExperiments: \n* The error over DPOT seems very comparable. How many finetuning examples are taken? How much care is given to optimization when finetuning to a new dataset (learning rates, batch size, frozen weights)? Will DPOT eventually catch-up to MOE with more examples and at what point (how many examples)? Finally, I understand that dropping the error from 0.02 to 0.01 is a 100% improvement, but how valuable is that if your error is already low? It seems like DPOT does get low errors already in most cases. I think it would be beneficial to stress-test on more out-of-distribution examples.\n* I appreciate the interpretability experiment. I feel like the authors could do more with this line of analysis for this paper. For example, when you look at classification accuracy, is it just for a training sample X or validation/test? How does the accuracy change as you pick X from a downstream task and systematically make that downstream task out-of-distribution? Do the MOE logits still point to some pre-training dataset or does it become more uniform?\n* Related to above, the authors mention that part of the MOE is to learn common representations (conservation laws) and the other part is to learn the PDE task distribution? Is there any evidence to suggest the former? Such as high activation for the shared experts for all datasets? Similarly, if you picked X (input) from an out-of-distribution example, would you still see the shared experts being activated? That would be a nice result.\n* Another related thread: if part of the MOE is just to distinguish the PDE, why not just one-hot encode that information at the beginning (or have the MOE as well at the beginning). It's understandable in fields like language modeling that the source of the tokens is pretty unclear but the science, data is often from structured simulation+experiments and hence it should be possible to encode that information directly?\n* The inference time experiment is not convincing. DPOT-M and MOE-M have roughly the same inference time. Tab 1 only compares the accuracy of these two models (as the best models), so I'm wondering what the advantage of MOE is? Also what is the accuracy of DPOT-L in Tab. 1?\n* How does the MOE block impact rollout accuracy? This probably is more important than one-step prediction accuracy for spatiotemporal models. It would be good to discuss the multistep performance as well.\n* [Minor] What is the performance of the FNO with the MOE? The FNO numbers seem disentangled from the whole experiment and seem to only emphasize that DPOT is a better architecture choice?\n\nMinor:\n* Fig labels and tick sizes could be made larger so it's clear\n* Tab 1. The caption says error greater than 20. I did not follow what that means\n* What is n in W_p (after Eqn (3))? embedding dimension?\n\nLIMITATIONS:\nThere is a minimal discussion on limitations.\n* It seems like a limitation that the MOE is evaluated on simple and small pre-training volumes (6 datasets from PDEBench). I would expect that evaluations on larger volumes such as The Well (https://polymathic-ai.org/the_well/datasets_overview/) would show up more interesting ways the MOE could benefit the training or show some instabilities with the large data distributions. Some discussion on this would be useful.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper trains a mixture of experts model for surrogate PDE models. Since different PDEs have different types of dynamics, initial conditions as well as boundary conditions, the authors show that training a mixture of experts model where a router routes different pdes to different experts (along with having two shared experts). The authors show that their model with 90M active parameters improves upon the zero-shot performance of baselines, and show that different experts do end up taking in PDEs of similar form, and therefore the type of the PDE data can potentially be inferred from the expert that the model ends up going to. \n\nThe architecture in this paper is similar to the DPOT architecture with Fourier attention layers with appropriate temporal aggretation layers. For the mixture of experts and to maintain sparsity, the Top-K activations are used. Appropriate regularization is applied to ensured that not all the examples go through the same tokens (Load balancing objective). \n\nThe authors train their method on 5 datasets from PDEBench and CFDBench., and show that their model consistently outperforms multiphysics baselines like DPOT, as well as single model per dataset type of baselines like FNO.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is easy to read and follow. \n\nI think that this is perhaps the first MOE papers for PDE solvers, something that makes a lot of sense, since as the authors mention, each PDE has different dynamics, and variability through initial and boundary conditions. \n\nThe fact that similar PDEs are routed through same experts, and that this emerges is quite promising, and interesting. \n\nHowever, there are few points that make me question a lot of the contributions of the paper. \n\n- First of all, the multi-physics models that they try are DPOT, and not models like Poseidon and MPP (especially Poseidon since they are currently the best performing multiphysics models).\n- The authors compare their numbers with FNO, but its a very old model and since then a lot of improvements have been suggested on top of FNO (such as FFNO, and others) that could have better performance over FNO.\n- The datasets that the authors train on, like PDE bench are not very difficult, and it is quite easy to get a good models on these datasets, therefore it is unclear how good the architecture is here.\n- The authors mention that they finetune the model for 200 epochs, that is quite a lot, esp since the model has been pre-trained. The authors do show that training from scratch does not work as much (in Table 2), but for single model per dataset baselines like FNO (and its better performing derivatives like FFNO), it would be interesting to see how well it performs if it is given the overall compute budget (of pretraining + finetuning).\n\nQUESTIONS:\nthe authors mention something to the form of “model leads to feature competition” and “knowledge illusion” in lines 95-96. What do they mean by that, and more importantly, can they cite relevant works that coin these terms?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MoE-POT, a Mixture-of-Experts Pre-training Operator Transformer, which modifies the MLP Layer of a DPOT[1] model using a router‐gating network that dynamically selects 4 routed experts from 16 total, alongside 2 shared experts, per layer. Pre-trained on six public PDE datasets with total parameters ranging from 30 M to 0.5 B (activated parameters 17 M–288 M), MoE-POT achieves up to 40% reduction in zero-shot L₂ relative error compared to existing dense DPOT models and infers input PDE types with 98% accuracy via gating patterns. \n\n[1] \"DPOT: Auto-regressive denoising operator transformer for large-scale pde pre-training\" Hao et al, ICML 2024\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n* Interpretability: The router-gating network reliably encodes dataset identity showing that expert selections reflect underlying PDE types\n### Weakness:\n* Novelty: The architecture is just DPOT with the MLP layer replaced with an MoE, which is not enough novelty in the architecture.\n* I don't see a clear win for using an MoE layer in the architecture. The accuracy gain seems to come at a huge cost in terms of memory and training cost.\n* Lack of Theoretical Analysis: The paper claims that router-gating decisions yield dataset classification but does not analyze the mathematical underpinnings or guarantees of this mechanism which is noted in limitations.\n\nQUESTIONS:\n* The paper claims that MoE-POT-M(same number of parameters as DPOT-L) which has a larger number of activated parameters than DPOT-M are equivalent. While inference time reported is similar, the accuracy improvement over DPOT-M seems to come at a significant memory and training cost. Moreover a comparison with DPOT-L is not provided. Can the authors provide a detailed analysis of the training and memory overhead and also add the DPOT-L results?\n* Does MoE effectively handle diverse PDE datasets as claimed by the paper? An ablation study showing performance gains with increasing number of datasets as compared to a normal DPOT would justify this claim.\n* It seems like the MLP layer of other transformer-based PDE foundation models like MPP[2] can also be replaced by an MoE layer resulting in a similar architecture. What makes MoE-POT unique?\n\n[2] \"Multiple Physics Pretraining for Physical Surrogate Models\" McCabe et al. NeurIPS 2024\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}