Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
Authors: Shanda Li
Shinjae Yoo
Yiming Yang
Venue: ICML 2025 poster (2025)
Keywords: Fourier Neural Operators
Hyperparameter transfer
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: We introduce $\mu$Transfer-FNO, a zero-shot hyperparameter transfer method for Fourier Neural Operators that enables efficient hyperparameter tuning by leveraging a novel Maximal Update Parametrization.

Abstract: Fourier Neural Operators (FNOs) offer a principled approach for solving complex partial differential equations (PDEs). However, scaling them to handle more complex PDEs requires increasing the number of Fourier modes, which significantly expands the number of model parameters and makes hyperparameter tuning computationally impractical. To address this, we introduce $\mu$**Transfer-FNO**, a zero-shot hyperparameter transfer technique that enables optimal configurations, tuned on smaller FNOs, to be directly applied to billion-parameter FNOs _without_ additional tuning. Building on the Maximal Update Parametrization ($\mu$P) framework, we mathematically derive a parametrization scheme that facilitates the transfer of optimal hyperparameters across models with different numbers of Fourier modes in FNOs, which is validated through extensive experiments on various PDEs. Our empirical study shows that $\mu$Transfer-FNO reduces computational cost for tuning hyperparameters on large FNOs while maintaining or improving accuracy.

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
Shanda Li
Shinjae Yoo
Yiming Yang'
Abstract
used in the network. is desirable to use large K for mod- eling complex PDE dynamics and capturing fine-grained spatio-temporal structures.
Fourier Neural Operators (FNOs) offer a princi- pled approach for solving complex partial differ- ential equations (PDEs). However, scaling them to handle more complex PDEs requires increas- ing the number of Fourier modes, which signif- icantly expands the number of model parame- ters and makes hyperparameter tuning compu- tationally impractical. To address this, we intro- duce v'TransferFFNO, zero-shot hyperparam- eter transfer technique that enables optimal con- figurations, tuned on smaller FNOs, to be directly applied to billion-parameter FNOs without addi- tional tuning. Building on the Maximal Update Parametrization (nP) framework, we mathemat- ically derive parametrization scheme that fa- cilitates the transfer of optimal hyperparameters across models with different numbers of Fourier modes in FNOs, which is validated through exten- sive experiments on various PDEs. Our empirical study shows that y TTransfer-FNo reduces compu- tational cost for tuning hyperparameters on large FNOs while maintaining or improving accuracy.
However, can be expensive to scale up the number of Fourier modes K in FNO. For a d-dimensional PDE, the parameter count of the kernel integral operator scales as ((K*), leading substantial computational requirements. For instance, even for 4-layer FNO with d = 3, increasing K from 3 to 24 expands the model from 1.7M to 906M parameters, with the Fourier integral operator accounting for over 98% of the total parameters. As the parameter count of FNOs grows, hyperparameter tuning- a computationally expensive process-become infeasible such scales. Standard approaches involve tuning hyperpa- rameters directly large models, which demands extensive computational resources and is often prohibitive in practice. This bottleneck motivates our central research question:
Can we efficiently tune hyperparameters for FNOs with large numbers of Fourier modes?
One straightforward idea is search for the optimal set of hyperparameters on small FNOs and apply it to large ones. However, naively transferring the hyperparameter across scales can lead to sub-optimal performance. On the left panel of Figure we show that the optimal hyperparam- eter changes as model scales under the standard training recipe. In this work, we address this challenge by develop- ing zero-shot hyperparameter transfer technique tailored for FNOs, / TTransferFFNO, The key insight is to properly scale the kernel integral operator parameters as we increase the number Fourier modes K, so that the optimal hy- perparameter configuration remain relatively stable across model sizes (c.g., the right panel of Figure 1). This insight suggests that small proxy model and transfer them to much larger FNOs without additional tuning.
1. Introduction
With the explosive success of deep learning, Fourier Neu- ral Operators (FNOs) (Li et al., 2021) have emerged as a promising paradigm for solving complex PDEs by learning mappings between function spaces. The kernel integral op- erator is the most crucial component in FNO. The operator first transforms the input function into the spectral domain and then updates the features for the low-frequency compo- nents while zeros out the high-frequency ones. The capacity of FNOs directly tied to the number of Fourier modes K
identify good hyperparameters using
'School of Computer Science, Carnegie Mellon Uni- versity, Pittsburgh, PA. USA 'Brookhaven National Labo- ratory, Upton, NY, USA. Correspondence to: Shanda Li <shanda.ees. cou.cddu
Our approach builds on the Maximal Update Parametriza- tion (nP) framework (Yang & Hu, 2021). Informally, parametrization refers to the scaling of initialization vari- ances and learning rates of model parameters when the
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Detailed model configurations can be found Section 4.1.

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
Standard Parametrization
LTTranffr-FNO (Ours)
-075
-0.75
K=3 (1.7M parameters)
-1.00
-1.00
-1.25
-1.2
K=6 (14M parameters)
6 oo (loss)
-1.50
e 2 (loss)
-1.50
K=12 (113M parameters)
-1775
-1775
K=18 (382M parameters)
-2.00
-2.00
K=24 (906M parameters)
-2.25
-2225
-2.50
-2.5
-275
-275
-3.00
-3.00
-4.0
3.5
-3.0
-2.5
-2.0
-4.0
33.5
-3.0
-2.5 -2.0
10910 (learning rate)
1og10 (learning rate)
Figure/ 1. Loss values of FNO-3D on the incompressible Navier-Stokes Equation, with varying numbers of Fourier modes K and different learning rates. The left and right panels correspond to models trained with standard techniques and j'Transfer-FNo respectively, The star on each curve marks the optimal learning rate which leads to the lowest loss value. Under standard parametrization, the optimal learning rates shifts as model size increases; while pTransfer ensures stable optimal learning rate when the model size scales from 1.7M to near IB.
model size increases. The Maximal Update Parametrization (uP) is unique parametrization with many desirable theo- retical and empirical properties which can be rigorously de- fined and mathematically derived. In particular, prior works show that μP enables hyperparameter transfer under model width/depth scaling for MLP and Transformers (Yang et al., 2022; 2024). However, whether μP for neural operators exists and whether can empirically enable hyperparameter transfer remain unknown. We derive the first μP formula- tion for FNO under the scaling of the Fourier mode size K. This extension is technically non-tr non-trivial due to the unique nature of FNOs: (1) Unlike MLPs and Transformers that process fimte-dimensional vectors/sequences, FNOs operate on continuous functions and is agnostic to the discretization; (2) The working mechanism of the kernel integral operator is significantly different from and more complicated than the common deep learning modules such as linear layers or embedding weights analyzed in prior works. Our analysis leads to the following result:
rithm 1), efficient hyperparameter tuning strategy for large-scale FNOs. We conduct extensive experiments to validate our theoretical results and demonstrate the effec- tiveness of pTransfer. We study three important PDE prob- lems: Burgers' Equation, Darcy Flow, and Navier-Stokes Equation. We empirically show that pTransfer preserves the optimal learning rate, training batch size, and optimizer configuration across model scales, and can be applied to both the vanilla FNO training and the advanced Physics Informed Neural Operators (Li et al., 2024). Specifically, on Aavier-Stokes Equations, f yTTranfferFFN successfully scales to FNOs with nearly 1B parameters and achieves lower test error than the baseline using only 0.30x training compute.
In summary, our contributions in this work are threefold:
We are the first to derive the Maximal Update Parametrization (μPP) theory for FNOs by employing new techniques for analyzing FNOs.
Theorem 1.1 (Informal version of Theorem 3.5). The uP for FNO scales the initialization variances of the kernel integral parameters by θ (Nlog K) and their learning rates by θ (aalok)) where and K denotes the PDE dimen- sionality and number of Fourier modes, respectively.
Based on our derived scaling rates in jP, we introduce the pTransfe-FFNO algorithm, for zero-shot hyper- parameter transfer in FNO.
We experimentally validate our theoretical claims on various PDFs, different training algorithms, and differ- ent hyper-parameters, showing the robustness of the theory. We also demonstrate that ja y'TransferFFNO can significantly reduce computational costs while main- taining improving accuracy in practice.
As can be seen in the above, the scaling rate is drastically different from all the existing results (e.g., 0(m-1) for width scaling and (I,21 for depth scaling), highlighting the uniqueness of the kernel integral operator and enriching the μu framework. Technically, this is because existing jP derivations generally boil down to analyzing the behavior of averages of random variables, while in our deviation, the maximum of Kd (sub-)Gaussian random variables emerges, which naturally leads to a Vdlog K term (Vershynin, 2018). This theoretical result leads to 1o y/Tranffe-FNNo (Algo-
2. Preliminaries
In this section, we introduce the fundamental concepts of parametric Partial Differential Equations (PDEs), Fourier Neural Operators (FNOs), and p/rranffer.
2

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
2.1. Parametric Partial Differential Equations
In practice, Q is typically rectangular region in R and discretized with N₁ x N points, where min N₂ > K.
Many problems in science and engineering can be modeled as Partial Differential Equations (PDEs), which describe how physical quantities evolve over time and space. Let Ω c R be a compact domain, A and 1/ be the parameter space and solution space which are typically infiniteddimensional function spaces. A general formulation of parametric PDE is given by:
1535d The input is represented as tensor in RBNxx."XXNN We assume this in our theoretical analysis.
2.3. Maximal Update Parametrization and j y'Transfer
Yang & Hu (2021) study the feature learning in over- parametrized networks by considering a rich class of net- work parametrizations, namely the abe- ode-parametrization,, and the infinite-width limits. They show that there exists a unique parametrization which enables maximal feature learning certain sense by scaling the initialization vari- ances and learning rates for model parameters based on theoretical principles. This parametrization scheme is re- ferred as the Maximal Update Parametrization (μP).
Lau(z)= = O, x E Q C R4, BLu(x)
(1)
02,
where La is a differential operator, Ba is a boundary con- dition operator (Kovachki et al., 2023; Saad et al., 2023). Solving the parametric PDE involves finding a solution op- erator 9: A + N whose output u E l satisfies both the governing equation and boundary conditions for the given parameter function a.
Later, Yang et al. (2022) propose Transsfr as a novel tech- nique for zero-shot hyperparameter transfer across model scales, which built upon the jP. The key insight that the training dynamics and hence the parameter landscape across remain consistent the model size scales under j.P. This consistency allows for the optimal hyperparameters iden- tified in a smaller model to be directly applied to larger target model without additional tuning.
Lu et al. (2021); Kovachki et al. (2023) introduce the idea of learning the solution operator 9 with neural network Go- Given training data {((()) (+) J operator learning aims to approximate the solution operator via standard supervised learning.
2.2. Fourier Neural Operators
Our work studies jP and μt Transfer for scaling up the num- ber of Fourier modes K in FNO since it contributes to the major computational cost of the model and is under- explored in existing relevant research. We formalize the terminologies and conduct rigorous derivation in Section 3.
Fourier Neural Operator (FNO) is the most widely used deep learning model for operator learning (Li et al., 2021). Unlike most other neural networks which model mappings between finite-dimensional spaces, FNO operates in infinite- dimensional function spaces as defined below:
3. Theory
Go = Qo (WL + KL) o ..... o ( + K1) (2)
In this section, we present our theoretical result that identi- fies the correct scaling rate for scaling up the kernel integral operator in FNO.
In Equation (2). P and Q are pointwise linear layers that encode the lower dimension function into higher dimensions and decode the higher dimension function back to the lower dimensions, respectively. & R + R is a point-wise activa- tion function. W. Rm m is a point-wise linear mapping (f E [L] where m denotes the hidden dimensionality. Ke is the kernel integral operator using the Fourier transform, defined as
Let [L]] (1..... L}. Define the pre- activated and activated hidden features as
Notations.
ho = Pu;
we = (We + K2) he-1, be= x(we) (L E [4]),
where is the input. Further denote by we, and he the features after i training iterations.
(Ko))()= F-1 R (FU)
Vx E 2
(3)
where v Ω + Imm is the input function to the operator, F and F-1 are the d-dimensional Fourier transform and its in- verse, Tx truncates the input to the lowest K Fourier modes in each dimension, and R E R xmxm is the parameter tensor which multiplies with the Fourier feature vector for each of the Kd frequency modes. Typically, K is treated as a tunable hyperparameter of FNO. Using a larger K can make the network more expressive.
3.1. Problem setup
We are interested in identifying the Maximal Update Parametrization (ạPP) for FNO under the scaling of the num- ber of Fourier modes K. The hidden dimensionality m and the model depth L are considered fixed in our work, since the scaling of these two factor has been rigorously studied in prior works (Yang & Hu, 2021; Yang et al., 2024).
3

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
We extends and generalizes the aba-earamtttizationn to the parameter tensor R in the kernel integral operator of the Fourier Neural Operator (FNO).
of MLP different when using SGD/Adam the optimizer. In work, we focus on the Adam optimizer (Kingma & Ba, 2015) because it is the de facto choice for training FNO and demonstrates strong empirical performances.
Definition 3.1 (abe-parameetrization for FNO). The general- ized abe- dece-parametrizationn for FNO is specified by mappings a, b, C:NN + R+ such that
As shown by Yang et al. (2022), jaP of is scale-invariant to the constant shift
(a) The tensor R in the kernel integral operator (Equa- tion (3)) is parametrized as R o(K)r for actual trainable parameter r where K denotes the number of truncated Fourier modes;
b,)b,c) - a bui, cá
for any 3 N - RR with the Adam optimizer. There- fore, we assume a(K) = in our subsequence analysis without the loss of generality. This assumption simplifies the exposition since R itself becomes the actual trainable parameter.
(b) Each entry of the trainable parameter r is initialized from (0, 6(K))) independently;
(c) The Adam learning rate for optimizing scaled as 17 c(K) nm where 7h is the "master" learning rate independent of K.
3.2. Deriving μP for FNO
In this subsection, we state our main theoretical result on the μP for FNO. We begin with formulating the necessary assumptions:
We note that the original definition was restricted K-* type of scaling, while we will see that the generalization here will indeed facilitate our analysis.
Assumption 3.3 (Activation functions). We assume the activation function á is tanh or a-GELU.
The standard parametrization for FNO. Following Yang et al. (2022), we refer to the commonly-used parametrization in the soen-source neural operator li- brary² as the standard parumetrization. Under standard parametrization, a(K) = 1, b(K) = m = @(1), and c(K) 1. As shown in Figure 1, the standard parametriza- tion does not preserve the optimal hyperparameter when using different numbers of Fourier modes, so one cannot reliably transfer the optimal training configuration of small models to large models. More in-depth empirical analysis is presented in Section 4.
A Assumption 3.3 standard in theoretical works in μP (Yang & Hu, 2021; Yang et al., 2022; 2024; Ishikawa & Karakida, 2024) and realistic in practice.
Assumption 3.4 (Sub-Gaussian updates). In the t-th iter- ation, denote by g. g. 1g=xxmxx the update tensor returned by the optimizer before multiplying with the learn- ing rate. We assume that the entries of (Re) are inde- pendent
random variables conditioning on the model parameters in the (t - 1)-th iteration for some constant C.
Our goal is to find the Maximal Update Parametrization for the kernel integral operator from the parametrization class in Definition 3.1. Maximal Update Parametrization is defined follows:
Assumption 3.4 states that the tails of the entries in 9t R2)) decays relatively fast. More discussions can be found in Section 3.3
Definition 3.2 (Maximal Update Parametrization for FNO).
With the definitions and assumptions stated above, we are ready to state our main theoretical result:
The Maximal Update Parametrization (nP) for FNO is abc- prr-aaaamttriaioonn that satisfies the following properties for any é E [L] with high probability:
Theorem 3.5 (Main theoretical result). Under Assump- lions 3.3 and 3.4, the following ale-paraaettrization is Maximal Update Parametrization (μP) of FNO with the Adam optimizer:
Stability at initialization: For a given input function v, the hidden features he.o, weo0 ((1).
a(K) = - ( (4)
=
Feature learning in every layer: For any / E N* the feature update A = W1,1 W1.0 = O(1).
where d is the PDE dimensionality.
We point out that the Maximal Update Parametrization is optimizer-dependent: The Maximal Update Parametrization
The detailed proof can be found in Appendix A. The major technique in our proof is to derive the spectral norms of K. at initialization and its update. Our calculation shows that the spectral norms the maximum absolute value of ((Kd)
"https://github www.tth.....//uuutteteeeee neuraloperator

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
(sub-) Sub-)GGusssian random variables. The scaling functions in Equation (4) control the scales of these variables and hence lead to (1) spectral norms, which in turn realizes the conditions in Definition 3.2.
operator. While this is not standard in existing analysis, it is natural for the Adam optimizer where entry-wise nor- malized gradient momentum is used in the update and the normalization controls the scale of the update. Furthermore, the sub- Gaussian condition can be explicitly enforced in practice through element-wise gradient clipping (Pascanu et al., 2013), which bounds the updates and thereby ensures their Gaussaasian
Our proof enriches the technique for Jul deviations. In prior works (Yang & Hu, 2021; Yang et al., 2022), yP deviations mainly rely on the central limit theorem and the law of large numbers to characterize the average of random variables. By contrast, in our study, the unique structure of FNO and the kernel integral operator leads to the analysis of the max- imum of random variables and consequently an interesting scaling function.
3.4. The f TTraanfer-FNN algorithm
As shown by Yang et al. (2022), the hyperparameter land- scape across neural networks of different sizes (NOs with different numbers of Fourier modes) is reasonably sta- ble if parametrized according to j.P. This property enables us to probe the hyperparameter landscape with a small fixed- sized proxy model and identify the optimal configuration. The optimal configuration will also be (near) optimal for the large model under JP, thus circumventing the direct and computationally intensive hyperparameter tuning of large models. We present this procedure in Algorithm 1.
FNOs
We also note that the scaling function does not depend on .Nj, ....... i... the discretization of the function domain Ω. This property is appealing in practice because it makes sure that μP does not change the resolution-agnosti nature of FNOs.
In the subsequent sections, we show that the theoretical result is practically useful by presenting the μTTransfer-FW algorithm and empirical study.
Algorithm 1 JrTransfer-FNO
3.3. Discussions on the result
Require: Hyper-parameter search space E. FNO training algorithm train(g, K)) which returns the final loss given hyperparameter & and number of Fourier modes K.
In this section, we make a few remarks on the theoretical finding.
Numbers of Fourier modes for target and small proxy models K** and K Kproyy-
Regarding the connection to Transformers. Recent re- search has shown that the Transformer attention module and its variants (c.g.. Continuum Attention) are closely related to operator learning (Kovachki et al., 2023; Rahman al., 2024; Calvello et al., 2024). While the Maximal Update Parametrization (μP) for FNO has been established by Yang et al. (2022), we note that the result is not readily applicable to the Fourier Integral Operator.
Ensure: Efficiently train target FNO using (near) optimal configurations.
I: L PARAMETER SWEEPING ON SMALL MODELS.
2: g" + argmingez train(5, Kproyy-
3: PARAMETER RESCALING.
Specifically, for attention module and its variants (e.g.. Con- tinuum Attention), the parametrization closely resembles vanilla attention, and its size is controlled by the hidden dimensionality (i.e., the model "width"). Hence existing re- sults on scaling up model "width" are applicable. In contrast, the Fourier Integral Operator is parametrized in a uniquely different way- - by modeling the Fourier Transform of a kernel function with K Fourier modes. This design dif- fers from all modules which have been explicitly studied in existing literature. Theorem 3.5 shows that its scaling rate is also different from that of other model components. That being said, in Fourier Integral Operators, the notion of model "width" still exists (corresponding to m in Section 2), and existing width scaling results may apply to it.
log K Kproxy log K*
4: Sleaming rate for R +
Slearning rate for R.
log K proxy
5: Sii of R f-
log K*
Sinit, of R
6: D TARGET MODEL TRAINING.
7: train(s",
4. Experiments
In this section, we empirically validate our theory and study the effectiveness of the generality, efficiency, and effective- ness μ FTTrnnffer-FNO algorithm (Algorithm 1). In particular, we aim at answering the following questions through exper- iments:
Regarding Assumption 3.4 and proactical applications of Theorem 3.5. Assumption 3.4 requires sub- GabGsassi updates to the parameter tensor R in the kernel integral
Does the juP for FNO stated in Theorem 3.5 preserve
5

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
the optimal learning rate?
We train FNO-2D to learn the mapping from the PDE co- efficient to the solution, 9 : a + u. The data resolution is 421 x 421 and downsampled 61 x 61 during training. We train FNO-2D with 4 layers, 64 hidden dimensions, the GELU activation function, and varying numbers of Fourier modes K. We use 1000/500 samples for training/evaluation. The model is trained with a batch size of 20 for 300 epochs. The learning rate halves the 100th, 150th, and 200th epoch by default. We also explore other batch sizes and optimizer hyperparameters in Section 4.4.
Is the ATransfer-FNe algorithm generalizable to more advanced FNO training techniques, e.g., Physics In- formed Neural Operator (PINO) (Li et al., 2024)?
Is the FNTransfer-FNO algorithm applicable to trans- fer hyperparameters across model scales beyond the learning rate?
How much computation cost does the y JTrrnnfer-FNN algorithm save compare to the naive hyperparameter tuning strategy?
Navier-Stokes Equation. We evaluate FNO-3D on the incompressible Navier-Stokes equations for a viscous, in- compressible fluid vorticity form on a torus in the 2D space:
We first detail the problem setup and training configura- tions in our empirical study, and then presents experimen- tal results to answer the above research questions in the subsequent subsections. We release our code at https: / //githull www.thu..comiitt.. com/Lithhumil www.uunntaaaaaaaeaaaeaae muTrans fer FNO.
d,w + u. Vw = AeAw + ʃ
x E 2, E (0,
V v.u(x,t) =
E 0 2, E [0,
w(z, = wo(r
x E (0,2m]
4.1. Problem setup and training configurations
Burgers' Equation. We use the ID Burgers' Equation as the testbed for the FNO-ID model. The equation takes the following form:
In this equation, w is the vorticity field, u is the velocity field, Re is the Reynolds number characterizing fluid vis- cosity, and f is the forcing term. The equations use periodic boundary conditions. The data is generated spatial grids and downsampled to 64 x 64 resolution. The temporal resolution is 512 steps per unit time, and we set the time interval [0, T] to [0, 0.125 We set the Reynolds number Re to 500 and the forcing term f to cos((2)):
(ā.u(z,t) t) + a,( t)2 = vx,) (u(z, (x)
E (0, 1] x E 1)
512 x 512
where un is the initial condition and v is the viscosity co- efficient. We assume periodic boundary conditions. In the above equation, is the spatial variable, while t is the tem- poral variable, which is slightly different from the notation in Equation (1) where x is denotes as the spatiotemporal variable in the PDE. We hope this abuse of notations does not confuse the readers.
We train FNO-3D to do spatial-temporal convolution jointly following Li et al. (2021; 2024); George et al. (2024). The model contains 4 layers, 64 hidden dimensions, the GELU activation function, and varying numbers of Fourier modes K. We use 800/200 samples for training/evaluation. The model is trained with a batch size of 2 for 50000 iterations. The learning rate halves at the 20000th and 40000th itera- tion. For y/raasfer-FNoO we apply element-wiss gradient clipping the kernel integral operator module and the clip value is set to 0.01.
We aim to learn the mapping from the initial condition to the solution, g uo(-) + 1). The spatial domain is discretized into 8192 grids and downsampled to 1024 during training. The viscosity coefficient is set to 0.1. We train FNO-ID with 4 layers, 64 hidden dimensions, the GELU activation function, and varying numbers of Fourier modes K. We use 800/200 samples for training/evaluation. The model is trained with a batch size of 20 for 750 epochs, and the learning rate halves every 50 epochs. We use Adam (Kingma& & Ba, 2015) the optimizer and set its (31, 32) to (0.9, 0.999) in this and all the subsequent experiments, unless otherwise specified.
The experiments for FNO-ID ID and FNO-2D are run on a NVIDIA GeForce RTX 2080 Ti GPU with 1 11GB memory. The experiments for FNO-3D are run on a NVIDIA RTX A6000 GPU with 50GB memory. Our code is built upon the open source library for Physics-informed Neural Operator* using PyTorch (Paszke et al., 2019).
4.2. Identifying optimal learning rates under scaling
Darcy Flow. We test FNO-2D models on the steady-state of the 2D Darcy Flow on the unit square. It is a second-order linear elliptic PDE:
We first examine how the learning rate landscape changes as we scale up the number of Fourier modes K on the three PDE problems. Figures 1, 2a and 2b show the loss values
-V (a(x)Vu(z)) =1 [ux)= 0
E 1)2 x E a(0 1)2
Pntopa://github cot/
6

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
Standard Parametrization
Standard Parametrization
Standard Parametrization
-1.5
K=3
-1.8
K=3
0.4
K=3
K
K=6
0.2
K=6
(09holloss)
-2.0
K=12
(olool
-2.0
K=12
l0gtolloss)
0.0
K=12
K=24
K=24
K=24
-2.5
-2.2
-0.4
-3.0
-2.4
-0.8
-1.0
-3.5
-2.6
-40
-30 log20(learning rate)
335
-2.
-2.0
-1.5
-3.6
-3.2
-2.8
-2,4
-2.0
-4.0
-...-33. -00 2.0-1,5-11 1.0
logiollearning rate)
l0g2ollearning rate)
UTTansfer-FNo
uTransfer-FNO
uTransfer-PINO
-15
K=3
-1.8
K=3
0.4
K=3
K=6
K=6
K=6
-2.0
K=12
log1o(loss]
-2.0
K=12
(og1o(loss)
0.0
K=12
10g1ellose
K=24
K=24
-0.2
K=24
-2.5
-2.2
-0.4
-0.6
-3.0
-2.4
-0.8
-1.0
-3.5
-2.6
-4.0 -3.5
-3,0 lognollearning rate)
-2.5 -2.0 -1.5
-3.6
-3.2
-2.8
-2.4 -2.0
-4.0
-4.0-3.5-3 13.5-35-3.0-2.11 -2.0 -2.0-1,5-11.
logno(learning rate)
log1o(learning rate)
(a) FNO-ID on Burgers' Equation.
(h) FNO-2D Darcy Flow.
(c) PINO-2D Darcy Flow.
Figure 2. I Loss values of FNOw/PINOs with varying numbers of Fourier modes K and different learning rates. The star each curve marks the optimal learning rate which leads to the lowest loss value. We observe that the optimal learning rate more consistent under Transferr, compared to the standard parametrization.
under both standard parametrization and y/Transfer-FNo trained with different learning rates.
Furthermore, we observe that under y'Transfer-FNO, the loss curves exhibit more consistent shapes across different K values, suggesting that the overall optimization landscape becomes more uniform.
Under standard parametrization (the left panel in Figure and the top row in Figure 2), we consistently observe clear shift the optimal learning rate as increases. For example, for FNO-3D on the the incompressible Navier- Stokes Equation, the optimal learning rate (marked by stars) shifts from 1.8 x 10 when K 3 to 7.4 10-4 when K 24. This shift indicates that naively transferring hy- perparameters tuned small models to larger ones leads to suboptimal training configurations.
4.3. Applications to Physics- Physics-Informed Informed Neural Operators
All the results presented in Section 4.2 are based on stan- dard supervised training of FNO. In this subsection, we fur- ther evaluate the effectiveness of our approach on Physics- Informed Neural Operators (PINO) which incorporates PDE constraints into the training objective (Li et al., 2024). Note that PINO does not change the architectural design of FNO. Hence the parametrization scheme developed in The- orem 3.5 is still valid. We refer to approach in this setting //Transfer-PINO, We use the same training recipe as described in Section 4.2, and set the ratio between the data loss and
In contrast, when using FNo (the right panel in Figure and the bottom row Figure 2), the optimal learn- ing rates remain remarkably stable across different values of K. For example, for FNO-2D on the the incompress- ible Navier-Stokes Equation, the optimal learning rate align approximately 4.2 x 10-3, This stability validates our argument in Theorem 3.5 and Section 3.4. That being said, the optimal learning rate for certain model size may some- times deviate from the expected value, e.g., on the curves of K in Figure (right), K = Figure 2a (bottom), and K in Figure 2b (bottom). We note that this instability typically occurs when K is very small and attribute this to the randomness during training.
informed loss to 5 1
We present experimental results of the standard parametriza- tion and /TransferPIINO on the Darcy flow problem in Figure 2c. The results share similar trends with those in Section 4.2. Under standard parametrization. PINO exhibits sensitivity to the learning rate K increases. The opti- mal learning rate gradually decreases when K increase. In contrast, / /'rransfer successfully stabilizes the learning rate landscape across different values for PINO, with the op-

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
Standard Param.
FFoO
Standard Param.
Transfer-FNNO
-1.6
K=3
-1.6
-2.1
K=3
-2.1
10g1o(lo55)
-1.8
K-6
(0g1o(1o5S)
-1.8
K-6
K=12
l0g1o(loss)
-2.2
K-6
10910(Ioss)
-2.2
K-6
K=12
-2.0
K-24
-2.0
K-24
-2.3
K-24
-2.
K=24
-2.2
-222
-2.4
-2.4
-2.4
-25
-2.5
-2.6
-2.6
0.98
0.99
0.999
-2.6
0.98
0.99
0.999
10
70
50
100
200
20
50
100
200
0
Batch size
Batch size
Adam B2
Adam B2
(a) Batch size.
(b) Optimizer hyperparameter.
Figure 3. Loss values of FNO-2D on the Darey Flow problem with different hyperparameters and varying numbers of Fourier modes K. The star on each curve marks the optimal setting which leads to lowest loss value. We observe that the optimal hyperparameter more consistent under μ Trannsfer, compared to the standard parametrization.
timal learning rates aligning approximately 5.6 x 10-3 This result is noteworthy given that PINO's training dy- namics are typically more complex due to the additional physics-informed loss terms. The consistent performance across both standard FNO and PINO shows the generality of Theorem 3.5, independent of the specific training objective variations.
Trransfer-FNO leads to lower I.2 relative test on the test set. This is natural because -Transfer-FNO sweeps hyperparameters on small models, hence can use a larger hyperparameter search space. Second, //Transfer-FNG sur- passes the test performance of the naive hyperparameter tuning method on the Navier-Stokes Equation while only uses 0.30 of the total training compute, which clearly demonstrates the benefit of tuning small proxy models to identify the hyperparameter choice. Finally, we note that j (TTannfer-FNOO is slower than the naive approach on the Darcy Flow problem because the training cost gap between small models and large models is not extremely large in this setting. practice, μ y/Transfer- FNO holds stronger potential in the large-scale FNO training settings, e.g., when scaling to paraamerr sizes.
4.4. Applications more hyperparameters
The experiments in Sections 4.2 and 4.3 all focus on the learning rate. In this subsection, we investigate whether FNO's benefits extend to other hyperparameters beyond learning rates. Specifically, we consider tuning the batch size and Adam's 32 parameter for FNO-2D on the Darcy flow problem.
Our experimental results are shown in Figure For batch size (Figure 3a), standard parametrization exhibits varying optimal values across different K: larger models (K~ 24) require larger batch size, while smaller models perform better with smaller batches. In contrast, under y'Transfer- FNO, the optimal batch size is consistently 20 regardless of model size. Similarly, for Adam's 's 33 parameter (Figure 3b), standard parametrization shows shifting optimal values as K increases. With j.Transfe-FFNO, the optimal B2 aligns at 0.98 across all model sizes. This consistency further vali- dates that pTrranffr-FNN stabilizes the overall optimization dynamics. These results demonstrate that μ Transfer-FNo enables reliable hyperparameter transfer across model scales for multiple optinization-related hyperparameters, making it a practical tool for efficient large-scale FNO training.
5. Related Works
5.1. Neural PDE solvers
with the growth of available data and computational re- sources, there have been growing interests in developing neural network approaches to partial differential equation solving (Han al., 2018; Khoo et al., 2021: Kochkov et al., 2021; Wang et al., 2022). One stream of works leverages neural networks ples include (Sirignano & Spiliopoulos, 2018; Raissi et al., 2019; Zang et al., 2020; He et al., 2023). Another active re- search direction is operator learning, where neural networks serve as an ansatz of solution operator (Long et al., 2018; 2019; Lu al., 2021; Li al., 2021; Cao, 2021; Kovachki et al., 2023; Wang al., 2024). few works present ap- proaches to combining the strengths of these two paradigms (Huang et al., 2022; Li et al., 2024).
ansatz of solutions, notable exam-
4.5. Practical effectiveness of TTTransfer-FNO
To demonstrate the effectiveness of -TTranser-FNOO in prac- tice, we compare model performances of directly tuning the large model and applying the T/ransfer-FNO algorithm. The results are reported in Table First, we note that
Our work falls into the operator learning category and specif- ically focuses on tuning large-scale FNO. There are several prior works studying similar topic: George et al. (2024) in- troduce Incremental Fourier Neural Operator (iFNO) which
8

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
Table 1. Test relative error comparisons between directly tuning the large model and JCTransfer FNNs.
Model
FNO-2D
FNO-3D
Problem
Darey Flow
Navier-Stokes Equation
Metric
L relative error
Total training cost
L relative enror
Total training cost
Tune the full model
1.25%
1x 1.38x
5.69%
1x 0.30
TTrnnsfe-FFNO (ours)
1.22%
5.34%
progressively increases both the number of Fourier modes and the training data resolution to enhance training effi- ciency. Alkin et al. (2024) propose Universal Physics Trans- formers (UPTs) as an efficient and unified approach to a wide range of spatio-temporal problems by propagating dynamics in the latent space. While this paper presents im- proved model architecture for operator learning, our work studies efficient hyperparameter searching for the most commonly-used FNO architecture.
cipled scaling laws and demonstrated impressive empirical success across wide range of domains, including board games (Jones, 2021), language modeling (Kaplan et al., 2020; Hoffmann et al., 2022: OpenAI, 2023), image model- ing (Henighan et al., 2020; Yu et al., 2022; Peebles & Xie, 2023), mathematical reasoning (Snell et al., 2024; Wu et al.,
2024), and very recently, PDE solving (Fan et al., 2025; Li et al., 2025). We believe this direction holds strong promise in neural PDE solvers and that our work contributes to this emerging area.
5.2. Theoretical analysis on FNOs
6. Conclusions and Limitations
Theoretical analysis of FNOs provides important research insights into model designs and upplications. Kovachki et al. (2021) prove the universal approximation theorem for FNOs. De Ryck& & Mishra (2022) prove the approximation error bounds on physics-informed operator learning. Koshizuka et al. (2024) delve into the expressivity and trainability of FNOs using mean-field theory. Kim & Kang (2024) study the generalization capabilities of FNOs by analyzing their Rademacher complexity. Le & Dik (2024) systematically present series of mathematical analysis of neural opera- tor. Our work formulates and derives the Maximal Update Parametrization of FNO and presents the application to zero- shot hyperparameter transfer.
Conclusions. We introduce //Transfer-FNO, a principled method for zero-shot hyperparameter transfer in FNO. We derive the μP for FNO theoretically, and our proof offers new technical insights to the study of j.P. On a wide range of PDEs, we experimentally demonstrate that the parametriza- tion scheme preserves the optimal hyperparameters under various setting. Our results show that TTransfer-FNOO is pooreticalll-princpphh and can serve as a practical and efficient strategy for tuning neural PDE solvers in billion- parameter scales.
Limitations. Although FNO is widely used in neural net- work methods for PDE solving, there are many other neural operator variants including Multipole Graph Neural Oper- ator (Li et al., 2020). DeepONets (Lu et al., 2021), Multi- wavelet NO (Gupta et al., 2021). and Transformer based foundation models (Ye et al., 2024). Our results are ded- icated to FNO and extending the results to more general neural operator variants can be avenues for future research.
5.3. j.P, j Transfer, and Scaling in deep learning
Our work builds upon the μP (Yang & Hu, 2021) and TTransfer (Yang et al., 2022) framework. The framework initially introduced for analyzing the limits under model width scaling and gradient descent training, and later ex- tended to depth scaling (Yang et al., 2024; Bordelon al., 2024; Chen al., 2024) and sccond-order optimization (Ishikawa & Karakida, 2024). Dey et al. (2023); Lin- gle (2024) present empirical applications of μ Transfer on Transformer-base based large language model training. Blake et al. (2024) incorporate Unit Scaling into JPP, combining the advantages of hyperparameter transfer and stable training in FP8 precision. Noci et al. (2024) show some properties of the loss landscape are preserved across model sizes under the right scaling of the architecture. Our work is, to the best of our knowledge, the first analysis on μP and p TTransfer for PDE solving and scaling up the FNO architecture.
Acknowledgements
The authors would like to thank the constructive suggestions from Chuwei Wang (Caltech) and the anonymous reviewers.
This material is based upon work supported in part by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research under Contract DE-ACC2-06CH1 1357. This research uses resources of the Oak Ridge Leadership Computing Facility and NERSC, which are a DOE Office of Science User Facility supported under Contract No. DE-AC02-0SCHI 1231 ERCAP0034238.
and under Contract using NERSC award ALCC-
More broadly, scaling up deep learning models is currently an active research topic. Existing research has derived prin-
9

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
Impact Statement
optimal language models trained the cerebras wafer- scale cluster. arXiv preprint ar arXiv:2304.0322 2023.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Fan, Z., Sun, Y., Yang, S., and Lu, Y. Physics-informet inference time scaling via simulation caaliate scien- tific machine learning, arXiv preprint arXiv:2504.1617 2025.
References
George, R. J., Zhao, J., Kossaifi, J., Li, Z., and Anand- kumar, A. Incremental spatial and spectral learning of neural operators for solving large-scale PDEs. Transac- tions on Machine Learning Research, 2024, ISSN 2835- 8856. URL https: /wtp://veneeeeeeeeeeeeeeeeeeeeee net /tt/uoorm id=x I 6c
Alkin, B., Fürst, A., Schmid, S. L.,, Gruber, L.,, Holzleitner, M., and Brandstetter, J. Universal physics transform- ers: A framework for efficiently scaling neural opera- tors. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: //openreview. net/ forum? d=ooxi NX5KRm.
Gupta, G., Xiao, X., and Bogdan, Multiwavelet- based operator learning for differential equations. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P. and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, vol- ume 34, pp. 24048-24062. Curran Associates, Inc. 2021. URL ccr/ppper_fileespppprr/////////1 Paper. c9e5c22bb22244488e110066666166/0 pdf.
Blake, C., Eichenberg, C.,, Dean, J., Balles, I.., Prince, L. Y., Deiseroth, B., Cruz-Salinas, A. F., Luschi, C., Weinbach, S., and Orr, D. u-up: The unit-scaled maximal update parametrization. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalabil- ity, and Resource Optimization (WANT@ ICML 2024), 2024. URL https: http//pprrreeieeeeeem nwt. www.w/wr.etereerre/// forum? id 10-44NKA 4NKKZ21n5.
Bordelon, B., Noci, L., Li, M. B., Hanin, B., and Pehle- van, Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit. In The Twelfth International Conference on Learning Representations, 2024. URL https: http//ppnweeeeeeeeeeeeeeeeeeeeep forruru id=K2JehvRKGD.
Han, J., Jentzen, A., and Weinan, E. Solving high- dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sci- ences, 115(3))8505-88100 2018
He, D., Li, S., Shi, W., Gao, X., Zhang, J., Bian, J., Wang, L., and Liu, T.-Y. Learning mnysics-informed neural networks without stacked pack-propagation. In Ruiz, F,, Dy, J., and van de Meent, J.-W. (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. http///ppeeeee...eeeeee ml mlr.press/ v206 2206//h2330 hh///he2aaaa 2a. htm1.
Calvello, E., Kovachki, N. B., Levine, M. E., and Stuart, A. M. Continuum attention for neural operators. arXiv preprint ar arXiv:2406.0644 06486, 2024.
Cao, S. Choose a transformer:
Fourier or galerkin. In Beygelzimer, A., Dauphin, Y., Liang. P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, 2021. URL hitps: /tttp::://ppeeeeee http://pppeeeeeeeeeeeeee forum? www..ddsssssh....ns.-...
PMLR, 25-27 Apr
2023. URL htwps:
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.1470.1 2020.
Chen, W., Wu, J., Wang, Z., and Hanin, B. Principled architecture-aware scaling of hyperparameters. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https:/ http:///oppeeeeeeeeee net// forum? ditmmmm2---nnddeey...
Hoffmann, J.. Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. I.., Hendricks, I.. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arX5v:2003115556 2022.
De Ryck,7 T. and Mishra, S. Generic bounds on the approxi- mation error for physics-informed (and) operator learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho. K., and Oh, A. (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp. Curran Associates, Inc., 2022.
Huang, X., Ye, Z.,, Liu, H., Ji, S., Wang. Z., Yang. K., Li,
Y., Wang, M., Chu, H., Yu, F., et al. Meta-autoodecoderr for solving parametric partial differential equations. Ad- vances in Neural Information Processing Systems, 35: 23426-23438, 2022.
Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J., et Cerebras-gpt: Open compute-
10

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
Ishikawa, S. and Karakida, R. On the parameterization of second-order optimization effective towards the in- finite width. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview.. n wwt.:/rrtooomrm d
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Stuart, A., Bhattacharya, K., and Anandkumar, A. Multipole graph neural operator for parametric partial differential equations. In Larochelle. H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6755-6766. Curran Associates, Inc., 2020. URL https::/prroceedgggg.. neurips.. cc se//aper_ffileeeppper////////// hb22cf96ddcf122223366322222..0.2 Paper. pdf.
Jones, A. L. Scaling scaling laws with board games. arXiv preprint arXiv:2 13, 2021.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08360 2020.
Li, Z.., Kovachki, N. B., Azizzadeneshelli K., liu, B.,, Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Fourier neu- ral operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL https:/ / wpwpeeweeeeeeeeeeeeeepeee net/ forum? i d=c8F 9NOVtmnO.
Khoo, Y., Lu, J., and Ying. L. Solving parametric pde prob- lems with artificial neural networks. European Journal of Applied Mathematics, 32(3):421-435, 2021.
Kim, T. and Kang, M. Bounding the rademacher complexity of fourier neural operators. Machine Learning, 113(5): 2467-2498, 2024.
Li, Z., Zheng, H., Kovachki, N., Jin, D., Chen, H., Liu, B., Arizzadenesheli, K., and Anandkumar, A. Physics- informed neural operator for learning partial differential equations. ACM/IMS Journal of Data Science, 1(3):1-27, 2024.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. URL http:// arxiv. org/ abs/ / 1411.112220 6980.
Lingle, L. A large-scale exploration of j-transfer. arXiv preprint arXiv:2404.05722, 2024.
Kochkov, D., Smith, J. A., Alieva, A., Wang, Q., Brenner, com-
M. P., and Hoyer, S. Machine putational fluid dynamics. Proceedings of the National Academy of Sciences, 11((1))e2111884118 2021.
Long, Z., Lu, Y., Ma, X., and Dong, B. PDE-net: Learning PDEs from data. In Dy, J. and Krause, A. (eds.), Proceed- ings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learn- ing Research, pp. 3208-3216. PMLR, 10-15 Jul 2018. URL https: http////oceeeeeiggeee/ee mlr. prass.pres//vee long18a 8a html.
Koshizuka, T., Fujisawa, M., Tanaka, Y., and Sato, I. Under- standing the expressivity and trainability of fourier neural operator: mean-field perspective. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https: httppss//operreviee..cer / /ttp///eeee..eeeeeeeeeee net//
Long, Z., Lu, Y., and Dong, B. Pde-net 2.0: Learning pdes from data with numeric-symbolic hybrid deep network. Journal of Computational Physicx, 399:108929 2019.
Kovachki, N., Lanthaler, S., and Mishra, S. On universal approximation and error bounds for fourier neural oper- ators. Journal of Machine Learning Research, 22(290): 1-76, 2021.
Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218-22) 2021.
Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Re- search, 24(89): 2023.
Noci, L., Meterez, A., Hofmann, T., and Orvieto, A. Super consistency of neural network landscapes and learning rate transfer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http:://///ppeeiiei/// net/ forum? id=rgwho7INt2.
Le, V.-A. and Dik, M. A mathematical analysis of neural operator behaviors. arXiv preprint arXiv:2410.21491, 2024.
OpenAl. Gpt-4 technical report, 2023.
Li, S., Marwah, T., Shen, J., Sun, W., Risteski, A., Yang, Y., and Talwalkar, A. Codepde: An inference frame- work for Il Imm-eriven pde solver generation. arXiv preprint arXiv:2505.077 08883, 2025.
Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty training recurrent neural networks. In International conference on machine learning, pp. 1310-1315 Pmlr, 2013.

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
Paszke, A., Gross, S., Massa, F,, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, I.,, Bai, J., and Chintala, S. PyTorch: An imperative style, high-performance deep learning library. In Wal- lach, H., Larochelle, H., Beygelzimer, A., d' Alché-Bue, F., Fox, E., and Garnett, R. (eds.), Advances in Neural In- formation Processing Systems 32, pp. Associates, Inc., 2019.
Wang, C., Berner, J., Li, Z., Zhou, D., Wang, J., Bae, J., and Anandkumar, A. Beyond closure models: Learning chaotic-systems via physics-informed neural operators. arXiv preprint arXiv:2408.05177 2024.
Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang. Y. Infer- ence scaling laws: An empirical analysis of compute- optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724 2024.
Curran
Yang, G. and Hu, E. Tensor programs IV: Feature learning in infinite-width neural networks. In International Con- ference on Machine Learning, pp. 11727-11737. PMLR, 2021.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint 26X46:200.03366, 2022.
Rahman, M. A., George, R. J., Elleithy, M., Leibovici, D., Li, Z.. Bonev, B., White, C., Berner, J., Yeh, R. A., Kos- saifi, J., Arizzadenesheli, K., and Anandkumar, A. Pre- training codomain attention neural operators for soly- ing multiphysics PDEs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https::////.......m. http///twnreeeeeeeeeeeee net forum? /tttps://ppenteeveee id=wSpIdUX2YX.
Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs VI: Feature learning in infinite depth neural networks. In The Twelfth International Conference Learning Representations, 2024. URL /tap::://openevveet net / forum? inttormm?iee 1in221dd1111 1ddd=117//ppww....
Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physics- informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Ye, Z., Huang, X., Chen, L., Liu, H., Wang, Z., and Dong, B. Pdeformer: Towards foundation model for one- dimensional partial differential equations. In ICLR 2024 Workshop on A14Diffeventaallquution In Science, 2024.
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang. Z., Va- sudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. Scaling autoregressive models for content-rich text-to-ine image gen- eration. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Saad, N., Gupta, G., Alizadeh, S., and Maddix, D. C. Guiding continuous operator learning through physics- based boundary constraints. In The Eleventh In- ternational Conference on Learning Representations, 2023. URL httph:t/////o. http:///pppeeeeeeeeee id GOES6
Zang, Y., Bao, G., Ye, X., and Zhou, H. Weak adversarial networks for high-dimensional partial differential equa- tions. Journal of Computational Physics, 411:109409, 2020.
Sirignano, J. and Spiliopoulos, Dgm: deep learning al- gorithm for solving partial differential equations. Journal of computational physics, 375:1339-1364, 2018.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.033: 033144 2024.
Vershynin, R. High-dimenssooal probability: An introduc- tion with applications in data science, volume 47, Cam- bridge university press, 2018
Wang, C., Li, S., He, D., and Wang, L. Is L physics- informed loss always suitable for training physics- informed neural network? In Advances in Neural In- formation Processing Systems, 2022.
12

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
A. Omitted theoretical results
A.I. Technical lemmas
We first present a few technical lemma as preparations for the main proof.
Lemma A.1. For any R, K. a linear operator.
Proof. We note that the Fourier transform, the truncation operator, the multiplication with R. and the inverse Fourier transform are all linear operators. K. as the composition of these linear operators, is still linear operator.
To highlight the kernel integral's dependency on the parameter tensor R, we slightly abuse notation and write K. = K.R. Lemma A.2. KR is linear in R.
Proof. The proof is straightforward based on the definition in Equation (3) and the linearty of the inverse Fourier transform.
Lemma A.3 (Feature learning condition). A parametrization of FNO satisfies feature learning in every layer if and only if the followings are both true with high probability
The feature coordinate size hig remain stable, i.e., he. ((1) every E [4] and N.
Kc updates maximally for every E [L] and t i... A1 = ( where A,Ke = Ketp Ke
Proof. We note that P, W1 + K1. ... WL Kz, and Q are all linear operators over the input functions, i.c., linear mappings over the input tensor in practice where functions are discretized. Therefore, the feature learning conditions (Conditions A.I & A.2 in (Ishikawa & Karakida, 2024)) still hold in our setting.
According the conditions, FNO satisfies feature learning in every layer if and only if the following two conditions satisfied with high probability:
Maximal updates: A. Ke)he-1,t O(1);
Maximal initialization: 2oA,h = (())
For the first condition, It is easy note that A W. does scale when K increases. Thus, A. + Ke)he-1,t 0(1) is equivalent to A,Kche-i,. 1,r O(1), which assume to be true (with high probability) in the lemma statement.
For the second condition is casy to check because the feature coordinate size h..0, hL.: = O(1). Thus 4yh = =(1). Also, the spectral norm of 2o is (1) and does not scale with K. Thus Q = (11) in the general case.
Therefore, the assumptions in the lemma statement implies Conditions A.2 (Ishikawa Karakida, 2024), and hence implies feature learning.
13

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
A.2. Proof of Theorem 3.5
In this subsection, we present the complete proof of the main theoretical result, Theorem 3.5.
Proof. We are only interested in scaling up K with m fixed. Thus, without the loss of generality, we assume m 1. The proof can easily be extended to the case with general m.
In the kernel integral operator Ke, let the parameter Rs { RE[K]a where r E RR.
Stability at initialization. We first consider the model behavior initialization and prove the stability condition by induction. Specifically, given hn (1) initialization, it suffices show that for l [7.) We,0 ((1) and hęo0 o(1) if 11-1,0- (1). For ease of notations, we omit the subscript 0 in the subsequent analysis the initialization.
By definition, Wehe-1 + K Obviously, W = (
Regarding Kghh, Lemma A.1 implies that K2 a linear operator. Given the discretized input fe E IR X Kg can be written as matrix as follows:
Ke = F
(5)
where F denotes the N1 x ..... x Na (multidimensional) discrete Fourier transform matrix, the superscript H denotes the conjugate transpose, and R² E R N....N. accounts for the truncation operator and multiplication with R.. The rows and columns are indexed by j E [Ni] x .... [Na]. Specifically, its (j, k-)tth entry is
RU k)
{0r
1 j= E [K]4
Otherwise
Note that F is an orthogonal matrix and R is diagonal matrix, hence Equation (5) is the eigen-decomposition of K,, Therefore, the spectral norm of Ke is
Kell2
max |/2
ke[K]]
Recall that {r/ [e[jja are Kd i.i.d random variables sampled from (0,, b(K))). By standard result in nigh-dimensional probability (Vershynin, 2018), with high probability.
K Kelle = max | = (M(K) K) = O(1). kE[K]"
Also note that hr-1 (1). Thus, Kehe-1 (L)) and hence wy Wehe-1 Kehe-1 @(1). Finally, by Assump- tion 3.3, h2 ((w)) (1). Therefore, show the stability initialization by induction on
Feature learning in every layer. We begin with characterizing the updates of Ke,, in each iteration 4. For | E N, by Lemma A.2,
Kg,++ - -e, R - Re. = R Br. = noe(K) ()))))))))
where g: (R₂) is defined in Assumption 3.4. By Assumption 3.4, using argument similar the above proof of stability initialization, we have
Kr,+1 - K2, O(1),
where Kg, denotes the discretized operator Ke,s-
Now prove feature learning in every layer by verifying the conditions in Lemma A.3:
The feature coordinate size he remain stable, i.e. he e(1) for every é e {0) [ and t N.
We prove by induction L. First, it is easy see the hott (1) for E N because P does not scale with K.
Second, assume that 1 (1) for some E E [Z]. Then
[Keel122 5 Ke Σ
K
=
[ 0(1).
7=0
14

---

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
Also note that We, 2, does not scale with K. Thus,
We,t
(Wes + Kcx)be-1==-01 heg ((wr,)))=)))))))))) @(1)
Therefore, h = (1) for every E [0] U [1] and E.N.
Kt updates maximally for every E [T] and / N", iee, 12 where A,Ke Ke kee In the above have shown that he-1,t = ((1). Moreover,
Kelle = |e, Ke,ll22 S Σ g - Ktr 9(1).
Therefore, in the general case, A, = ((1), which concludes the proof.
Finally, according Lemma A.3 and the above two statement, we prove feature learning in every layer.
By Definition 3.2, the parametrization in Theorem 3.5 is μP for FNO.
15


Reviews:
[{"review_id": "1cWg0BHCM1", "reviewer": "Reviewer_tRrn", "summary": "This paper introduces $\\mu$Transfer-FNO, a zero-shot hyperparameter transfer technique for Fourier Neural Operators (FNOs). Based on the Maximum Update Parametrization (μP) framework, the authors propose a parametrization scheme that enables hyperparameters tuned on smaller FNOs to be transferred directly to much larger models without retraining. The authors derived a novel scaling law for the parameters of FNOs with respect to the number of Fourier modes. Extensive experiments on multiple PDEs like the Burgers' equation, Darcy Flow equation and the Navier-Stokes equation together demonstrate that μTransfer-FNO enables scaling of the nearly billion-parameter FNOs while reducing computational cost and maintaining accuracy.\n\n## Update After Rebuttal\nThe reviewer is positive about the results presented in the paper and satisfied with the authors' responses to the questions raised in all reviews. Hence, the reviewer would like to remain the score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for supporting our paper! We respond to your questions and concerns below.\n\n**Regarding the connection to Transformers.** This is an interesting point! The Fourier Integral Operator and Continuum Attention are both nonlocal operator classes, but they have different parametrizations:\n\n- For Continuum Attention, the parametrization closely resembles vanilla attention, and its size is controlled by the hidden dimensionality (the model \"width\"). Therefore, existing results on $\\mu$Transfer for scaling up vanilla attention are directly applicable.\n- In contrast, the Fourier Integral Operator is parametrized in a uniquely different way—by modeling the Fourier Transform of a kernel function with a pre-defined number of Fourier modes (i.e., $K$ in our paper's notation). This design differs from all modules studied in existing Maximum Update Parametrization ($\\mu$P) and $\\mu$Transfer literature, and our findings show that its scaling rate is also drastically different from that of other model components.\n\nThat being said, in Fourier Integral Operators, the notion of model \"width\" still exists (corresponding to $m$ in our paper's notation). The size of Fourier Integral Operators is controlled by both $m$ and $K$. Existing width scaling results apply to $m$, while our analysis applies to $K$, which is the unique aspect of Fourier Integral Operators.\n\n**Regarding to other architectures within the operator learning framework.** We acknowledge that our current findings are limited to FNOs. We believe it would be a meaningful future direction to study $\\mu$P and $\\mu$Transfer for other operator learning models, e.g., DeepONets.\n\nWe sincerely hope that our responses address your concerns. We are also happy to discuss further if you have any additional questions. Thank you again for your time!", "claims_and_evidence": "Here is the main claim made in this paper: $\\mu$Transfer-FNO allows zero-shot hyperparameter transfer from small-scale FNOs to large-scale FNOs. Sufficiently many experiments are included to compare $\\mu$Transfer-FNO with standard parametrization, which supports the claim in a clear and convincing way.", "methods_and_evaluation_criteria": "Yes. The experiments are mainly based on the Burgers’ Equation (1D), the Darcy Flow Equation (2D), and the Navier-Stokes Equations (3D), which are standard examples used in the original FNO paper. The authors studies the performance of $\\mu$Transfer-FNO on the finetuning of multiple parameters like the learning rates and batch sizes with respect to different number of Fourier modes. Experiments on other variants of FNO like PINO (Physics-Informed Neural Operator) are also included to justify the efficacy of the proposed methodology.", "theoretical_claims": "The only theoretical claim in the paper is Theorem 1.1 (informal version of Theorem 3.5). Its proof in the supplement has been verified to be correct.", "experimental_designs_or_analyses": "Yes, please refer to the \"Methods And Evaluation Criteria\" section above.", "relation_to_broader_scientific_literature": "This work is based on the Maximal Update Parametrization ($\\mu$P) framework proposed in the series of work on tensor programs, which is originally used in the studies of MLPs and Transformers. However, this paper should be mainly posited within the literature (FNOs, DeepOnets and their variants) on operator learning and neural PDE solvers.", "essential_references_not_discussed": "Below is one preprint that probably needs to be discussed in the paper is [1]. Specifically, the main result in [1] discussed the connection between neural operators and transformers (the attention architecture), which is also partially addressed in the cited paper [2]. The authors should discuss how the $\\mu$Transfer-FNO methodology proposed in this paper differs from the $\\mu$Transfer framework originally proposed for transformers, since neural operators can be linked with transformers under certain circumstances. \n\nReferences:\n\n[1] Calvello, Edoardo, Nikola B. Kovachki, Matthew E. Levine, and Andrew M. Stuart. \"Continuum attention for neural operators.\" arXiv preprint arXiv:2406.06486 (2024).\n\n[2] Kovachki, Nikola, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. \"Neural operator: Learning maps between function spaces with applications to pdes.\" Journal of Machine Learning Research 24, no. 89 (2023): 1-97.", "other_strengths_and_weaknesses": "To the best of the reviewer's knowledge, this is one of the first work that studies how to finetune parameters for large-scale FNOs based on good parameters trained on small-scale FNOs, which is of not only practical significance but also theoretical insights. For future work, it might be meaningful to study the proposed methodology under the setting of infinitely wide neural networks proposed in the series of paper on tensor programs. However, one potential weakness of this work is that it might be meaningful to test the proposed methodology on other architectures within the operator learning framework, such as DeepONets.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "JkcYhCVr69", "reviewer": "Reviewer_Xe5F", "summary": "The authors discuss how Fourier Neural Operators (FNOs), which is a state-of-the-art SciML method, have been used to solve complex PDEs. However, they identify issues with scaling FNO to more intricate PDEs that requires increasing the number of Fourier modes. Increasing the number of Fourier modes increases the number of model parameters and makes HPO very expensive. The authors propose $\\mu$Transfer-FNO, which is a zero-shot hyperparemeter transfer method with optimal hyperparameters tunes on smaller FNOs to be applied to large billion-parameter FNOs zero-shot. The method is based upon the Maximum Update Parametrization ($\\mu$P) framework. the authors show that $\\mu$Transfer-FNO reduces the cost of tuning parameters on large FNOs and maintains the accuracy.\n\n## Update after rebuttal\nI appreciate the authors' detailed rebuttal, motivating example and additional experiments. In light of that, I raised my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for supporting our paper! We respond to your main questions and concerns below.\n\n**Regarding motivating the need for larger Fourier modes.** A concrete example is based on Kolmogorov microscales in fluid dynamics: When simulating turbulent flows governed by the Navier-Stokes equations, the Kolmogorov microscales represent the smallest scales at which energy dissipation occurs, which are proportional to $\\mathrm{Re}^{-3/4}$ where $\\mathrm{Re}$ is the Reynolds number. For realistic engineering applications with high Reynolds numbers, these microscales become extremely small and require careful modeling of high-frequency features. In such cases, larger numbers of Fourier modes are preferred.\n\n**Regarding more PDEs.** Following your and other reviewers' suggestions, we conduct additional experiments on a mixed equation dataset involving Burgers' Equation, Advection Equation, and Reaction-Diffusion Equation from PDEBench. This mixed equation dataset setting is relevant for PDE foundation model training as noted by other reviewers. We first train FNOs with $K=6$ using different learning rates, then train FNOs with $K=24$ under both Standard Parametrization and $\\mu$Transfer-FNO. We present the loss for each run below:\n\n| $\\log_{10}$(learning rate)        |  -2.0   |  -2.2   |  -2.4   |  -2.6   |  -2.8   |    -3.0     |    -3.2     |  -3.4   |  -3.6   |\n| --------------------------------- | :-----: | :-----: | :-----: | :-----: | :-----: | :---------: | :---------: | :-----: | :-----: |\n| $K=6$                             | 0.08314 | 0.04933 | 0.04665 | 0.03985 | 0.03822 | **0.03576** |   0.05366   | 0.05672 | 0.06318 |\n| $K=24$ (Standard Parametrization) | 0.98508 | 0.03785 | 0.03904 | 0.03200 | 0.03041 |   0.02852   | **0.02616** | 0.02989 | 0.03191 |\n| $K=24$ ($\\mu$Transfer-FNO)        | 0.03836 | 0.03517 | 0.03391 | 0.03192 | 0.02842 | **0.02599** |   0.02728   | 0.02925 | 0.03406 |\n\nThe results are consistent with our original findings: On the mixed equation dataset, the optimal hyper-parameter shifts when the model size scales up under standard parametrization. In contrast, $\\mu$Transfer-FNO stabilizes the optimal configuration. We believe these additional results strengthen our work. Thank you for the constructive comment!\n\n**Regarding the novelty compared to existing works on $\\mu$P and $\\mu$Transfer.** While the high-level algorithmic idea of $\\mu$Transfer-FNO is based on existing research, we are the first to derive the Maximum Update Parametrization ($\\mu$P) for FNOs. The result is novel in two aspects:\n\n1. We introduce new technical tools for analyzing neural network parametrization, going beyond LLN and CLT commonly used in literature.\n\n2. The $\\Theta\\left(\\frac{1}{\\sqrt{d\\log K}}\\right)$ scaling rate is drastically different from existing results on all the other model components (e.g., $\\Theta(m^{-1})$ for width scaling and $\\Theta(L^{-1/2})$ for depth scaling). \n\nWe point out that directly applying existing results does not work for FNOs because the design of the kernel integral operator is significantly different from other standard neural network modules such as embedding layers and linear transforms. \n\n**Regarding more general NOs than just FNO.** Unfortunately, our current theoretical analysis is specific to the kernel integral operator and FNO. We will mention this as a limitation of our work in the paper revision and include references to other NO variants including Multi-wavelet NO, Multipole Graph Neural Operator, etc.\n\n**Regarding extrapolation in time.** Our experiments focus on more standard settings for FNO and do not consider extrapolation in time like the Markov Neural Operator. We believe that our technique is still applicable to this setting since the design of FNO is unchanged.\n\n**Regarding Table 1.** Thank you for catching the typo! As for the total training cost, this difference occurs because the size of FNO scales as $\\mathcal{O}(K)^d$. Thus, when $d$ increases, the efficiency gain from tuning models with small $K$ and applying $\\mu$Transfer-FNO becomes much more significant. Specifically, the Darcy Flow problem with FNO-2D is relatively simple. Direct tuning on the full-sized model with a smaller hyper-parameter search space can still lead to decent results with reasonable computational cost. Hence, this setting is not particularly favorable to $\\mu$Transfer-FNO. However, for the Navier-Stokes Equation with FNO-3D, tuning the full model becomes prohibitively expensive. In this more complex setting, $\\mu$Transfer-FNO offers substantial efficiency gains by allowing us to tune models with small $K$ and then transfer the optimal hyper-parameters to larger models.\n\nDue to the word limit, we cannot respond to other comments individually. But we would like to assure you that we value your comments and will modify our paper accordingly based on your writing suggestions and references. We sincerely hope that our responses address your concerns!", "claims_and_evidence": "- The proposed transfer method makes sense to help scale HPO tuning for large FNOs. I would like to see a concrete example where larger Fourier modes are required to solve the PDE for better problem motivation. \n- The authors correctly motivate that FNO scales as $\\mathcal{O}(K)^d$, where $K$ is the number of Fourier modes and $d$ is the dimension of the problem, solving that the complexity grows exponentially with the dimension so for practical 3D space and time problems, this can be very expensive.\n- Rather than showing the loss in Figure 1 , it may be more informative to show the validation accuracy as a more meaningful metric. It is interesting that the optimal learning rate with the proposed method remains approximately the same regardless of the model size. To better show this, it may be good to put a vertical line through the star points.", "methods_and_evaluation_criteria": "- It makes sense to try to increase the number of Fourier modes and the proposed transfer HPO method seems effective. To do so, the proposed method scales the kernel integral operator parameters as the number of modes $K$ is increased so that the optimal hyperparameters stay approximately the same across model sizes. This motivates HPO on the small model and then directly transferring to larger FNOs.\n- The result cover a good range of PDEs, i.e., Burgers, Darcy Flow and the challenging Navier-Stokes equations. I would like to see a larger number of equations tested. See the comprehensive Neural Operator benchmark in Saad et al., \"Guiding continuous operator learning through Physics-based boundary constraints\", ICLR 2023.\n- The authors show that the proposed transfer learning preserves the same optimal learning rate, training batch size and optimizer configurations across increasing model sizes.\n- With the proposed approach, the method is able to scale to large FNOs with nearly 1B parameters with better accuracy and only `0.3x` training compute.\n- In Figure 2, even with the proposed approach the optimal learning rates are not exactly the same and seem slightly shifted but less than the base especially in Figure 2a-b. Is there a quantitative metric to better measure this difference?", "theoretical_claims": "- The main theoretical result is presented in Theorem 3.5  on calculating the scaling rate for kernel up the FNO kernel integral operator. The theorem shows the abc-parametrization is a $\\mu$P of FNO with the Adam optimizer and scales according to 1 / sqrt(d), where d is the dimensionality. The proof is detailed and provided in the appendix with a proof outline of the main concepts in the main body.\n- The authors highlight a difference in their proof than in the standard $\\mu$P proofs that the past proofs rely on the CLT to measure the average of random variables, whereas there is a technically difference in the structure of FNO and the kernel integral operator that leads to the analysis of the maximum of the random variables. It is good that the authors clarify this difference.\n- It is also good that the proposed scaling function in the proof is not resolution dependent so that the desired resolution invariance property of NOs is not broken with this methods - do the authors have proof of this, i.e.,that resolution invariance still holds?", "experimental_designs_or_analyses": "- Nice that experiment section is organized according to various questions to show that aforementioned theory also holds empirically.\n- Please clarify that Burgers' in 4.1 is actually viscous Burgers' for $\\nu \\ne 0$, which is simpler to solve than the sharp shock solution case where $\\nu = 0$ and there is no artificial diffusion from the viscosity term.\n- Good that the authors consider 1D and 2D problems and also vary mapping the initial condition to solution in Burgers' and PDE coefficient to the solution in Darcy Flow.\n- Since one of the main motivations of the proposed method is for large-scale problems, I think 3D spatial cases should also be used because that when $d=3$ is when the FNO becomes very computational expensive. It is good that 3D Navier-Stokes is tested. Is this FNO-3D because it is the standard 2D space + time test case? I think 3D space should also be considered. An example practical, real-world 3D Car Surface Pressure Prediction test case is provided in Ma et al., \"Calibrated uncertainty quantification for operator learning via conformal prediction\", 2024 and Li et al., \" Geometry-informed neural operator for large-scale 3d pdes\", 2023. Both of these works should be cited as well.", "relation_to_broader_scientific_literature": "- FNOs have shown large impact on solving PDE problems with ML. This paper focuses on more challenging PDEs where larger parameter FNOs are needed with more Fourier modes. The authors in the introduction go directly into discussing the specifics of FNO and the effect of its modes. I think the authors should first motivate solving PDE problems, why ML methods have been developed to solve these problems, mention the three main method classes, e.g., PINNs, Neural Operators and MeshGraphNets and then state that this paper will focus on FNOs. The authors can tell clearly identify the more challenging PDEs with regular size parameters FNOs struggle to motivate these larger-parameter FNO models which require HPO.\n- The authors mention the $\\mu$ P (Yang & Hu (2021) and $\\mu$Transfer methods (Yang et al. (2022)). Please clearly differentiate the novelty of the proposed approach. Does $\\mu$Transfer-FNO just directly apply these methods to FNO? If so, please explain the technical challenges of applying them to the FNO architecture.\n- My main concern is that the proposed method is too similar to the past Yang & Hu (2021)  and Yang et al. (2022) in literature and hence the novelty needs to be clarified.\n- The authors state that the number of hidden dimensions $m$ and model depth $L$ are fixed in this work and have been studied in  Yang & Hu (2021) and Yang et al. (2022). Please briefly summarize their effect here or in an appendix so that this work can be fully understood end-to-end without relying on these prior works for key concepts.", "essential_references_not_discussed": "- It is good that the authors include the known boundary conditions in the problem definition in Eqn. 1. There is missing reference to Saad et al., \"Guiding continuous operator learning through Physics-based boundary constraints\", ICLR 2023 that enforces boundary conditions as an exact constraint.\n- The authors mention that FNO is most commonly NO. They should state the reason is because the FFT is very computational efficient to compute the kernel vector $\\mathcal{K}x$ products. Having said that, the paper is missing references to other neural operators with different bases, e.g.,  Gupta et al., “Multiwavelet-based Operator Learning for Differential Equations”. In: Advances in Neural Information Processing Systems. Vol. 34. PMLR, pp. 24048–24062, 2021, Li, Zongyi, Nikola Kovachki, et al. (2020b). “Multipole Graph Neural Operator for Parametric Partial Differential Equations”. In: arXiv preprint arXiv:2006.09535.\n- It is good that the authors also compare to PINO. I think the authors should be a bit careful referring to the PINNs loss as \"more advanced training techniques\". Both the PINO paper and Saad et al., ICLR 2023 show that in several cases the unconstrained FNO outperforms PINO. Also there have been several instabilities with PINNs training that have been reported in the literature that are not cited, e.g., Krishnapriyan et al., \"Characterizing possible failure modes in physics-informed neural networks\", Advances in neural information processing systems 34, 26548-26560, 2021. This looks like it is also observed by the authors own training of PINO in Figure 2(c) with the spike and oscillations in the loss function.\n- References to transfer learning methods other than $\\mu$ P (Yang & Hu (2021) and $\\mu$Transfer methods (Yang et al. (2022)) are missing.", "other_strengths_and_weaknesses": "## Weaknesses\n- The author should better define what they mean by \"intricate PDEs\" in the abstract.\n- The introduction is missing a contribution section. I would also move Theorem 1.1 to the method section and clearly state the contributions in the Introduction.\n- The authors should define $a_i$ in the training pairs, e.g., initial conditions, PDE parameters. Using $a_i$ for the notation is confusing since later in 4.1 a is used as the Darcy coefficient as well and sometimes for Burgers the initial condition$u_0$ is used in the mapping\n\n\n## Strengths\n- The authors address a common problem with FNOs in scaling to higher dimensions, i.e., the curse of dimensionality and propose a simple transfer learning approach of the hyperparameters to scale to larger FNOs with more modes to allow costly HPO on these larger architectures.", "other_comments_or_suggestions": "Typos\n- Line 221 left column, \\citet should be used instead of \\citep for the reference.\n- Capital W on line 374 right column to start the sentence.\n- \"serveral\" on line 413 left.", "questions_for_authors": "1. How would this method apply to more general NOs than just FNO, e.g., the Multi-wavelet NO in Gupta et al., 2021?\n2. Are any of the NO method exampels in this work including time or extrapolation in time like the Markov Neural Operator, Li et al., \"Learning Dissipative Dynamics in Chaotic Systems\", 2021.\n3. Why is the total training cost increased from tuning the full model for Darcy Flow in FNO-2D but decreased for FNO-3D in Table 1. Also the table caption states that three PDEs are shown in the table but it only shows 2.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MjmIH6x6O8", "reviewer": "Reviewer_apVL", "summary": "This paper introduces μTransfer-FNO, a zero-shot hyperparameter transfer method for Fourier Neural Operators (FNOs). The core idea is to derive a Maximum Update Parametrization (μP) for FNOs that allows hyperparameters tuned on small FNOs to be directly transferred to larger FNOs without additional tuning, even for models with billions of parameters. The paper theoretically derives the μP for FNOs under the scaling of Fourier modes $K$, finding that initialization variances of kernel integral parameters should be scaled by $O\\Big(\\dfrac{1}{d \\cdot \\log K}\\Big)$ and learning rates by $O\\Big(\\dfrac{1}{\\sqrt{d \\cdot \\log K}}\\Big)$, where $d$ is the PDE dimensionality. Experiments on Burgers' Equation, Darcy Flow, and the Navier-Stokes Equation demonstrate that μTransfer-FNO maintains optimal learning rates, batch sizes, and optimizer configurations across model scales, reducing computational tuning costs while preserving or improving accuracy. The method is also shown to be applicable to Physics-Informed Neural Operators (PINOs).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for supporting our paper! We respond to your main questions and concerns below.\n\n**Regarding the scope of PDEs.** Following your and other reviewers' suggestions, we conduct additional experiments on a mixed equation dataset involving Burgers' Equation, Advection Equation, and Reaction-Diffusion Equation from PDEBench. This mixed equation dataset setting is relevant for PDE foundation model training as noted by other reviewers. We first train FNOs with $K=6$ using different learning rates, then train FNOs with $K=24$ under both Standard Parametrization and $\\mu$Transfer-FNO. We present the loss for each run below:\n\n| $\\log_{10}$(learning rate)        |  -2.0   |  -2.2   |  -2.4   |  -2.6   |  -2.8   |    -3.0     |    -3.2     |  -3.4   |  -3.6   |\n| --------------------------------- | :-----: | :-----: | :-----: | :-----: | :-----: | :---------: | :---------: | :-----: | :-----: |\n| $K=6$                             | 0.08314 | 0.04933 | 0.04665 | 0.03985 | 0.03822 | **0.03576** |   0.05366   | 0.05672 | 0.06318 |\n| $K=24$ (Standard Parametrization) | 0.98508 | 0.03785 | 0.03904 | 0.03200 | 0.03041 |   0.02852   | **0.02616** | 0.02989 | 0.03191 |\n| $K=24$ ($\\mu$Transfer-FNO)        | 0.03836 | 0.03517 | 0.03391 | 0.03192 | 0.02842 | **0.02599** |   0.02728   | 0.02925 | 0.03406 |\n\nThe results are consistent with our original findings: On the mixed equation dataset, the optimal hyper-parameter shifts when the model size scales up under standard parametrization. In contrast, $\\mu$Transfer-FNO stabilizes the optimal configuration. We believe these additional results strengthen our work. Thank you for the constructive comment!\n\n**Regarding our theoretical results.**\n\n- **Spectral Norm Calculation:** Our proof is applicable to a general dimension $d$. We only use the fact that the Fourier transform is orthogonal, and that the Fourier transform, the truncation operator, the multiplication with the parameter tensor, and the inverse Fourier transform are all linear operators. All these properties hold regardless of the dimensionality.\n\n  **The sub-Gaussian assumption:** We note that our analysis focuses on the Adam optimizer which uses entry-wise normalized gradient momentum for updates. Furthermore, one can optionally apply value-based gradient clipping in practice, which strictly enforces all entries of the update to be bounded and hence sub-Gaussian. We have empirically verified this assumption in our preliminary experiments and will add a remark on this in the paper revision.\n\n  **Extending to the case with a general $m$:** In this setting, the analysis can be broken down by dealing with each entry of the matrix-vector product separately and writing the product as the summation of entry-wise multiplication. Then one can treat $\\widetilde{\\boldsymbol R}_{\\ell}\\in\\mathbb{R}^{N_1 \\cdots N_d\\times N_1 \\cdots N_d\\times m\\times m}$ as $m\\times m$ instantiations of $N_1 \\cdots N_d\\times N_1 \\cdots N_d$ matrices and apply the analysis in the $m=1$ case to arrive at the same result. Based on this argument, a general $m$ would not incur complicated dependency on the scaling rate of $K$.\n\n  **Discretization Effects:** Our proof does not rely on any property of a specific resolution. We mainly leverage the fact that the Fourier transform matrix is an orthogonal matrix in the discrete formulation, and that the main building blocks in the kernel integral operators are all linear. Connecting to the continuous formulation of FNOs, these facts can be understood as consequences of orthogonality and linearity of the corresponding operators over functional spaces.\n\n**Regarding theoretical analysis on hyper-parameter transfer.** To the best of our knowledge, no theoretical bounds exist for hyper-parameter transfer across different sizes or architectures. Most existing work, including our results, focuses on asymptotic scaling rates, building on or adjacent to the $\\mu$Transfer framework as discussed in Section 5.3. Another relevant recent result is [1], which we will include in our discussion.\n\n[1] Super Consistency of Neural Network Landscapes and Learning Rate Transfer, NeurIPS 2024\n\n**Regarding alternative parametrizations.** This is an interesting question! Our analysis is not tied to any specific PDE, similar to how other existing analyses are not tied to any specific task or dataset. However, $\\mu$Parametrization does vary for different optimization algorithms. For example, [2] derives $\\mu$P for K-FAC and Shampoo optimizers. Our analysis focuses on Adam because of its popularity.\n\n[2] On the parameterization of second-order optimization effective towards the infinite width.\n\nDue to the word limit, we cannot respond to other comments individually. But we would like to assure you that we value your comments and will modify our paper accordingly based on your comments. We sincerely hope that our responses address your concerns!", "claims_and_evidence": "The paper supports its claims with evidence, but some areas could benefit from further clarification or stronger support:\n\n**Strengths:**\n* **µP Derivation:** The mathematical derivation of the µP for FNOs appears sound, although verifying the full proof requires careful examination of the appendix.\n* **Learning Rate Stability:** The experiments clearly demonstrate that µTransfer-FNO stabilizes the optimal learning rate across different K values, supporting the central claim. Figures 1 and 2 provide visual evidence of this stability.\n* **Generalizability to PINO:** The extension to Physics-Informed Neural Operators (PINOs) is well-supported by the experiments on Darcy flow (Figure 2c). \n* **Transfer of Other Hyperparameters:** The experiments on batch size and Adam's $\\beta_2$ (Figure 3) provide evidence that µTransfer-FNO can transfer hyperparameters beyond the learning rate, strengthening the overall argument.\n\n**Weaknesses:**\n* **Instability at Small $K$:** The paper acknowledges that the optimal learning rate can deviate at very small $K$ values, attributing it to training randomness. While plausible, further investigation or discussion of this instability would be beneficial. Perhaps providing error bars or statistics across multiple runs would strengthen this point.\n* **Computational Cost Savings:** While Table 1 shows computational savings for the Navier-Stokes equation, the Darcy flow experiment shows increased training cost. The paper argues this is due to the smaller gap between small and large models in this setting. However, a more thorough analysis of the computational cost trade-offs, perhaps considering a wider range of model sizes, would be helpful. Quantifying the computational cost of the hyperparameter search itself would also be valuable.\n* **Test Error Improvement:** The paper claims lower test error with µTransfer-FNO, attributing it to the ability to explore a larger hyperparameter search space with smaller models.While this is a reasonable explanation, it would be stronger to directly compare against a baseline that uses the same larger search space but tunes the large model directly (albeit at higher computational cost). This would isolate the benefit of µTransfer-FNO from simply using a larger search space.\n* **Limited Scope of PDEs:** While the chosen PDEs are representative, exploring a wider range of PDE problems (PDEBench) would further strengthen the claim of generality. Including more complex or higher-dimensional PDEs would be particularly valuable.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria make sense for the problem of hyperparameter transfer in FNOs for PDE solving.\n* **µP as a Method:** The use of µP as a foundation for hyperparameter transfer is well-motivated.The core idea of maintaining consistent training dynamics across model sizes is directly relevant to the goal of transferring hyperparameters. The theoretical derivation provides a principled basis for the proposed scaling factors.\n\n* **Algorithm 1:** The µTransfer-FNO algorithm is a straightforward and logical application of the µP theory. It clearly outlines the steps involved in transferring hyperparameters from a small proxy model to a larger target model.\n\n* **Choice of PDEs:** The selected PDEs (Burgers' Equation, Darcy Flow, and Navier-Stokes Equation) represent a reasonable range of complexity and dimensionality. They are commonly used as benchmarks in the FNO literature, allowing for comparison with existing work. While a broader set of PDEs (PDEBench) would be even better, the chosen set provides a good starting point.\n\n* **Evaluation Metrics:** Using the L2 relative test error is a standard and appropriate metric for evaluating the performance of PDE solvers. It directly measures the accuracy of the solution obtained by the FNO.\n\n* **Comparison with Baseline:** Comparing µTransfer-FNO against directly tuning the large model provides a relevant baseline.This allows for assessing the computational cost savings and potential performance gains of the proposed method. However, as mentioned in the previous response, a stronger baseline would involve using the same expanded hyperparameter search space for both methods.", "theoretical_claims": "I've reviewed the provided proof of Theorem 3.5, which establishes the µP for FNOs. While the overall structure of the proof seems reasonable, there are some specific points and potential issues:\n\n* **Spectral Norm Calculation:** The proof relies heavily on calculating the spectral norm of the kernel integral operator $\\mathcal{K}$. The argument that this norm is equivalent to the maximum absolute value of the parameters r seems plausible given the diagonal structure of $\\mathcal{R}$ after the Fourier transform. However, the interaction between the multidimensional Fourier transform, the truncation operator $\\mathcal{T}_K$, and the parameter tensor $\\mathcal{R}$ could be complex, and a more detailed justification of this step would be beneficial.\n\n* **High-Probability Bound:** The proof invokes a standard result from high-dimensional probability to bound the maximum of $K^d$ sub-Gaussian random variables. While this is a common technique, the specific constants and assumptions underlying this result need careful checking against the properties of the parameters $r$. The paper assumes these are sub-Gaussian, which needs to be verified in practice.\n\n* **Simplification to $m=1$:** The proof simplifies the analysis by assuming a hidden dimension $m=1$. While extending to general $m$ is mentioned as straightforward, explicitly showing this extension, or at least providing a sketch, would strengthen the proof. The interaction between $m$ and $K$ in the scaling factors could be non-trivial.\n\n* **Discretization Effects:** The proof works with the discretized version of the FNO. While this is necessary for practical implementation, the impact of discretization on the theoretical results is not explicitly discussed. Ideally, the proof should connect back to the continuous formulation of FNOs.", "experimental_designs_or_analyses": "I've reviewed the experimental designs and analyses presented in the paper. While they provide support for the claims, there are some areas where the soundness and validity could be improved:\n\n* **Hyperparameter Search Space:** The paper doesn't explicitly define the hyperparameter search space $\\Xi$ used in Algorithm 1. Knowing the range and granularity of the search for each hyperparameter (learning rate, batch size, $\\beta_2$) is crucial for interpreting the results. A larger search space could lead to better results regardless of µTransfer-FNO, so specifying the search space is essential for a fair comparison.\n\n* **Baseline Comparison:** As mentioned previously, the comparison with the baseline of directly tuning the large model is not entirely fair. µTransfer-FNO benefits from exploring a potentially larger search space with the smaller proxy model. A stronger baseline would involve using the same expanded search space for tuning the large model directly, even if it's computationally more expensive. This would isolate the benefit of the transfer method itself.\n\n* **Limited Number of Runs:** The paper presents results for single runs of each experiment. Given the stochastic nature of neural network training, reporting results averaged over multiple runs (e.g., $10-20$) with standard deviations or error bars would provide a more robust evaluation and account for potential variability.\n\n* **Computational Cost Analysis:** The analysis of computational cost is somewhat limited. Table 1 only provides relative training costs ($1\\times$, $1.38\\times$, $0.30\\times$). Reporting absolute training times or FLOPs would be more informative. Furthermore, the cost of the hyperparameter search itself is not factored into the comparison. A more comprehensive analysis should include the total cost of both the search and the final training.\n\n* **Lack of Ablation Study:** An ablation study investigating the individual contributions of the initialization variance scaling and the learning rate scaling would provide further insights into the effectiveness of the proposed µP. This would help determine the relative importance of each component.\n\n* **Details of Data Generation:** While the paper describes the PDEs used, more details about the data generation process would improve reproducibility.\n\n* **Code Availability:** Providing the code used for the experiments would significantly enhance reproducibility and allow for independent verification of the results.", "relation_to_broader_scientific_literature": "This paper's main contributions relate to various areas within the broader scientific literature:\n\n* **Fourier Neural Operators (FNOs) and Operator Learning:** The paper directly builds upon the existing literature on FNOs for solving PDEs (Li et al., 2021).\n\n* **Physics-Informed Neural Networks (PINN):** The paper extends its method to Physics-Informed Neural Operators (PINOs) (Li et al., 2024), demonstrating its applicability beyond standard supervised learning for PDEs. \n\n* **Maximal Update Parametrization (µP) and µTransfer:** The core theoretical contribution relies heavily on the µP framework introduced by Yang & Hu (2021) and the concept of µTransfer (Yang et al., 2022).\n\n* **Hyperparameter Transfer Learning:** The overall goal of the paper is to enable efficient hyperparameter tuning for large models by transferring knowledge from smaller models.", "essential_references_not_discussed": "The paper covers the most relevant literature regarding FNOs, PINNs, and µP/µTransfer. However, a few potential areas could benefit from additional discussion or referencing:\n\n* **Theoretical Analysis of Hyperparameter Transfer:** While the paper provides a theoretical justification for µP in FNOs, it could benefit from discussing any existing theoretical work on hyperparameter transfer in general. Are there any theoretical guarantees or bounds on the effectiveness of transferring hyperparameters across different model sizes or architectures? Connecting to this broader theoretical literature would strengthen the paper's contribution.\n\n* **Alternative Parametrizations:** The paper focuses on µP, but other parametrization schemes might exist for FNOs. Discussing potential alternatives and their potential advantages or disadvantages compared to µP would provide a more complete picture. For example, are there parametrizations specifically designed for different optimization algorithms or different types of PDEs?", "other_strengths_and_weaknesses": "In summary, the paper is a valuable contribution, excelling in originality and practical use. Strengths and weaknesses were detailed in the preceding sections.", "other_comments_or_suggestions": "* Page 5: \"descritized\" needs to be replaced by \"discretized\"\n* Page 7: \"serveral\" needs to be replaced by \"several\"", "questions_for_authors": "Questions were depicted in the preceding sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Na1pXfgaTe", "reviewer": "Reviewer_r9fK", "summary": "This paper applies the Maximum Update Parametrization (µP) framework to Fourier Neural Operators (FNO), demonstrating that a single set of hyperparameters can effectively work for both large-scale and small-scale FNO models.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments and constructive suggestions. Let us respond to your concerns one-by-one below.\n\n**Regarding the significance of our work.**\n\nOur main contributions can be summarized as follows:\n\n* **On the theory side:** We are the first to derive the Maximum Update Parametrization ($\\mu$P) for FNOs, identifying the unique scaling rate for kernel integral parameters that leads to $\\mu$P. The result is novel in two aspects:\n  1. We introduce new technical tools for analyzing neural network parametrization, going beyond LLN and CLT commonly used in literature.\n  2. The $\\Theta\\left(\\frac{1}{\\sqrt{d\\log K}}\\right)$ scaling rate is drastically different from existing results on all the other model component (e.g., $\\Theta(m^{-1})$ for width scaling and $\\Theta(L^{-1/2})$ for depth scaling). Directly applying existing result does not work for FNOs.\n\n* **On the algorithm side:** Based on our derived scaling rates in $\\mu$P, we introduce $\\mu$Transfer-FNO for zero-shot hyper-parameter transfer in FNO.\n\n* **On the experiment side:** We validate our theoretical claims on various PDEs, different training algorithms, and different hyper-parameters. The experiments consistently show the robustness of the theory. We also demonstrate that $\\mu$Transfer-FNO can significantly reduce computational costs while maintaining or improving accuracy in practice.\n\nIn particular, we believe that scaling up FNO is meaningful. Conceptually, FNO is designed to model continuous functions and enjoys the resolution-invariant property which standard Transformers do not have, making it a strong candidate for modeling PDE data. Practically, FNO is popularly used in recent research on pretraining and foundation models [1-4]. Therefore, we expect that our findings are relevant and interesting to the community.\n\nWe also fully agree with the reviewer that Transformers are a powerful backbone for building foundation models. Developing techniques to scale up Transformers is also an important research direction, but it is orthogonal to the focus of this work. We appreciate your references and will add discussions on these related works in the revision.\n\n[1] Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior, NeurIPS 2023\n\n[2] Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning, NeurIPS 2024\n\n[3] Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs, NeurIPS 2024\n\n[4] UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation, TMLR 2024/11\n\n**Regarding the dataset choice of our work.**\n\nWe first point out that the primary focus of our experiments is to validate the robustness and generality of our theoretical claims, rather than benchmarking on the most challenging PDE datasets. The datasets we used are representative and commonly used in existing research.\n\nWe also agree with the reviewer that mixed equation training is a common setting in PDE foundation model training. Following your suggestion, we conducted additional experiments on a mixed equation dataset involving Burgers' Equation, Advection Equation, and Reaction-Diffusion Equation using the data curated in [5]. Similar to the experimental setup in our submission, we first train FNOs with $K=6$ using different learning rates. Then train FNOs with $K=24$ under Standard Parametrization/$\\mu$Transfer-FNO. We present the loss for each run below:\n\n| $\\log_{10}$ (learning rate)       |  -2.0   |  -2.2   |  -2.4   |  -2.6   |  -2.8   |    -3.0     |    -3.2     |  -3.4   |  -3.6   |\n| --------------------------------- | :-----: | :-----: | :-----: | :-----: | :-----: | :---------: | :---------: | :-----: | :-----: |\n| $K=6$                             | 0.08314 | 0.04933 | 0.04665 | 0.03985 | 0.03822 | **0.03576** |   0.05366   | 0.05672 | 0.06318 |\n| $K=24$ (Standard Parametrization) | 0.98508 | 0.03785 | 0.03904 | 0.03200 | 0.03041 |   0.02852   | **0.02616** | 0.02989 | 0.03191 |\n| $K=24$ ($\\mu$Transfer-FNO, ours)  | 0.03836 | 0.03517 | 0.03391 | 0.03192 | 0.02842 | **0.02599** |   0.02728   | 0.02925 | 0.03406 |\n\nThe results are consistent with our original findings: On the mixed equation dataset, the optimal hyper-parameter shifts when the model size scales up under standard parametrization. In contrast, $\\mu$Transfer-FNO stabilizes the optimal configuration with the lowest loss consistently obtained with the learning rate being $10^{-3.0}$, enabling zero-shot optimal hyper-parameter transfer from small models to large ones. We hope the additional result strengthen our work. Thank you for the constructive comment!\n\n[5] PDEBench: An Extensive Benchmark for Scientific Machine Learning, NeurIPS 2022\n\nWe sincerely hope that our responses address your concerns and that you reevaluate our work based on the additional information. Thank you again for your time!", "claims_and_evidence": "While I understand the authors' claims, the significance of this research question remains unclear. Scaling up transformer-like models appears to be a more straightforward alternative.", "methods_and_evaluation_criteria": "The authors apply a framework for transferring hyperparameters from small to large FNO models.", "theoretical_claims": "I have not verified the details of the theoretical proofs.", "experimental_designs_or_analyses": "The datasets used in this study appear overly simplistic. The necessity for FNO models with billions of parameters in these tasks is questionable. More challenging datasets, such as those involving multiple mixed equations for training PDE foundation models [1,2], would provide more meaningful scenarios for investigating hyperparameter tuning in large-scale models.\n[1] Unisolver: PDE-Conditional Transformers Are Universal PDE Solvers\n[2] PDEformer: Towards a Foundation Model for One-Dimensional Partial Differential Equations", "relation_to_broader_scientific_literature": "This work may contribute to hyperparameter tuning research.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The rationale for scaling up FNO is unclear, as FNO itself does not seem particularly suitable for scaling. Current transformer-based models demonstrate superior performance and stronger scaling capabilities compared to FNO, raising questions about the significance of scaling up FNO.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a hyperparameter transfer technique for FNOs based on the maximum update parameterization framework. The authors provide both theoretical and experimental evidence that their framework allows to transfer hyperparameters from small to large models. The reviewers raised concerns about context, evaluation, and framing, which the authors carefully addressed in their rebuttal, leading to one of the scores being raised.

The scores are 2 "Accept", "Weak accept", and "Weak reject". The "weak reject" review is primarily concerned with significance; specifically, the reviewer doesn't see the importance of studying and scaling FNOs (favoring transformers instead). This opinion is not universal, as evidenced by the other three reviews. The authors' rebuttal also carefully explains the significance and ubiquity of FNOs in the field.
Decision: Accept (poster)