Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs
Authors: Robert Joseph George
Jiawei Zhao
Jean Kossaifi
Zongyi Li
Anima Anandkumar
Venue: Accepted by TMLR ()

Abstract: Fourier Neural Operators (FNO) offer a principled approach to solving challenging partial differential equations (PDE) such as turbulent flows. At the core of FNO is a spectral layer that leverages a discretization-convergent representation in the Fourier domain, and learns
weights over a fixed set of frequencies. However, training FNO presents two significant challenges, particularly in large-scale, high-resolution applications: (i) Computing Fourier transform on high-resolution inputs is computationally intensive but necessary since fine-scale details are needed for solving many PDEs, such as fluid flows, (ii) selecting the relevant set of frequencies in the spectral layers is challenging, and too many modes can lead to overfitting, while too few can lead to underfitting. To address these issues, we introduce
the Incremental Fourier Neural Operator (iFNO), which progressively increases both the number of frequency modes used by the model as well as the resolution of the training data. We empirically show that iFNO reduces total training time while maintaining or improving
generalization performance across various datasets. Our method demonstrates a 38% lower testing error, using 20% fewer frequency modes compared to the existing FNO, while also achieving up to 46% faster training and a 2.8x reduction in model size.

Published in Transactions on Machine Learning Research (09/2024)
Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs
Robert Joseph George
ryoorge edu
Department of Computing and Mathematical Sciences California Institute of Technology
Jiawei Zhao
jinweieealeech edu
Department Compuling and Mathematical Sciences California Institute of Technology
Jean Kossaifi NVIDIA Al
jkossaifi@nvviia..mm com
Zongyi Li
zongylli@ealtech.coo
Department of Computing and Mathematical Sciences California Institute of Technology
Animashree Anandkumar
anéma@caltechh, edu
Department of Computing and Mathematical Sciences California Institute of Technology
Reviewed on OpenReview: https: opt/r/view.evee..moo net/ forum? id- =IxI6@@00pp0
Abstract
Fourier Neural Operators (FNO) offer principled approach to solving challenging partial differential equations (PDE) such turbulent flows. At the of FNO is spectral layer that leverages a discretization-convrgen representation in the Fourier domain, and learns weights over fixed set of frequencies. However, training FNO presents two significant challenges, particularly in large-scale, high-resolution applications: (i) Computing Fourier transform on high-resolution inputs computationally intensive but necessary since fine- scale details are noeded for solving many PDEs, such as fluid flows, (ii) selecting the relevant set of frequencies in the spectral layers challenging, and too many modes lead to overfitting, while too few can lead to underfitting. To address these issues, introduce the Incremental Fourier Newral Operator (iFNO), which progressively increases both the number of frequency modes used by the model as well as the resolution of the training data. We empirically show that iFNO reduces total training time while maintaining or improving generalization performance across various datasets. Our method demonstrates a 38% lower testing error, using 20% fewer frequency modes compared to the existing FNO, while also achieving up 46% faster training and 2.8x reduction in model size.
1 Introduction
Recently, deep learning has shown promise in solving partial differential equations (PDEs) significantly faster than traditional numerical methods in many domains. Among them, Fourier newral operator (FNO), proposed by Li et al. (a), a family of neural operators that achieves state-of-the-arl performance in solving PDEs Azizzadeneshelii et al. (2024), in applications modeling turbulent flows, weather forecasting Pathak et (2022), material plasticity Liu et al. (2022), and carbon dioxide storage Wen et al. 2022) FNO is specifically designed learn mesh-independentt solution operators for PDEs. Unlike traditional neural networks that. learn point-wise mappings, FNO learns functional mapping between infiniteddimensional

---

Published in Transactions on Machine Learning Research (09/2024)
lowest modes
FFT
truncate
weights
iFFT
F-1
iFNO Block
Spectral Conv
N loyers
Spectral Conv
Lifting
MLP
MLP
Projection
Loss L
Dynamically add modes
Incremental algorithm
Dynamically increase training resolution
Figure 1: Top: Fourier convolution operator in FNO. After the Fourier transform F, the layer first truncates the full set of Iroquencies to the lowest ones using a dynamically set truncation before applying learnable linear trausformation (blue) and finally mapping the frequencies back to the linear space using the inverse FFT F The previous method (Li et al. a picks a fixed K throughout the entire training. The FNO architecture does not include the incremental algorithm and resolution. Bottom: full iFNO architecture. Our model takes as input functions at different resolutions (discretizations). The operator consists a lifting layer, followed by a series of iFNO blocks. The loss is used by our method to dynamically update the input resolution and the number of modes K in the Spectral Convolutions. The incremental algorithm detailed in section and algorithm
function spaces. It approximates the action of the solution operator any input function, allowing it to generalize across different discretizations and resolutions.
Unlike conventional neural networks, FNO possesses a property termed discretiaation-coneeree can oulput predictions at different resolutions, and those predictions converge to a unique solution upon mesh refinement. The diseretizationncooneeeneee property relies on kernel integration, which, in the case of FNO, is realized using the Fourier transform. A series of such spectral layers, along with normalization layers and non-linear activations, compose the core of the FNO architecture. For many PDE systems, like fluid flows, there is decay of energy with frequency, i.e., low-frequency components usually have larger magnitudes than high-frequency ones. Therefore, as an inductive bias and a regularization procedure to avoid overfitting, FNO consists of a frequency truncation function Tk in each layer that only allows the lowest K Fourier modes to propagate to the next layer, and the other modes are zeroed out, as shown in Figure
mcaning
It thus crucial to select the correct number of frequency modes K in various spectral layers in FNO to achieve optimal generalization performance. Too few mumber of modes leads to underfitting, while too many can result in overfitting. Further, we need to train on high-cnough resolution data to capture all the physical dynamics and approximate the ground-truth solution operator faithfully.
Our approach: We propose the Incremental Fourier neural operator (iFNO) to address the above chal- lenges. iFNO incrementally and automatically increases the number of frequency modes K and resolution R during training. leverages the explained ratio a metric to dynamically increase the number of frequency modes. The explained ratio characterizes the amount of information in the underlying spectrum that the current set of modes explain. A small explained ratio (i.e., below fixed threshold) indicates that the current modes are insufficient to represent the target spectrum and that more high-frequency modes should be added, as illustrated Figure Simultaneously, iFNO also starts by first. training on low-resolution data and progressively increases the resolution, akin to the principle of curriculum learning, thereby increasing training efficiency.
Our contributions are summarized follows:
1. We introduce the Incremental Fourier Neural Operator (iFNO), a novel approach that progressively increases the number of spectral coefficients and the data resolution used during training. These can applied jointly or individually.
2

---

Published in Transactions on Machine Learning Research (09/2024)
2. We perform a thorough empirical validation across range of partial differential equation (PDE) problems, including Burgers, Darcy Flow, the Navier-Stokes Equation, and the Kolmogorov Flow.
3. Using our proposed iFNO, we demonstrate uplo 38% lower testing error by acting as a dynamical spectral regularization scheme, using 20% fewer frequency modes compared to the existing FNO, while also achieving 46% faster performance and 2.8x reduction in model size, enabling larger scale simulations.
2 Related Works
The applications of neural networks partial differential equations have rich history (Lagaris et al. 1998;
Dissanayake & Phan- Thien
1994p. These deep learning methods be classified into three categories: (1) ML-enhanced numerical solvers such as learned finite element, finite difference, and multigrid solvers [Kochkov et al. 2021; Pathak et al.
2021 Greenfeld et al. 2019) (2) Network network-based solvers such (PINNs), Deep Galerkin Method, and Deep Ritz Method (Raissi et al., 2019 Sirignaoo & Spiliopoulos, 2018 Weinan & Yu 2018): and (3) the data-driven surrogate models such
as Physics-Inifee Informed Neural Networks
as QGuo ct al., [216)] Zhu & Zabaras, 2018 Bhatnagar et al., 2019). Among them, the machine lcarning surrogate models directly parameterized the larget mapping based on the dataset. They do not require a priori knowledge of the governing system and usually enjoy light-fast fast inference speed. Recently, novel
class of surrogate models called neural operators was developed (Li et al. b Lu et al.) 2021 Kissas et al. 2022). Neural operators parameterize the PDEs' solution operator in the function spaces and leverage its mathematical ical structures in their architectures. Consequentiall ially, neural operators usually have a better empirical performance (Takamoto et al. 2022 and theoretical guarantees Kovachki et al. (0021) combined with conventional deep lcarning models.
The concept. of implicit spectral bias was first proposed by Rahaman al. as possible explanation for the generalization capabilities of deep neural networks. There have been various results towards theoretically understanding this phenomenon [Cao et al.] Basri et al. Fridovich-Keil et al.] further propose methodologies to measure the implicit spectral bias on practical image classification tasks. Despite a wealth of theoretical results and empirical observations, there has been no prior research connecting the implicit spectral bias with FNO to explain its good generalization across different resolutions.
The notion of incremental learning has previously been applied to the training of PINNs Krishnapriyan et al. (2021); Huang & Alkhalifah| However, those focus single PDE instance and apply incremental learning to the complexity of the underlying PDE, e.g., starting with low-frequency wavefields first and using high-freeu high-frequency ones later Huang & Alkhalitah Our work is orthogonal to this direction, and rather than modifying the underlying PDE, we directly incrementally increase the model's capacity and the data's 's resolution.
3 Fourier Neural Operator
Fourier Neural Operators belong to the family of neural operators, which are formulated as a generalization of standard deep neural networks to operator setting (Li et al. b). A neural operator learns a mapping between two infinite dimensional spaces from finite collection of obscrved input-output pairs. Let D be a bounded, open set and A and U be separable Banach spaces of functions of inputs and outputs.
We want to learn neural operator Go Ax + U that maps any initial condition E A to its solution u E U. The neural operator Go composes lincar integral operator K. with pointwise non-lincar activation function a lo approximate highly non-linear operators.
Definition 3.1 (Neural operator). The neural operator Go is defined as follows:
G. = Q o (Wr. + K..) o a (Wi + o
where P, Q are the pointwise neural networks that encode the lower dimension function into higher dimen- sional space and decode the higher dimension function back to the lower dimensional space. The model stack L layers of (W1 + KG) where Wi are pointwise linear operators (matrices), K₁ are integral kernel operators, and a are fixed activation functions. The parameters 0 consist of all the parameters in P, Q, Wi, Kr.
3

---

Published in Transactions on Machine Learning Research (09/2024)
Input.
input
Prediction
Frediction
20
20
20
30
00
60 20
40
Figure 2: FNO with higher frequency modes captures stnalerr-scale structures in the fluid. Prediction of the Darcy flow by FNO with = 10 and K 90. Insufficient modes lead to overly strong dumping and fail capture the finer details in Darcy flow.
Li et. al. proposes FNO that adopts a convolution operator for IC as shown in Figure which obtains state-of-the-as thr-r results for solving PDE problems.
Definition 3.2 (Fourier convolution operator). Define the Fourier convolution operator as follows:
(KCu,) (x) = (R Tk (x) Vo E D,
where FF and JF1 are the Fourier transform and its inverse, R is learnable transformation and Tk is a fixed truncation that restricts the input to lowest K Fourier modes.
FNO diseretization-convergoe such that the model can produce a consistent solution for any query points, potentially not in the training grid. In other words, FNO can be trained on low resolution and evaluated high resolution. This property is highly desirable it allows transfer of solutions between different grid resolutions and discretizations.
Frequency truncation. To ensure disrretization-oovvrgeee FNO truncates the Fourier series Fo at a maximal number modes K using Tk. In this case, for any discrete frequency mode {l,...,K' we have Fu(k) E CC and R(k) CCxC, where C the channel dimension of the input v. The size of linear parameterization R depends on K. For example, R has the shape of Kd x C C for d-dimensional problem. We also denote the modes 1,... K as the effective frequency modes. In the standard FNO, Li et al. (a) view K as additional hyperparameter to be luned for each problem.
Frequency strength. As R(k) reflects how the frequency mode is transformed, we can measure the strength of transforming the frequency mode &o by the square the Frobenius norm of R(k), such that:
S = ΣΣ R
where Sy denotes the strength of the k-th frequency mode. A smaller Sy indicates that the k-th frequency mode is less important for the output.
Implicit spectral bias in neural networks It has been well studied that neural networks implicitly learn low-frequency components first, and then learn high-frequency components a later stage Rahaman et al. Xu et al. (0119). This phenomenon is known as implicit spectral bias, and helps explain the excellent generalization ability of overparameterized neural networks.
Since FNO performs linear transformation R in the frequency domain, the frequency strength Ss of each frequency mode is directly related to the spectrum of the resulting model. As FNO is neural network and trained by first-order learning algorithms, it follows the implicit spectral bias such that the lower frequency modes have larger strength in R. This explains why FNO chooses to preserve set containing the lowest froquency modes, instead of any arbitrary subsct of froquencics.

---

Published in Transactions on Machine Learning Research (09/2024)
Low-frequency components in PDEs Learning frequency modes is important as large-scale, low-frequency compo- nents usually have larger magnitudes than small-scale, high- froquency components PDEs. For dissipative systems (with diffusion terms) such as the viscous Burgers' equation and in- compressible Navier-Stokes equation, the energy cascade in- volves the transfer of energy from large scales of motion the small scales, which leads to the Kolmogorov spectrum with the
Spectrum of Kolmogorov Flows
10t0
A-1 scaling
10p
ground truth
10
Energy
slope of -5 in the inverse cascade range (Figure 30, and k-3 in the direct-cascade range (Boffetta et al. 2012). The smallest scales in lurbulent flow is called the Kolmogorov mieroscales. Therefore, one should choose the model frequencies with re- spect to the undorlying equation frequencies when designing machine learning models. It would be challenge to select the correct model frequencies in advance, without knowing the properties of the underlying PDEs.
10
10a
10"
10 Wave Number
10P
Figure 3: The spectrum of Kolmogorov flow decays exponentially with the Kolmogorov scale of 5/3 in the inverse cascade range.
Algorithm Incremental Learning of FNO
Input: Initial modes Ko, mode buffers b, threshold a, epoch e increase resolution, downsample factors r, data resolution Rdato
Initialize: Randomly initialize Ru E C(Ko+b)xXxX, set initial resolution Rewrr Rdata/ra
For iteration in I::
compute St = S2,..., ( Sk. {compute the frequency strength for each mode)
Kg + K,-1 While g(K,, st) < a: ffind K2 that explains at least a of $t}
K, 4- K,+1
construct R +bKx+))XXXC where Re(0 (Kz-1 + b), R and initialize the rest randomly If (t mod c e 0)
Sel next resolution Rewrr = R Train FNO model with (Ke, Rearr)
4 Incremental Fourier Neural Operator
Although FNO has shown impressive performance in solving PDEs, the training remains challenging. In this section, we discuss the main difficulties in training FNOs and propose the Incremental Fourier Neural Operator (IFNO) to address the challenges.
4.1 Incremental frequency FNO
While frequency truncation Tx ensures the dissretizaaiioninvariinn property of FNO, it is still challenging to select an appropriate number of effective modes K, as it is task-dependent and requires careful hyperpa- rameter tuning.
Inappropriate selection of can lead to severe performance degradation. Setting K too small will result too few frequency modes, such that it docsn't approximate the solution operator well, resulting underfit- ting. As shown in Figure 2 in the prediction of Darcy flow, FNO requires higher frequency modes to capture small-scale structures in the fluid for better performance. On the other hand, larger K with too many effective frequency modes may encourage FNO to interpolate the noise in the high-frequency components, leading to overfitting.
Previously, people have chosen the number of effective modes in FNO by estimating the underlying data frequencies or using heuristies borrowed from pscudo-spectral solvers such as the 2/3 dcaliaising rule (Hou & Li (207)), i.c., picking the max Fourier modes as 2/3 of the training data resolution. However, it's usually not easy to estimate the underlying frequency, and the resolution of data may vary during training and testing.
5

---

Published in Transactions on Machine Learning Research (09/2024)
Layer
Layer
Layer
Layer
low
1.90
2.3
2.0
Strength
15
frequency
2.00
1.0
high
1.9 Iterations
32
Icerations
ieeration
Iterations
Figure 4: Frequency evolution of first and fourth Fourier convolution operators in FNO and IFNO during the training Burgers' equation. We visualize FNO on the left figure and iFNO on the right figure for each layer. Each frequency strength S₃ is visualized across training. FNO tuned with the optimal weight decay strength.
These problems lead hand-tuning or grid-searching the optimal effective modes, which can be expensive and wasteful. We propose the incremental Fourier Neural Operator (iFNO) that starts with a "small" FNO model with limited low-frequency modes and gradually increases K and resolution I based on the training progress or frequency evolution. One of the key challenges in training iFNO is determining an appropriate time to proceed with the increase of K. We consider two variants of iFNO (method), which mcans we only increase the frequency modes based on the method.
iFNO (Loss): A common practice is to use the training progress a proxy to determine the time, as seen in previous work such as Liu et al.] Specifically, we let the algorithm increase K adaptively only when there is a decrease in training loss between N consecutive epochs that is lower than a threshold e. We denote it as the iFNO (loss) algorithm. Despite its simplicity, this houristic has shown be effective in practice. However, finding an appropriate and N is challenging as it depends on the specific problem and the loss function. To address this issue, propose novel method to determine the expanding time by directly considering the frequency evolution in the parameter space.
iFNO (Freq): As shown in Equation [ i-th frequency strength S, reflects the importance of the i-th frequency mode in the model. The entire spectrum of the transformation R can be represented by the collection of frequency strengths (when is sufficiently large):
sp = [S1. Se,,....S.p Spl:
(2)
where sp E IR, When applying the frequency truncation Tk (K < p), we can measure how much the lowest K frequency modes explain the total spectrum by computing the erplanation ratio g g(K,sp) sp).
K
S
g( (K, 8p)
-=
(3)
2 S
We define a threshold a, which is used to determine if the current modes can well explain the underlying spectrum. If 9(K, sp) > a, it indicates that the current modes are sufficient to capture the important information in the spectrum, and thus no additional modes need to be included in FNO. Otherwise, more frequency modes will be added into the model until sp) > a is satisfied.
Although s, reflects the entire spectrum when p is sufficienlly large, maintaining a large p is unnecessary and computationally expensive. Instead, we only maintain a truncated spectrum 8K+0 E RRK+ which consists of K effective modes and buffer modes. The buffer modes contain all mode candidacics that potentially would be included as effective modes in the later stage of training.
In practice, at iteration with K, effective modes, construct transformation R2 C(K++b)xCXC for ID problems. After updating RS at itcration + 1, we will find Ket1 that satisfics ((K+1,8)) > a. In this way, we can gradually include more high-frequency modes when their evolution becomes more significant, as illustrated in Fi We denote this variant as iFNO (Freq). We also describe how iFNO (Freq) is trained for solving 1D problems in Algorithm [ iFNO extremely efficient as the training only requires part of frequency modes. In practice, it also converges to a model requiring fewer modes than the bascline FNO without losing any performance, making the model efficient for inference. Notably, the cost of computing the

---

Published in Transactions on Machine Learning Research (09/2024)
frequency strength and explanation ratio is negligible compared to the total training cost. It can be further reduced by performing the determination every T iterations, which does not affect the performance.
4.2 Regularization in FNO training
As discussed in the previous section, learning low-frequency modes is essential for the successful training of FNO. However, the standard regularization techniques used in training neural networks are not capable of explicitly promoting the learning of low-frequency modes.
As shown Figure 1 although FNO with well-tuned weight decay strength successfully regularizes the high-frequency modes, its regularization can be too strong for certain layers (such as the fourth operator in the figure). This leads to the instability of the frequency evolution, which damages the generalization performance, as shown in Table 2 On the other hand, insufficient decay strength cannot regularize the high-frequency modes and can even lead to overfitting the associated high-frequency noise.
Dynamic Spectral Regularization: However, we find that iFNO can be served as a dynamic spectral regularization process. As shown in Figure | iFNO properly regularizes the frequency evolution without causing instability or overfitting sigh-frequency modes. As we will present in the experiment section, the dynamic spectral regularization gives iFNO significant generalization improvement.
4.3 Incremental resolution FNO
Finally, we introduce the incremental algorithm on the resolution part of our approach.
Computational cost in training large-scale FNO Training the FNO on large-scale, high-resolution data to solve PDEs with high-frequency information, a large effective mode number is necessary to capture the solution operator. However, this requires constructing a largo-scale transformation R, which dominates the computational cost of all the operations in FNO. This makes the training and inference more computationally expensive. For example, the Forecastnet (Pathak al.] 2022) for global weather forecast based on FNO is trained on 64 NVIDIA @ Tesla A100 GPUs, only covering few key variables. To simulate all the weather variables, it potentially needs the parallel FNO, which scales to 768 GPUs (Grady II et al. 2022
Incremental Resolution: To mitigate the computational cost of training FNO on high-resolution data, incorporate curriculum learning to improve our model (Soviany et al. (0022)) This approach involves training the models in logical sequence, where easier samples are trained first and gradually progressing to more challenging ones. In our case, we start with lover-resolttion images, allowing the model to capture large-scale features first, potentially accelcrating learning and reducing computational costs in the carly stages of training, and then moving on to higher-resolution images. This approach allows the model to initially focus on learning low-frequency components of the solution using small data, avoiding wasted computation and overfitting. Then, the model progressively adapts capture higher frequency components the data resolution increases. This model combines both incremental frequency (specifically the iFNO Freq)) and incremental resolution and is denoted by iFNO. Algorithm T showcases the joint implementation.
Table 1: Evaluation on Re5000 dataset on full data regime. The average training L2 loss and testing L2 loss on super-resolution are reported on runs with their standard deviation.
Method
Train L2
Test L2 (Super-Res Res) (1e-1) Training Time (Mins)
iFNO
0.149 + 0.004
0.961 + 0.022
670 + 40
Standard FNO
0.169 + 0.004
0.988 + 0.031
918 + 10
iFNO (Freq)
0.163 0.003
1.019 + 0.031
700 + 30
IFO (Loss)
0.155 + 0.005
0.948 + 0.040
690 + 30
iNN
(Resolution only)
0.155 + 0.003
0.974 + 0.010
668 + 40

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: Evaluation on various PDEs in the low-data regime. The average training and testing L2 loss are reported with their standard deviation 3 repeated All iFNO variants compared with standard FNO with fixed K 10, 30, 60, 90 mumber of modes. Losses have been divided by the scale under the column.
Method
Burgers
Darcy Flow
Navier Stokes (FNO2D)
Train (1e-3) - 3)
Test (le - 2)
Train (le- -00-3) 3)
Test (1e-2)
Train (1e - 2)
Test (1)
iFNO
0.418 = 0.002
0.139 = 0.006
2.466 + 0.855
0.129 + 0.028
3.533 = 0.003
0.106 + 0.011
IFN (Freq)
0.422 + 0.001
0.141 = 0.001
1.491 + 0.298
0.215 + 0.010
1.548 = 0.029
0.177 + 0.008
INO (Loss)
0.430 + 0.001
0.140 = 0.007
2.262 + 0.985
0.198 + 0.029
2.466 = 0.092
0.202 + 0.020
INN
(Res)
0.423 = 0.001
0.146 = 0.006
1.410 + 0.020
0.123 + 0.001
3.576 = 0.021
0.105 - 0.001
FNO (10)
1.135 = 0.002
0.270 = 0.006
1.471 + 0.223
0.162 0.003
5.133 = 0.182
0.166 + 0.027
FNO
(30)
0.455 + 0.001
0.156 = 0.002
1.462 + 0.102
0.140 + 0.011
3.262 = 0.001
0.154 + 0.001 -
FNO
(60)
0.421 + 0.001
0.153 = 0.006
1.468 + 0.212
0.130 + 0.007
FNO
(90)
0.423 + 0.001
0.146 = 0.006
1.425 + 0.212
0.127 + 0.004
-
Method
Navier Stokes (FNO3D)
Kolmogorov Flow
Train (le - 2)
Test (le 1)
Train (1)
Test (1)
IFNO
1.694 0.009
0.308 + 0.018
0.832 + 0.279
0.358 + 0.009
IFNO (Freq)
1.402 0.001
0.559 + 0.011
1.455 + 0.008
0.487 + 0.004
IFNO (Loss)
0.832
0.001
0.281 + 0.011
0.740 + 0.230
0.381 0.010
IFO (Res)
1.704 0.032
0.519 0.016
0.882 + 0.067
0.342 0.001
FNO (10)
1.892
0.020
0.002 0.527
2.111 + 0.009
0.543 = 0.009
FNO (30)
1.298 0.001
0.544 0.001
1.053 + 0.190
0.384 0.010
FNO (60)
0.562 + 0.001
0.348 0.001
FNO (90)
0.475 + 0.001
0.343 0.001
5
Evaluations
In this section, detail the experimental setting, datasets used and analyze the empirical results.
5.1 Experimental Setup
We evaluate iFNO and its variants on five different datasets of increasing difficulty. We retain the structure of the original FNO, which consists stacking four Fourier convolution operator layers with the ReLU activation and batch normalization. More specifically, for the time-dependent problems, we use an RNN structure in time that directly learns space-tind") The initial and coefficient conditions are sampled from Gaussian random fields (Nelsen Stuart, 2021 for all the datasets.
Burgers' Equation We consider the 1D Burgers' equation, which is a non-linear PDE with various applications, including modeling the flow of a viscous fluid. The 1D Burgers' equation takes the form:
d.l(x, () O. (u² (x, 0)/2) = vdrnu(E,t) f). u( 0) uz( x (0,1)
EE (0,
where un is the initial condition and v is the viscosity coefficient. We aim to learn the operator Go, mapping the initial condition to the solution. The training performed under the resolution of 1024, and we consider the learning rate halved every 50 epochs and weight decay to be 0.0001.
We share our code the neural operator codebase
8

---

Published in Transactions on Machine Learning Research (09/2024)
Darcy Flow Next, consider the steady-state of the 2D Darey Flow the unit box, which is the second-order, linear, elliptic PDE:
(a(x)Vu(z)) = f(z), x E (0,1)² u(x) O, x E a(0, 1)²
With a Dirichlet boundary where a is the diffusion coefficient, and f the forcing function. We are interested in learning the operator mapping the diffusion coefficient to the solution. The resolution is set to be 421 x 421.
Navier-Stokes Equation We also consider the 2D+time Navier-Stokes equation for viscous, incom- pressible fluid on the unit torus:
a,l(r, t) + u(xx t) Vw((x, t) -/ww(1 f(x),
V V-u(r,t) = 0, x 10, w(x, 0) = wo( E (0,7)
where w = x a the vorticity, wi is the initial vorticity, v E I the viscosity coefficient, and [(x) (si(si((2721) + x2)) + cos(2 + x2))) the forcing function. We are interested in learning the operator mapping the vorticity up to time 10 to the vorticity up to some later time. We experiment with the hardest viscosity task, ie., le 1e-8 5. The equivalent Reynolds number is around 500 normalized by forcing and domain size. We set the final time T = 20 the dynamic becomes chaotic. The resolution is fixed to be 64 x 64.
5.
We consider solving this equation using the 2D ENO with an RNN structure in time or using the 3D FNO do space-time convolution, similar to the benchmark Li @a. As FNO requires significantly more parameters, evaluating 3D FNO with more than 30 modes is computationally intractable due to the hardware limitation.
Kolmogorov Flow: We evaluate the models on a very challenging dataset of fighfrreyency Kolmogorov Flow. The Re5000 problem is the most challenging dataset we consider. It is governed by the 2D Navier- Stokes equation, with forcing sin)=sin(nn2)) ( $1, where 21 is the unit vector and a larger domain size [0, 2m]. as studied (Li et al. 2021 Kochkov el. al. 2021 More specifically, the Kolmogorov flow (a form of the Navier-Stokes equations) for viscous, incompressible fluid,
Ou
= u Vu Pu- Au + sin(ny) Re
V vuu= -u=00 0,
0t
2 2 (0, oo)
with initial condition 0)- uo where u denotes the velocity, p the pressure, and Re > 0 is the Reynolds number. It has an extremely high Reynolds number with Re = 5000. The resolution of this dataset is 128 x 128. We consider learning the evolution operator from the previous time step to the next based on 2D FNO. We also consider variation of the dataset with a lower Re value, which we call the Kolmogorov Flow problem. This resolution is fixed to be 256 x 256 to show high-frequency details.
Methods We present results for each dataset where consider four standard FNO baselines with different numbers of frequency modes K 10,30, 30, 60, 90. For each dataset, all methods are trained with the same hyperparancters, including the Icarning rate, weight decay, and the number of epochs. Unloss otherwise specified, we use the Adam optimizer to train for 500 epochs with an initial learning rate of 0.001, which is halved every 100 epochs, and weight decay is set to be 0.0005. For the re5000 dataset, we train for epochs, and experiment details are explained in the Appendix. All experiments are run using NVIDIA@ @ Tesla @ V100/A100 /1100 GPUs. We also test our incremental methods on Tensorized Factorized Neural Operators Kossaifi et al. (023) by sweeping over different. factorization schemes and ranks for the weights.
9

---

Published in Transactions on Machine Learning Research (09/2024)
Hyperparameter settings We find iFNO is less sensitive to the choice of lyperparaaeteers across different datasets. This property allows the same iFNO architecture and hyperparameteess to be directly applied to a wide range of problems without expensive parameter validation and without needing to set an appropriate K. Consequently, we use the same hyperparameters for all datascts, including the number initial modes Ko = 1, the number of bulfer modes b 5, and the threshold a = 0.9999. We also use a uniform threshold 0.001 in iFNO (Loss).
5.2 Experiment details for Re5000 dataset iFNO
Regarding setting the scheduler, the incremental resolution approach significantly alters the loss landscape and distribution of the training data each time the resolution is increased. As noted by Smith (017)) cyclic learning rates oscillating between higher and lower values during training can be highly beneficial when dealing with such changing loss landscapes. Therefore, adopts combination of the cyclic learning rate and step learning rate scheme. We that this combina- tion provides improved and more stable convergence throughout the curriculum of incremental resolution increases, contributing to strong empirical results for both the incremental resolution model and the iFNO model. A triangular cyclic learning rate allows the model to first explore with larger learning rates to find new optimal parameters after a resolution increase before narrowing down with smaller rates to converge. This prevents the model from getting stuck in sub-optimal solutions. Our experiments utilize 3 different modes, i.e., triangular, triangular with decreased height, and exponential decay. We also sweep over various step sizes and maximum and minimum bounds. The cyclic period is set to epochs. Although the cyclicLR method is indeed suitable, we find that after we switch to the highest resolution, using stepLR converges to a better minimum compared to cyclicLR. This makes sense, once we start training on the highest resolution, instead of changing our learning rates in an oscillatory manner, we should stabilize and take smaller steps for convergence, like how the bascline FNO is trained.
incremental resolution training methodology
Our algorithm increases resolution every e epoch from Fast Fourier Transform speed. The hyperparameter can be tuned, and list determining how low we sub-sample data. In the re5000 dataset case, have set 10 and [4, 2, 1], which results in starting from resolution 32 all the way up to the highest resolution 128.
and then 128. We use powers of 2 for optimal
5.3
Results
IFNO (Freq)
90
Number of requeecy Modes
(Freq)
dation Loss
80
Training
Number Modes
Loss
60
Number
50
0.06
0.04
30
Burgers
Darcy
NS:2D
N5:30
Kalmogorov
(a) Number of frequency modes K in the converged FNO and iFNO models across datasets. We report in the first Fourier convolution operator. NS denotes Navier-Stoker equations.
Epochs
(b) Testing loss, training loss, and number of modes K during the training of FNO and iFNO Kol- mogorov flow.
Figure iFNO comparison, both modes and loss curves.
10

---

Published in Transactions on Machine Learning Research (09/2024)
Table 3: Evaluation on various PDEs in the full data regime, The average training and testing L2 loss are reported with their standard deviation over 3 repeated runs. Frequency-based and loss-based iFNO are compared with standard FNO with fixed K- = 10, 30, 60, 90 number of modes. Losses have been divided by the scale under the column.
Method
Burgers
Darcy Flow
Navier Stokes Train (1e - 1)
(FNO 2D)
Train (1e-) - 3)
Test (le - 3)
Train (le -3) 3)
Test (1e-3)
Test (1e-1)
iFNO
0.891 = 0.022
0.760 = 0.039
4.580 + 0.855
0.264 = 0.020
1.022 = 0.043
0.824 + 0.016
IFNO (Freq)
0.898 + 0.022
0.778 = 0.005
2.453 + 0.260
0.387 + 0.009
1.102 + 0.132
1.265 + 0.044
INN (Loss)
0.919 + 0.033
0.779 = 0.034
3.113 + 0.887
0.322 + 0.068
1.135 + 0.003
1.375 0.224
IFNO (Res)
0.979 = 0.053
0.854 = 0.034
4.463 + 0.920
0.271 0.010
1.112 = 0.080
0.737 = 0.227
FNO (10)
2.523 = 0.028
1.063 = 0.025
2.991 + 0.302
0.481 = 0.047
10.66 = 0.736
1.286 + 0.178
FNO
(30)
1.032 + 0.017
0.868 = 0.033
2.981 + 0.046
0.412 + 0.029
1.223 + 0.045
1.184 + 0.107
FNO
(60)
0.966 + 0.023
0.845 = 0.016
2.596 + 0.433
0.466 + 0.013
1.112 + 0.029
1.178 + 0.047
FNO
(90)
0.973 + 0.042
0.851 = 0.029
2.521 + 0.046
0.379 + 0.043
1.155 + 0.001
1.577 + 0.002
Method
Navier Stokes (FNO 3D)
Kolmogorov Flow
Train (le - 2)
Test (le - 1)
Train (le - 2)
Test (le 1)
IFNO
0.820 = 0.041
0.272 0.003
2.665 0.034
1.507 = 0.009
iFNO (Freq)
0.855 + 0.007
0.262 + 0.002
4.345 + 0.001
1.443 + 0.001
IFNO LLsss)
0.832 + 0.001
0.281 + 0.011
3.393 + 0.001
2.684 + 0.003
IFNO (Res)
7.533 + 0.006
0.281 + 0.001
2.336 + 0.003
0.985 + 0.001
FNO (10)
1.035 = 0.027
0.296 0.016
5.462 0.004
2.227 = 0.012
FNO (30)
0.977 + 0.001
0.587 = 0.001
3.571 + 0.002
1.312 + 0.012
FNO (60)
2.751 #0.001
1.015 #0.001
FNO (90)
2.330 + 0.002
1.042 + 0.002
First, we study the performance of our model on the large-scale Re5000 dataset, where we use 36000 training samples and 9000 testing samples. The results in Table [ show that iFNO achieves significantly better gencralization performance than just incrementally inereasing either frequency or resolution alone. This highlights the advantages of the iFNO while gaining around a 30% efficiency and much better accuracy than the stand-alone Incremental methods. The training time is reduced due to starting with very small model and input data. There is one important note to mention that, in fact, for the Re5000 dataset, since the problem is highly challenging, i.e., having high Reynolds number, need to reach all 128 modes to capture even the high-frequency components.
Next, we evaluate all methods in the low-data regime across different datasets, where the models only have access to a few data samples. This is common in many real-world PDE problems where accurately labeled data is expensive obtain. In this regime, the ability to generalize well highly desirable. We consider a small training sot (5 to samples) for Burgers, Darcy, and Navicr-Stokes datascts. As the data trajectories are already limited in the Kolmogorov dataset, we choose 8x larger time step (sampling rate) for each trajectory, resulting in fewer time frames.
As shown in Table 2 iFNO consistently outperforms the FNO basclincs across all datasets, regardless of the number of modes K. It also shows that FNO with larger K achieves lower training loss but overfits the training data and performs poorly on the testing set. On the other hand, iFNO (Freq) achieves slightly higher training loss but generalizes well to the testing set, which demonstrates the effectiveness of the dynamic spectral regularization. We also notice that although iFNO (Loss) has significant fluctuations, especially in the training results, it does achieve better generalization on most datasets against the baselines.
11

---

Published in Transactions on Machine Learning Research (09/2024)
5.4 Full Data Results
We also evaluate the models with the most data samples available in each dataset. As shown in Table 3 IFNO achieves good performance comparable with standard FNO.
1. Generalization: iFNO achieves up to 38% better generalization performance than the base- lines, and all iFNO methods beat the baselines in testing error across tasks like Burgers, Darcy, NS2D+time, and NS3D.
2. Efficiency: iFNO exhibits remarkable efficiency gains, ranging from 21% to 46% reduction in computational time compared to the baselines, depending the lask.
Model Size: iFNO achieves substantial model size reductions, with up to 2.8x smaller model size Ihan the baselines, without compromising performance.
5.5
Ablation Studies
Layer
Layer
Layer
low
1.75
2.5
1.50
Strength
1.25
frequency
1.00
1.5
0.75
1.0
0.0
0.0-
high
100
200
500
100
200
300
100
200 Iterations
300
Iterations
Iterations
Layer
Layer
Layer
low
3.0
Strength
frequency
high
120
100
200
100
200
300
Iterations
lterations
lterations
Figure 6: Frequency evolution of the Fourier convolution operators in FNO with and without weight decay. Top: with weight decay; bottom: without weight decay. When training with weight decay, the strength of each mode converges around 100 training iterations. And the weights of lower frequency tend to have high strength. On the other hand, the weight continues growing throughout the training without the weight decay. In this case, higher frequencies can also have high strength.
Figure 5b compares FNO (90) and iFNO (Freq) over the training on Kolmogorov flow. shows that iFNO requires much fewer frequency modes during training. As the linear transformation R dominates the number of parameters and FLOPs in FNO when K is large, iFNO (Freq) can significantly reduce the computational cost and memory usage of training FNO. iFNO achieves better generalization and even requires fewer modes at the end of training. In Figure 5a we compare iFNO (Freq) with FNO with a particular K that achieves the best performance in each dataset. The results suggest that the trained iFNO (Freq) consistently requires fewer frequency modes than its FNO counterpart, making the iFNO inference more efficient. This also indicates that iFNO can automatically determine the optimal number of frequency modes during training without predetermining K an inductive bias that could potentially hurt the performance. To further demonstrate the adaptability and broader applicability of incremental approach, we conducted case
12

---

Published in Transactions on Machine Learning Research (09/2024)
study applying iFNO to Tensorized Fourier Neural Operators (TFNOs) Kossaifi et al. 2023 TFNOs represent recent advancement in efficient operator learning, and we were interested in exploring how our incremental techniques could enhance their performance. The full details of these experiments are presented in Appendix B.6 In summary, we show that TENOs with our incremental approach (iTFNO) achieve up to 11% better generalization error and up to 22% performance speed up compared to baselines.
6 Standard Regularization in training FNO
We also visualize the frequency evolution of FNO and iFNO. As shown in Figure G the weight decay used the optimization has a significant effect on the learned weights. When the weight decay is active, the strength of weights will converge around the first 100 iterations of training. Further, the strength of frequencies is ordered with respect to the wave numbers where the lower frequencies have high strength and the higher frequencies have lower strength. On the other hand, when training without weight decay, the strength of all frequency modes will continuously grow and mix up with each other.
7 Conclusion
In this work, we proposed the Incremental Fourier Neural Operator (iFNO) that dynamically selects fre- quency modes and increases the resolution during training. Our results show that iFNO achieves better generalization while requiring less number of parameters (frequency modes) compared to the standard FNO, We believe iFNO opens many new possibilities, such as solving PDEs with very high resolution and with limited labeled data and training computationally efficient neural operators. In the future, plan to apply iFNO more complex PDEs and extend it to the training of physics-informn neural operators.
8 Impact statement
This paper presents work with the goal of advancing the field of machine learning and PDEs. FNOs have shown promise to solve challenging physics and engineering problems accurately and efficiently. Our research has made FNOs more optimized through incremental learning and, hence, could lead to FNOs being applied to solve larger real-world probloms, allowing for advances fields like engincering, weather forecasting, and climate science while reducing training times with fewer computing resources. There might also be potential negative societal consequences of our work, but none of them are immediate be specifically highlighted here.
9 Acknowledgements
Robert Joseph George is supported by Caltoch Graduate Fellowship. Anima Anandkumar is supported by the Bren Named Chair, Schmidt AI 2050 Senior fellow, and ONR. (MURI grant N00014-18-12624).
References
Kamyar Azizzadenesheli, Nikola Kovachki, Zongyi Li, Miguel Lin-Schiaffnn,, Jean Kossaifi, and Anima Anandkumar. Neural operators for accelerating scientific simulations and design, 2024. URL https:
Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. URL
00425 type: article.
Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Prediction of acrodynamic flow fields using convolutional neural networks Computational Mechanics, 64(2):525-545, 2019.
Guido Boffetta, Robert Ecke, al. Two-dimensional turbulence. Annual review of fluid mechanics, 44(1): 427-451, 2012.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Yuan Cao, Zhiying Fang, Yue Wu, Dinn-XXunn Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. URL
org/abs/1112 01198
type: article.
MWMG Dissanayake and Nhan Phan-Thien Neural-network-based approdmations for solving partial dif- ferential equations. communications in Numerical Methods in Engineering, 10(3):195-20) 1994.
Sara Fridovich-Keil, Raphael Contijo-Lopes, and Rebecca Roelols. Spectral bias practice: The role of type: article.
function frequency generalization. URL httt:://arxiv.org///aa.../ htwgg//ab///10..../ 02424
Thomas J Grady II, Rishi Khan, Mathias Louboutin, Ziyi Yin, Philipp A Witte, Ranveer Chandra, Russell J Hewett, and Fclix J Herrmann. Towards large-scale Icarned solvers for paramctric pdes with model-paraliel fourier neural operators. arXie preprint urXiv:2204.014 01205, 2022.
Danicl Greenfeld, Mcirav Galun, Ronen Basri, Irad Yavnch, and Ron Kimmel. Learning to optimize multigrid pde solvers. In International Conference on Machine Learning, pp. 2415 2423. PMLR, 2019.
Xiaoxiao Guo, Wei Li, and Francesco lorio. Convolutional neural networks for steady flow approximation. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 481-48 190, 2016.
Thomas Y Hou and Ruo Li. Computing nearly singular solutions using pseudo-spectral methods. Journal of Computational Physics, 226(1):379 397, 2007.
Xinquan Huang and Tariq Alkhalifah. PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting. URL http://arxiv/ org/ aba/2109, 14536
Georgios
Kissas, Jacob Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George Pappas, and Paris Perdikaris. Learning operators with coupled attention. Journal of Machine Learning Research, 23 (215):1-63, 2022.
Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang, Michael P Bremner, and Stephan Hoyer. Machine learning accelerated computational fluid dynamics. arXie preprint #xXiv2212.001010 2021.
Jean Kossaifi, Nikola Kovachki, Kamyar Avizzaennshelli, and Anima Anandkumar. Multi-grid tensorized fourier neural operator for high-resolution pdes. arXiv preprint arXiv:2310.000 2023.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Avizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint wrXiv:2108.0848.. 2021.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, vol- ume 34, pp. 26548 26560. Curran Associates, Inc., 2021. URL pdd
http:///pooceeiingg..eerruug
Isaac E Lagaris, Aristidis Likas, and Dimitrios 1 Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9(5):987-1000, 1998.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations, a. URL http: http:://arxiv.ooom ieweiivorrg/aan http:/1as22010... 08895
type: article.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations, b.
URL http:///artv.orrgaa///00..0000
org/abs///00s 03485 03485
Zongyi Li, Nikola Kovachki, Kamyar Azimadencsheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stu- art, and Anima Anandkumar. Markov neural operators for learning chaotic systems. arXiv preprint arXiv:2106 06898, 2021.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Burigede Liu, Nikola Kovachki, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, Andrew M Stuart, and Kaushik Bhattacharya. A learning-based multiscale method and its application to inelastic impact problems. Journal of the Mechanics and Physics of Solids, 158:104668, 2022.
Qiang Liu, Lemeng Wu, and Dilin Wang- Splitting steepest descent for growing neural architectures. URL
attp://arxiv.org/ awtpaabss...///////. 19t1/199100..218688 02366
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218 229, 2021.
Nicholas H Nelsen and Andrew M Stuart. The random feature model for inpul-outpul maps belween banach spaces. SIAM Journal on Scientific Computing, A3(5) AA2443, 2021.
Jaideep Pathak, Mustafa Mustafa, Karthik Kashinath, Emmanuel Motheau, Thorsten Kurth, and Marcus Day. MI-pde: A framework for machine learning enhanced pde solver. Bulletin of the American Physical Sociely, 2021.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Fourcastnet: A global data-driven resoutonon weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202. 11214, 2022.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. URL http://arxxv.org// type: article.
1806. 08734
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics- neural networks: deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686 707, 2019.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: deep learning algorithm for solving partial differ- ential equations. Journal of computational physics, 375:1339-1364, 2018.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 464-472. IEEE, 2017.
Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum lcarning: A survey. Interna- lional Journal of Computer Vision, 130(6):1526 1565, 2022.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. arXiv preprinl arXiv:2210.07183 2022.
E Weinan and Bing Yu. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
Gege Wen, Zongvi Li, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M Benson. U-fno an enhanced fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources, 163:104180, 2022.
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint (rXie:1901.06522 2019.
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415-447, 2018.
15

---

Published in Transactions on Machine Learning Research (09/2024)
A Additional Results
A.1 Frequency mode evolution during training
As shown in Figure 7 the effective modes K adaptively increase during the training phase on the Burgers, Darcy, Navier-Stokes (2D & 3D formulation). and Kolmogorov Flows. For smooth problems such as viscous Burgers equation and Darcy Flow, the mode converges in the early epoch, while for problem with high froquency structures such as the Kolmogorov Flows, the mode will continuously increase throughout the training phase.
baper Layar
Durers Laper
Darant
Eugan
18:3
15.5
14:3
1321
121
Epacre
Lenty Loyer
Layer
Eency
200
23.00
112
314
312
30.50
11.
113
114
Excchs
256
Ereck
200
Socit
200
11.00
174
31.4
315
3..00
314
105
50.00
114
112
356
356
250
150
945
3.0
300
55
Figure Mode evolution during training for each layer across datasets.
16

---

Published in Transactions on Machine Learning Research (09/2024)
A.2 Training and testing loss during learning
Figure and 9 show the training curves and testing curves across the five problem settings, ten rows in total.
Overall, IFNO methods have a slightly higher training error but lower validation error, indicating its better generalization over the few data regime.
Marker heaiin Simple. 10
0.30
Repms Nendrr
100
011
Purgen Neekera Seare 140
9:5
Sunteral Teceeyeeeeo 1990
PSD
030
bugan haiiigg
0.30
413
Eurges
620
HSp
0.00
600
6.00
434
100
400
500
255
100
500
300
F1.0 Tequenc/orss
1.1179
casus
vauce
P1D Joc-dased
3
2 53155
50000s
Peatus
5000
350
314
135
310
201
100
Duccy
000000
Daicy
500011
casus
cauce
50000
33g
focda
318
135
219
201
100
101
faaking
ng
43
Movi Sadee Menica Nanaass
0.0
Mocer 10 1D Sande Tharng Kandss
Racn Stal Secar Ta Kanmi 333
Mow: Sabic
042
0.5
1951
210
541
Figure 8: Loss vs Epochs across datasets (part 1: Burgers, Darcy, Navier-Stokes 2D). Each top row: training L22 loss. Each bottom row: validation L2 loss
17

---

Published in Transactions on Machine Learning Research (09/2024)
Sowar stanes
1adee bunae wwg.aaasss 123
Rlaa stiue tariig haruus cada
Movor SA Sarbe far
Morke haamn Serpins
Rartr tokes Memper the Tirperlless 199
Raaie Nakio Manten Thrnee Sendies 503
Siverr Number Sanelo 1906
PSD
Exccs
210
435
dirigate-ttatee Thiiing 12000
025
kshnoparo-
HSD
0.5
1.35
6.8
0.00
600
625
mogoror- Mumber Thaiiig Samplo 49039
lomrepomw Numberat www.geaeennngnnanoonoooo 2000
otree of Thiiiig Samplo- 12000
Kahropamy- Mumber wwwhhr damm 5633
038
520
190
438
5.4
244
5.5
Epetre
Coce
Figure 9: Loss vs Epochs across datasets (part 2: Navier-Stokes 2D, Kolmogorov Flows). Each top row: training L2 loss. Each bottom row: validation L2 loss
B Additional Experiments and Results
B.1 Selection of Hyperparameters
In Tables 4 and 5 we present the results of our hyperparameter search for in iFNO (Freq) and in iFNO (Loss) respectively, across various tasks. These hyperparameters control the threshold for adding new modes in our incremental approaches. For iFNO (Freq), represents the cumulative energy threshold in the Explanation Ratio criterion. We explored values ranging from 0.6 to 0.9999, covering wide spectrum from more aggressive (lower a) to more conservative (higher a) mode addition strategies. For iFNO (Loss), represents the threshold for the relative decrease in loss that triggers the addition of new modes. We investigated values from 0.1 to 0.00001, encompassing both rapid (higher e) and gradual (lower e) mode addition schemes. For most tasks, moderate values of afaround 0.99) and e (around 0,001) tend to perform well, balancing between adding sufficient modes for expressiveness and avoiding overfitting. However, some tasks like Kolmogorov Flow benefit from more aggressive mode addition (lower a or higher c), likely due the complexity of the underlying dynamics.
18

---

Published in Transactions on Machine Learning Research (09/2024)
Table 4: Evaluation of a in iFNO (Freq) across various tasks.
Tasks
Selection of a
0.6
0.8
0.9
0.99
0.999
0.9999
Burgers (le-3)
1.723
0.975
0.583
0.544
0.559
0.587
Darcy Flow (1e-4)
0.621
0.595
0.552
0.509
0.475
0.513
Navier Stokes (FNO 2D)
0.341
0.256
0.223
0.189
0.194
2.010
Navier Stokes (FNO 3D)
0.524
0.499
0.461
0.464
0.468
0.471
Kolmogorov Flow
0.133
0.122
0.102
0.043
0.049
0.051
Table 5: Evaluation of e in iFNO (Loss) across various tasks.
Tasks
Selection of
0.1
0.01
0.001
0.0001
0.00001
Burgers (1e-3)
0.672
0.572
0.595
1,190
13.90
Darcy Flow (1e-4)
0.486
0.465
0.412
0.424
0.490
Navier Stokes (FNO 2D)
0.192
0.194
0.191
0.284
0.322
Navier Stokes (FNO 3D)
0.471
0.469
0.454
0.449
0.452
Kolmogorov Flow
0.0614
0.0616
0.053
0.578
0.579
B.2 FNO over different sizes
Table 6: Evaluation of changing the width of FNO models on Burger. Validation losses have been divided by 1e-3.
Models
Dimension
32
64
80
96
128
iFNO (Freq)
0.785
0.5901
0.532
0.469
0.513
iFNO (Loss)
0.746
0.708
0.605
0.578
0.714
FNO
0.603
0.519
0.525
0.501
0.553
Table 7: Evaluation of changing the width of FNO models Darcy. Validation losses have been divided by 1e-4.
Models
Dimension
32
64
80
96
112
iFNO (Freq)
0.462
0.462
0.4395
0.4304
0.4479
IFNO (Loss)
0.4549
0.4553
0.4599
0.3859
0.4321
FNO
0.4683
0.5051
0.4116
0.4922
0.4834
19

---

Published in Transactions on Machine Learning Research (09/2024)
Table 8: Evaluation of changing the width of FNO models on Navier Stokes 2D, Validation losses have been divided
by 1.
Models
Dimension
16
20
32
64
80
iFNO (Freq)
0.205
0.192
0.180
0.170
0.176
iFNO (Loss)
0.199
0.191
0.181
0.177
0.177
FNO
0.203
0.198
0.182
0.182
0.177
Table 9: Evaluation of changing the width of FNO models Kolmogorov Flow. Validation losses have been divided by le-1.
Tasks
Dimension
16
20
32
64
80
iFNO (Freq)
1.513
1.450
1.312
0.634
(NR)
iFNO (Loss)
0.879
0.821
0.654
0.534
(NR)
FNO
1.072
1.024
0.872
0.743
0.721
B.3
Incremental FNO under different initialization scales
Evaluation of different initialization scales of FNO/IFNO models on Kolmogorov Flow. We again do full sweep and report the hest values.
Table 10:
Tasks
Scale of Initialization
1e-2
lc-1
1
lel
le2
iFNO (Freq)
0.064
0.062
0.061
0.064
0.067
IFNO (Loss)
0.050
0.054
0.070
0.053
0.055
FNO
0.073
0.073
0.072
0.072
0.073
B.4 Standard FNO with different frequency modes
We conduct experiments to showcase the classic bias-variance tradeoff in FNOs when increasing the modes.
Table 11: Evaluation of different frequency modes on standard FNO on Kolmogorov Flow. In this task, the minimum value bolded across each row. Validation losses have been divided by the value next to the dataset across each row.
Tasks
Frequency Modes
10
20
30
40
60
80
90
100
120
Kolmogorov Flow (1e-1)
1.327
0.906
0.674
0.598
0.482
0.509
0.451
0.471
0.485
B.5 Number of Paramaters
We have compiled summary of our results comparing iFNO to FNO (90 modes) across various tasks:
Burgers equation:
Generalization: iFNO improves testing error by 11
20

---

Published in Transactions on Machine Learning Research (09/2024)
Efficiency: iFNO is 21
Model size: iFNO achieves a 1.3x reduction (7.9 million vs. 12 million parameters).
Darcy Flow:
Generalization: iFNO improves testing error by
Efficiency: iFNO is 44
Model size: iFNO achioves a 2.3x reduction (471 million vs. 1.061 billion parameters).
Navier-Stokes 2Dmt
Generalization: iFNO (Resolution) improves testing error by 38
Efficiency: iFNO is 38
Model size: iFNO achieves a 2.3x reduction (471 million vs. 1.061 billion parameters).
Navier-Stokes 3D:
Generalization: iFNO (freq) improves testing error by
Efficiency: iFNO is 31
Model size: iFNO achieves a 2.8x reduction (1.25 billion vs. 3.539 billion parameters).
Navier-Stokes High Resolution:
Generalization: iFNO (Resolution) improves testing error by
Efficiency: iFNO is 46
Model size: iFNO achieves a 2.3x reduction (471 million vs. 1.061 billion parameters).
These results demonstrate that iFNO consistently outperforms FNO (90 modes) in terms of generalization,
efficiency, and model size across various tasks. We chose to compare primarily with FNO (90 modes) as it represents the most challenging baseline. Other iFNO variants, while potentially larger than the base iFNO model, still maintain efficiency advantages over FNO.
Table 12: Evaluation and number of parameters (in Millions) for the various models in Table
Method
Navier Stokes (FNO 2D)
Kolmogorov Flow
Number of Parameters (Millions)
Test (le (le-1) 1)
Number of Parameters (Millions)
Test (le - 2)
iFNO (Freq)
0.92M
1.872
321M
4.602
iFNO (Loss)
23.8M
1.911
530M
5.127
FNO (10)
0.64M
1.941
6.57M
13.27
FNO (30)
5.76M
1.924
59.0M
6.462
FNO (60)
23.0M
1.918
235M
4.825
FNO (90)
51.8M
1.918
530M
4.596
B.6
Tensorized Factorized FNO under different tensor ranks
TTFN incorporates our incremental mode selection and incremental resolution techniques into the TFNO framework. Specifically, we apply the following modifications to TFNOs:
Incremental mode selection: Similar to iFNO, iTFNO starts with a small number of modes and gradually increases the number of modes during training. We experiment with both frequency- basod (iFNO (Freq)) and loss-basod (iFNO (Loss)) criteria for determining when add modes. This allows iTFNO adaptively adjust the model capacity and focus on the most relevant frequency components for the given task.
21

---

Published in Transactions on Machine Learning Research (09/2024)
2. Incremental resolution: iTFNO also adopts the incremental resolution technique from iFNO, where the model is trained on progressively higher resolutions of the input data. This enables ITENO to capture fine-grained details and scale to higher resolutions more efficiently.
3. Factorization schemes and ranks: In addition to the incremental techniques, we explore different factorization schemes and ranks for the tensor decomposition of the weights in iTFNO. This allows us to investigate the impact of the tensor structure on the performance and efficiency of the incremental approach.
Table 13: Evaluation of 2 different factorization ranks of iTFNO models on Re5000. We do full sweep and report the best values.
Weight Factorization
5%
25%
Test (1e-1)
Runtime (Mins)
Test (le-1)
Runtime (Mins)
iTFNO
0.646
122
0.793
259
iTFNO (Freq)
0.752
144
0.904
333
TFNO
0.719
166
0.872
280
B.7 Different Losses
We have also conducted our models on the H1 Loss. We acknowledge that the L2 loss, while commonly used, may not fully capture the desired properties and behavior of the learned operators. The L2 loss treats all parts of the signal equally and may not prioritize important structural or geometric features. Our proposed IFNO((req) and iFNOO(oss) methods are closely lied to controlling the L2 loss, either directly or indirectly.
We have conducted additional experiments using the Sobolev loss (H1). The Sobolev loss considers not only the pointwise differences between the predicted and target signals but also the differences in their derivatives or gradients. This loss is more sensitive to the smoothness and regularity of the learned operators and can provide insights into their ability to capture important spatial or temporal structures. We have evaluated our proposed methods, including iFNO, (Freg) iFNOO)oss), and iFFOOrres) as well as the baseline FNO and TENO models, on three datasets: Re5000, Navier-Stokes 2D, and Burgers. The results are presented in the tables above. For the Re5000 dataset, we observe that iTFNO outperforms both iTFNO(freq) and the bascline TFNO terms of H1 loss, indicating that the incremental approach combined with the lensor factorization can lead to improved accuracy in capturing the spatial regularity of the solutions. The performance gap is consistent across different tensor ranks (5 and 25).
Table 14: Comparison of iTFNO and TFNO
rank 5
rank 25
ITFNO
0.2396
0.2984
ITFNO (Freq)
0.2889
0.3321
TENO
0.2725
0.3496
In the Navicr-Stokes 2D datasot, iFNO and its variants (freO) iFNO(loss). irNO(res)) generally achieve lower H1 loss compared to the baseline FNO. This suggests that the incremental mode selection and resolution techniques can help learn operators that better preserve the smoothness and continuity of the fluid dynamics. Similarly, for the Burgers dataset, iFNO and its variants demonstrate competitive slightly improved performance in terms of H2 loss compared to the baseline FNO. The iFNO(res) variant, which focuses on incremental resolution, achieves the lowest H1 loss, highlighting the importance of adaptively refining the spatial resolution for this specific problem.
Finally, we include one more comparison on the NS2D dataset where we present results for training on L2 loss and evaluating both Ta2 and Hj losses, as well training on H, loss and evaluating both Tog and Hi
22

---

Published in Transactions on Machine Learning Research (09/2024)
Table 15: NS2D Results
Model
Test H1 Loss
IFNO
0.1511
iFNO Freq
0.1476
iFNO
Loss
0.1606
iFNO Res
0.1507
FNO
0.1619
Table 16: Burgers Results
Model
Test H1 Loss
iFNO
0.02913
iFNO Freq
0.02892
IFNO Loss
0.03005
iFNO Res
0.02675
FNO
0.03160
Josses. The incremental techniques in iFNO and its variants can lead to improved performance in L2 and Hi losses, demonstrating their effectiveness in capturing both pointwise accuracy and spatial regularities. The iFNO variants show different behaviors when trained on L2 versus H1 loss, suggesting that the incremental techniques interact differently with these loss functions.
Table 17: NS2D+Time Results
Model
Train (L2)
Eval (L2)
Eval (Hz)
Train (H₂)
Eval (L2)
Eval (H1)
IFNO
0.0524
0.0377
0.1284
0.0972
0.0831
0.1543
IFNO
Freq
0.0514
0.0369
0.1 0.1259
0.0963
0.1244
0.1476
iFNO
Loss
0.0504
0.0359
0.1242
0.1144
0.1181
0.1606
iFNO
Res
0.0546
0.0379
0.1316
0.1152
0.0881
0.1507
FNO
0.0517
0.0360
0.1258
0.1103
0.1584
0.1619
These results provide reassurance that proposed incremental techniques not only improve the L2 loss but also lead to better performance in terms of the Sobolev loss, which is more sensitive to the spatial or temporal regularity of the learned operators. However, we acknowledge that the Sobolev loss is just one example of an alternative metric, and some many other relevant losses and distances could be considered, such as Lı loss, Wasserstein distance, or problem-specific metrics. We believe that a comprehensive evaluation across multiple losses and datasets crucial for understanding the strengths and limitations of operator learning methods. We will investigate this in future work.
23


Reviews:
[{"review_id": "Cp5p4Vvwd1", "reviewer": "Reviewer_jqV8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe idea appears to be novel, and most experiments support that proposed techniques lead to decreased training time and better test error. The main body of the paper is well-written, and it is mostly possible to understand what the authors want to convey.\n\n**Weaknesses**\n\nThat said, the significance of accuracy improvement seems to be exaggerated. Some ideas are insufficiently explained. Besides, there are unclear parts (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Authors propose a special training protocol for Fourier Neural Operator (FNO) with two primal innovations:\n1. The number of modes is selected adaptively based on one of two proposed criteria (based on loss and based on \"frequency strength\").\n2. The spatial resolution of training data is gradually increasing during training.\nIt is shown in a set of experiments that these two novelties allow for a decrease in training time and improve test error.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I mostly find the present research to be of sufficient quality and merely want to clarify several things that confuse me and may lead to the confusion of the reader alike. Below is the list of questions and suggestions:\n\n1. (page 1) \"Recently, deep learning has shown promise in solving partial differential equations (PDEs) significantly faster than traditional numerical methods in many domains.\"\n   \n   It is hard to find support for that claim in literature. Neural operators and other surrogate models are not solvers per se, since they require datasets generated or collected by other means. Physics-informed neural networks are much slower than their classical counterparts (although they can be more accurate for high-dimensional problems, e.g., quantum chemistry). Physics-informed neural operators and physics-informed DeepONet are also extremely slow. All methods known to me that are competitive or even faster are hybrid approaches when a part of the classical solver is replaced with a DL counterpart. These approaches can be 2-3 times faster at best. I suggest authors to back up their claim by pointing to appropriate literature.\n\n2. (page 1) \"FNO learns the solution operator of PDE that maps the input (initial and boundary conditions) to the output (solution functions).\"\n   \n   Somewhat meaningless description: any neural network maps its input to the output.\n\n3. (page 3) \"Li et al. (a) proposes FNO that adopts a convolution operator for K as shown in Figure 1, which obtains state-of-the-art results for solving PDE problems.\"\n   \n   This claim is not supported by literature. FNO was introduced in 2020 and provided state-of-the-art results (among the architectures tested by authors). There are examples now when classical architectures perform better (https://arxiv.org/abs/2112.15275) and other operators perform better (https://arxiv.org/abs/2111.13802). I suggest authors provide references that support their claim.\n\n4. (page 3) \"FNO is discretization-convergent, such that the model can produce a high-quality solution for any query points, potentially not in the training grid. In other words, FNO can be trained on low-resolution but generalizes to high-resolution.\"\n   \n   This description is misleading. FNO is consistent when evaluated on different grids, but it does not provide a high-quality solution. It is also not convergent in the sense of classical numerical analysis. Besides, it does not generalize to high-resolutions. As a rule, the error produced by FNO remains the same for different resolutions (e.g., figure 3, https://arxiv.org/abs/2010.08895), so there is no generalization, only consistency.\n\n5. (page 3) \"To ensure discretization-convergence, FNO truncates the Fourier series Fˆv at a maximal number of modes K using TK.\"\n   \n   Arguably, \"discretization-convergence\" is mostly achieved with FFT and inverse FFT that consistently map functions defined on different uniform grids. It is easy to come up with architecture without hard truncation that is still discretization-convergent.\n\n5. (page 4) \"Figure 2\"\n   \n   As I understand the left picture provides a ground truth solution and the two right pictures are outputs of two trained FNOs with different numbers of modes. Both predictions seem to be completely unrelated to the reference solution. Why in this case a large number of modes is better if the error is that large?\n\n6. (page 4) \"As FNO is a neural network and trained by first-order learning algorithms, it follows the implicit spectral bias such that the lower frequency modes have larger strength in R. This explains why FNO chooses to preserve a set containing the lowest frequency modes, instead of any arbitrary subset of frequencies.\"\n   \n   I am not sure I can follow this logic.\n   \n   First, FNO preserves low-frequency modes not because of the f-principle but by construction. The authors of FNO designed it in such a way.\n   \n   Second, the f-principle is an observation that when a neural network is trained to approximate some function, it first learns low frequencies and struggles with high frequencies. FNO is an operator, so if we apply the f-principle (I am not completely sure we can do that) it would mean that trained FNO should be biased toward \"smooth operators\". This supposedly includes an identity operator, that should retain all frequencies of the input.\n\n7. (page 5, Algorithm 1) \"Randomly initialize $R_0$\"\n   \n   This part seems to be important and not well discussed. How exactly $R_0$ is initialized?\n   If we look at the description of iFNO (Loss) on page 5, we can see that one increases the number of active modes. Does it mean that initially $R_0$ is as small as possible and the size of $R$ can only increase?\n\n8. (page 5, Algorithm 1) \"$K_t \\leftarrow K_t + 1$\"\n   \n   How the modes are added in higher dimensions? What is the order? Is it technically possible to add a single mode?\n   \n9. (page 5, Algorithm 1) \"and initialize the rest randomly\"\n\n   When $R$ is increased, how precisely new elements are initialized for both iFNO (Loss) and iFNO (freq)? In the latter case, this is especially interesting since the importance of modes is judged based on the value of $R$ itself.\n   \n10. (page 5, Figure 4)\n    \n    I suggest specifying where FNO is and where iFNO is directly on the picture.\n\n11. (page 7) \"Our samples are images; hence, we can easily distinguish between easy and difficult samples based on their resolution. Higher-resolution images are more complex to train on, so it’s best to start with lower-resolution samples and gradually move up to higher-resolution ones.\"\n    \n    I have several objections. First, the resolution is not directly related to difficulty, it is easy to produce simplistic high-resolution images. Second, the difficulty in operator learning is the difficulty of the mapping, not the images themselves. For example, we can take an extremely detailed turbulent flow and map it into itself. This will be a trivial operator learning problem when FNO needs to learn an identity map.\n\n12. (page 7, table 1), (page 8, table 2)\n    \n    Please report the relative $L_2$ norm in place of the value of the loss. Relative $L_2$ norm is a standard error metric present in most of the articles on neural operators, it is much easier to judge the quality of learned surrogate based on this metric.\n\n13. (page 10) \"The results in Table 1 show that iFNO achieves significantly better generalization performance than just incrementally increasing either frequency or resolution alone.\"\n    \n    The results in Table 1 show that all test losses are very close to each other. For example, for standard FNO we have $0.988 \\pm 0.031$, and for best iFNO, we have $0.948 \\pm 0.040$. They overlap even when a single std is considered. If we take $2$ std, there is no difference between these results. This is not a \"significantly better generalization performance\". \n    \n14. (page 10, figure 5b)\n    \n    iFNO starts training with a small number of active modes. What happens if one starts with a large number of active modes? Is this the case that iFNO will decrease the number of modes automatically?\n\nAs a final comment, I want to suggest an analogy with another field of study. One can look at the approach proposed by authors from the perspective of numerical linear algebra. For a fixed grid resolution, the FNO kernel is a low-rank block circulant matrix (quasimatrix, if different resolutions are considered) and the authors propose an algorithm that dynamically selects the rank of this circulant. How to do this was extensively studied in many contexts, so I suggest authors look at this direction for inspiration. For example, in deep learning rank adaptation of this sort is proposed at https://arxiv.org/abs/2205.13571. More specifically, in this paper authors first draw an analogy between the optimization process and the solution of ordinary differential equation (gradient flow). Next, they apply rank adaptive ODE integrator to the resulting system of equations. This provides training coupled with rank adaptation. I hope this perspective can be of use to the authors of the current contribution.", "broader_impact_concerns": "na", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ajpI7SV2Zp", "reviewer": "Reviewer_ZURE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Generally good self-contained introduction relating to FNO and issues in/expense of training\n- The new methods are well explained, and approach seems sensible\n- Good variety of target systems used to test the new approach\n\n**Weaknesses**\n\n- Experiment results and setup are harder to follow\n- Experiments in the appendix (and some in the main body) could use additional context and commentary\n- Tests on TFNO need more discussion and introduction", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a modified training for Fourier Neural Operators to reduce required hyperparameter tuning by dynamically adjusting the number of frequency modes used internally by the model during training, and reduces training overheads by adjusting the resolution of training samples. The authors' approach increases the number of modes used based on signals from the training loss or from observed strength of the unused frequencies making adjustments when the overall strength of modes currently in use drops below a threshold. The paper includes tests on a variety of PDE systems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The largest category of changes I would appreciate would be significant additional clarity in the experiments section. In particular, some more commentary on how readers should interpret some of the results in the tables would, I think, be welcome. I found the details of the experiment setup somewhat scattered and collecting these and streamlining the section would reduce the reader's need to jump around the section for details or comparisons which I think made the section harder to follow and obscured the results.\n\nBelow are some examples of questions that might benefit from some added discussion in the paper.\n\n1. The need for additional commentary is particularly true for the tables and plots in appendix B. Some explanation would be helpful (for instance in the hyperparameter selection tables regarding what values are included and what factors were weighed in choosing the fixed values from these results). For iFNO (loss), was the parameter eta renamed to epsilon in the body?\n\n2. Regarding the method listed as \"iFNO\" in each table (for example Table 1, 2, 3), which of the two (loss vs freq) approaches was used combined with the incremental resolution method? I believe in section 4.3 you mention that the \"iFNO\" entry could be using either.\n\n3. In Tables 2 and 3 the bolding is sometimes split between means and standard deviations. In table 3 for Navier-Stokes FNO 3D training results the largest value is bolded. Are these correct?\n\n4. In particular, the experiments on the TFNO are introduced *very* briefly and somewhat abruptly. I think these could especially use expansion or added explanation if these are important parts of testing the method.\n\n5. For the results listed in section 5.4 some additional discussion references to some backing data would help as well. It looks like some of the relevant values are included in the appendix for model size (Table 12) but is there a reference for the efficiency gains for systems other than re5000 in table 1?\n\nAnd one minor caption issue I noticed on figure 8, it seems it should reference Navier-Stokes 3D (vs. 2D) based on the titles of the plots", "broader_impact_concerns": "No specific concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LNbAOPgR4f", "reviewer": "Reviewer_XZ6r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes, and empirically demonstrates, a solution to a real problem in FNOs, which is selecting them to have sufficiently many Fourier modes to produce accurate predictions, but also sufficiently few Fourier modes to be efficient.\n2. the empirical study (section 6) or what happens to the modes under regularization schemes is neat. To my mind this is the biggest news in the paper and actually quite interesting.\n\nWeaknesses:\n \n1. justification is largely heuristic (\"we add modes as needed based on a reasonable rule of thumb that looks a bit like the Signal-to-Noise Ratio\"). If it works, this is OK, I guess, but it would be great to have a justification that was easier to generalize because it was more formal. *Why* might we think that controlling the number of modes by a squared Frobenius norm will control anything about accuracy?\n2. Accuracy is evaluated in terms of $L_2$ loss. There are a _lot_ of papers questioning whether this is a great way to measure accuracy of an operator; too many to list here, so I have (lazily) supplied an incomplete bibtex bibliography of some representative papers which interrogate this and consider at other losses (see below). This might feel like splitting hairs, but I argue it is important, since we are leaning on $L_2$ a lot now; both iFNO(freq) and iFNO(loss) do something that is similar to controlling $L_2$ loss, IMO, so we can reasonably ask if this procedure has done anything bad to other losses (Sobolev loses, $L_1$ losses, geometric distance in the natural manifold induced by the operator, _anything_ might be reassuring)\n3. comparison should be against SOTA attempts to adaptively construct optimal FNOs. There are two of these of which I am aware\n\n    1. Tensorized FNO: http://arxiv.org/abs/2310.00120 — the authors, to their credit, do use this and argue that their method is complementary, becuase they have a combination they call iTFNO in B.6 so this feels *close* to done. The only problem is that I think that the heuristic argument about the Explanation Ratio in iFNO(freq) don't really make sense for a tensorized FNO (the parameters are even further from spectral 'importances' so... some kind of explanation of this seems necessary even if to say \"our method seems empirical good for this even if our argument looks tenuous\". I think the iFNO(loss) heuristic still works?\n    2. _Transform once_ https://proceedings.neurips.cc/paper_files/paper/2022/hash/342339109d7d756ef7bb30cf672aec02-Abstract-Conference.html which *also* adaptively selects spectral bands, as well as doing some other fancy stuff to achieve good results. This is a more direct \"competitor\" in that they also adaptively select spectra bands. They also, crucially, argue that selecting only the lowest modes is **not** optimal (see fig 3.2), which seems like it might contradict the major result of this paper, which is that iFNO is best, and it uses strict low-pass filtering. There might be a counter-argument, which is that the Transform Once method is tedious or something? But I think such an argument needs to be made for the paper's main claim as I understand it to stand.\n\n4. occasionally confusing phrasing (see below)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors extend the standard approach to training an FNO with an incremental version that successively adds spectral bands to the inner Fourier-Transform layer. The extension is justified as being superior to grid search and classic regularization by, e.g. weight decay, in terms of both efficiency and accuracy (i.e. it is possibly more accurate than $\\|\\ell\\|_2^2$ weight decay regularisation and at lower cost)", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "on the basis of weakness 2 and 3.2 I  argue that the submission is not \"supported by accurate, convincing and clear evidence\" of the main claim, that  _We empirically show that iFNO reduces total training time while maintaining or improving generalization performance across various datasets_. There exist other methods that do something similar and undermine or contradict the main claim of the paper as it currently stands, and which are only partially addressed in this text. Changes that would be needed to redress this are mentioned below.\n\n**in 4.1 there is some confusing phrasing:**\n\n> iFNO (Loss): A common practice is to use the training progress as a proxy to determine the time, as seen\n> in previous work such as Liu et al.. Specifically, we let the algorithm increase K only decrease in training\n> loss between N consecutive epochs is lower than a threshold ϵ\n\nI think the authors might mean something like \n\n> iFNO (Loss): A common practice is to _add modes at fixed epochs_\n> in previous work such as Liu et al.. _By contrast we increase K adaptively when the decrement in training loss_ between N consecutive epochs is lower than a threshold ϵ\n\nIs that correct? Please clarify, this is a core bit of the paper.\n\n**what is iTFNO?**\n\nAs discussed in the \"weaknesses\", I think this is an important bit of work that the authors seem to have done, but which I think the paper does not explain. It is mentioned in the experiments in the appendix, but I can't work out what it is. Am I missing something? I found the sentence saying \"We also test our incremental methods on Tensorized Factorized Neural Operators Kossaifi et al. (2023) by sweeping over different factorization schemes and ranks for the weights\" but as mentioned above I think this is incomplete.\n\n**deal with _transform once_**\nI humbly reuest the authors to either \n\n1. relax the claim to exclude _transform once_ from consideration and justify that decision, or test against their method and demonstrate an improvement against it, as they have presumably done with TFNO (once they explain that)\n\n**losses other than $L_2$**\n\nAs referenced above, a very-incomplete bibliography of papers which in various capacities investigate the deficiencies of $L_2$ loss for FNO evaluation and/or training. I don't necessarily think the authors need to include all of these, or even *any* of them, but I think their claims would be strengthened if they at least mentioned the concept of physics-specific losses, rather than the computationally-convenient $L_2$ loss. If they are wedded to the $L_2$ loss only, there needs to be a justification for this choice.\n\nThere are at least two open-source tools which extend the losses for PDE solutions in various ways:\n\n* [pdebench/PDEBench: PDEBench: An Extensive Benchmark for Scientific Machine Learning](https://github.com/pdebench/PDEBench)\n* [pdearena/pdearena](https://github.com/pdearena/pdearena)\n\n\n```\n@misc{FaroughiPhysicsGuided2023,\n  title = {Physics-{{Guided}}, {{Physics-Informed}}, and {{Physics-Encoded Neural Networks}} in {{Scientific Computing}}},\n  author = {Faroughi, Salah A. and Pawar, Nikhil and Fernandes, Celio and Raissi, Maziar and Das, Subasish and Kalantari, Nima K. and Mahjour, Seyed Kourosh},\n  year = {2023},\n  month = feb,\n  number = {arXiv:2211.07377},\n  eprint = {2211.07377},\n  publisher = {arXiv},\n  doi = {10.48550/arXiv.2211.07377},\n  archiveprefix = {arXiv}\n}\n\n@article{LiPhysicsInformed2021,\n  title = {Physics-{{Informed Neural Operator}} for {{Learning Partial Differential Equations}}},\n  author = {Li, Zongyi and Zheng, Hongkai and Kovachki, Nikola Borislavov and Jin, David and Chen, Haoxuan and Liu, Burigede and Stuart, Andrew and Azizzadenesheli, Kamyar and Anandkumar, Anima},\n  year = {2021},\n  month = nov\n}\n\n@misc{PestouriePhysicsenhanced2022,\n  title = {Physics-Enhanced Deep Surrogates for {{PDEs}}},\n  author = {Pestourie, Rapha{\\\"e}l and Mroueh, Youssef and Rackauckas, Chris and Das, Payel and Johnson, Steven G.},\n  year = {2022},\n  month = nov,\n  number = {arXiv:2111.05841},\n  eprint = {2111.05841},\n  publisher = {arXiv},\n  doi = {10.48550/arXiv.2111.05841},\n  archiveprefix = {arXiv}\n}\n\n@inproceedings{TakamotoPDEBench2022,\n  title = {{{PDEBench}}: {{An Extensive Benchmark}} for {{Scientific Machine Learning}}},\n  shorttitle = {{{PDEBench}}},\n  author = {Takamoto, Makoto and Praditia, Timothy and Leiteritz, Raphael and MacKinlay, Dan and Alesiani, Francesco and Pfl{\\\"u}ger, Dirk and Niepert, Mathias},\n  year = {2022},\n  month = jun\n}\n\n@inproceedings{WangPhysicsinformed2020,\n  title = {Towards {{Physics-informed Deep Learning}} for {{Turbulent Flow Prediction}}},\n  booktitle = {Proceedings of the 26th {{ACM SIGKDD International Conference}} on {{Knowledge Discovery}} \\& {{Data Mining}}},\n  author = {Wang, Rui and Kashinath, Karthik and Mustafa, Mustafa and Albert, Adrian and Yu, Rose},\n  year = {2020},\n  month = aug,\n  series = {{{KDD}} '20},\n  eprint = {1911.08655},\n  pages = {1457--1466},\n  publisher = {Association for Computing Machinery},\n  address = {New York, NY, USA},\n  doi = {10.1145/3394486.3403198},\n  archiveprefix = {arXiv},\n  isbn = {978-1-4503-7998-4}\n}\n```", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I am a new AE who was assigned the submission recently. The authors proposed a method to improve the accuracy and training speed of neural operators for solving large-scale PDEs.  All reviewers acknowledged that the results are strong and the performance gain is valuable.  One reviewer remained concerned about the lack of theoretical justifications. While this is a valid concern, the submission has satisfied the acceptance criteria set by TMLR; thus, I recommend accepting the submission.
Decision: Accept (poster)