Amortized Fourier Neural Operators
Authors: Zipeng Xiao
Siqi Kou
Zhongkai Hao
Bokai Lin
Zhijie Deng
Venue: NeurIPS 2024 poster (2024)
Keywords: neural operator
Fourier neural operator
Kolmogorov–Arnold Network
Primary area: machine_learning_for_physical_sciences

Abstract: Fourier Neural Operators (FNOs) have shown promise for solving partial differential equations (PDEs).
Typically, FNOs employ separate parameters for different frequency modes to specify tunable kernel integrals in Fourier space, which, yet, results in an undesirably large number of parameters when solving high-dimensional PDEs. 
A workaround is to abandon the frequency modes exceeding a predefined threshold, but this limits the FNOs' ability to represent high-frequency details and poses non-trivial challenges for hyper-parameter specification. 
To address these, we propose AMortized Fourier Neural Operator (AM-FNO), where an amortized neural parameterization of the kernel function is deployed to accommodate arbitrarily many frequency modes using a fixed number of parameters. 
We introduce two implementations of AM-FNO, based on the recently developed, appealing Kolmogorov–Arnold Network (KAN) and Multi-Layer Perceptrons (MLPs) equipped with orthogonal embedding functions respectively. 
We extensively evaluate our method on diverse datasets from various domains and observe up to 31\% average improvement compared to competing neural operator baselines.

Amortized Fourier Neural Operators
Zipeng Xiao', Siqi Kou', Zhongkai Hao², Bokai Lin', Zhijie Deng' Qing Yuan Research Institute, SEIEE, Shanghai Jiao Tong University Dept. of Comp. Sci. & Tech., Tsinghua University (xiaozp_28 happy-karry)@sstuu edu cn, hzj216mails Smails. tsinghua, edu. (19821172068, jhi edu.
Abstract
Fourier Neural Operators (FNOs) have shown promise for solving partial differen- tial equations (PDEs). Typically, FNOs employ separate parameters for different frequency modes to specify tunable kernel integrals in Fourier space, which, yet, re- sults in undesirably large number of parameters when solving high-dimensional PDEs. A workaround is to abandon the frequency modes exceeding predefined threshold, but this limits the FNOs" ability represent high-frequency details and poses non-trivial challenges for hyper-parameter specification. To address these, we propose AMortized Fourier Neural Operator (AM-FNO), where amortized neural parameterization of the kernel function is deployed to accommodate arbi- trarily many frequency modes using fixed number of parameters, We introduce two implementations of AM-FNO, based on the recently developed, appealing Kol- mogorov-Arnod Network (KAN) and Multi-Layer Perceptrons (MIPs) equipped with orthogonal embedding functions respectively. We extensively evaluate our method on diverse datasets from various domains and observe up to 31% average improvement compared to competing neural operator baselines.
Introduction
Neural operators (NOs) have been extensively studied for their potential accelerating the solving
of partial differential equations (PDEs) in science and engineering fields [19] 20
18
24,
25
2).
In contrast to approaches limited to specific discretizations or PDE instances [8] 36 28
29), NOs
characterize the solving operator of family of PDEs across different discretizations and hence enjoy
higher efficiency and usability, e.g., for weather forecasting 26 and material analysis (5).
Fourier neural operator (FNO) I8 and its variants [7 of NOs, which explore the convolution theorem and Fast Fourier Transform (FFT) to efficiently performs kernel integral, central module for the learning operator of PDEs. Typically, FNO separately parameterizes the values of the frussee-transformeed kernel function for different frequency modes, and hinges frequency exceeding some threshold to reduce modeling costs, particularly for high-dimensional PDEs.
33
35| stand out significant subclass
caaattion-aaaddoon the parameters corresponding to frequencies
Frequency truncation can be problematic when solving PDE systems with intense high high-ffrquency components. To address this, IFNO dynamically adjusts the threshold for frequency truncation during training, though it still experiences exponential parameter growth with increasing dimensionality (35] This complexity can result in substantial memory consumption and hinder the development of large- scale pretrained models [9]. Conversely, AFNO [7] employs shared MLP to transform the outcomes of FFT, but the uniform treatment of frequency modes may constrain expressiveness and lead to suboptimal performance.
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Frequency-specific paramcterization
Amortized neural paramcterization
Kernel Function Value
Kerel Function Value
MLP
KAN
Truncation
Figure 1: Comparison between FNO and AM-FNO: FNO assigns each value at the discretized frequencies of the Fourier-transformed kernel function as a learnable parameter, while AM-FNN utilizes neural network parameterization (MLP or KAN) to approximate the mapping between frequencies and function values. The frequencies are embedded using a set of orthogonal basis functions before being processed by the MLP.
We address this by developing AMortized Fourier Neural Operator (AM-FNO), where we introduce neural networks (NNs) specify the kernel integral operator amortize the modeling cost. As illustrated in Figure AM-FNO simple and intuitive- NN transformations are leveraged directly define the Fourier-space kernel function accommodate arbitrarily many frequency modes at the cost of fixed number of parameters. An amortized parameterization also provides an inherent regularization mechanism for resisting overfitting to potential high-frequency noise.
The NN transformation in AM-FNO can be flexibly defined. One natural choice is the Kol- mogorov-Arnold Network (KAN) due to its superior accuracy in funetion fitting 21). Doing so, we, for the first time, reveal the potential of KAN for the operator learning of PDEs. Considering the widely criticized inefficiency issues of KAN in both time and memory consumption, investigate the regular Multi-Layer Perceptrons (MLPs) for amortized parameterization. We empiri- cally identify the necessity of embedding the frequency modes with orthogonal basis functions before ML. transformation.
also
We experiment on challenging benchmarks governed by diverse typical PDEs, covering six standard PDE benchmarks [8, 17 30). AM-FNO (KAN) and AM-FNO (MLP) achieve an average 22% these benchmarks, respectively. We also analyze the error of
and 31% reduction in relative different frequency modes, and the results show that our models outperform in all frequency ranges. Additionally, perform zero-shot super-resolution three benchmarks assess the generalization ability of AM-FNO across discretizations. We observe that AM-FNOs outperform the baselines and achieve lower error even compared to FNO trained the test resolution. The results reflect that AM-FNOs have the promising potential to substantially reduce both the data collection and training costs for solving high-resolution PDE problems.
2 Related Works
Neural Operators. Neural operators have attracted considerable interest for their capacity to map dinnii-dimmnsional function spaces [24 T]. thereby facilitating solutions across diverse discretizations without retraining. The pioneering work DeepONet (24) introduces trunk and branch network architecture grounded the universal approximation theorem for operators. Transformer- based neural operators represent a notable line of work in the domain. Galerkin Transformer [2] proposes self-attention operators that theoretically correspond to learnable kernel integral operator and projection. OFormer [16] introduces architecture with input and query encoders for querying arbitrary output locations. GNOT [00 proposes a heterogeneous normalized attention layer to encode different input information.
FNO and its Variants. Fourier neural operator (FNO) [8] represents a novel approach with excep- tional efficiency and accuracy due to its ability to learn kernel integral operators in the Fourier domain. The theoretical proof establishes its capability to approximate arbitrary continuous operator (15). Tailored for addressing multiphase flow problems, U-FNO
33 augments its representation in higher frequency information through U-Net paths. Geo-FNO [17] extends the applicability of FNO beyond uniform grids by employing learnable mapping from irregular domains to uniform latent meshes.

---

F-FNO 31 reduces the model parameters and addresses performance degradation with increasing layers by factorizing the integral operator and enhancing the architecture. AFNO [ transforms the function values after FFT with a shared MLP for each frequency, effectively reducing the parameter count. FNOs have made notable contributions across various challenging tasks [6] 25 33 26]- However, minimizing model complexity while effectively managing high-frequency information and ensuring generalization across different discretizations poses a persistent challenge in FNOs. Additionally, to the best of our knowledge, there has been no attempt to incorporate KANs with neural operators.
Preliminary
This section presents the foundations of operator learning and the integral operator used in FNO.
3.1 Operator Learning
Consider the input function space A A(DD; (Rd))) and the target function space /4 (D; Rd:) defined a bounded and open set D C. Rd Operator learning seeks to learn 0-parameterized operator Go to approximate the ground-truth mapping G : :AA l specified by a PDE. This learning process is based on finite set of function observations {a;, u4 J where functions a, and u, discretized on meshes {æy E D) The optimization problem is:
[9o(a;) =ul12
min
|
where the regular mean-squared error (MSE) extended with a normalizer ||u:||z to handle scale variations across benchmarks, denoted l2 relative error. The relative error also be substituted with other loss functions.
3.2 Fourier Integral Operator
A learnable kernel integral is a central module for defining mappings among functions. Specifically, we denote the hidden state of the input function in the (-th transformation stage as (y) D - &d where da represents the dimensionality, assumed to be consistent across all stages. The kernel integral operator makes the following transformation:
())())))))) = S/mCe, n(x, u) (y)dy, Ve E D
However, the integral is not computation friendly within deep learning frameworks. To address this, FNO [8] assumes the kernel is theorem to efficiently compute the integral in the Fourier domain:
i.c., x(x, 8) <(x =(x-y), and leverages the convolution
()(())))(<)) = =F ((h))))),, E
where F and FF1 denote the FFT and its inverse (IFFT), and R(k): E xddx the tranfor-trnnsoormeed complex-valuee kernel function with E E denoting a frequency mode. Typically, FNO individually parameterizes the values of R(K) for fixed range of frequency modes (denoting the number as kr) to avoid high modeling costs. This, yet, limits the exploration of high- frequency details in the function and poses non-trivial challenges for hyper-parameter specification.
xdx) represents
4 Method
This section elaborates on amortized FNO (AM-FNO), which amortizes an arbitrary number of frequency modes by sharing fixed number of parameters.
4.1
Amortized Parameterization
Instead of parameterizing the kernel function point-by-point, we propose to build the mapping between frequency and the values of the fourier-transformee kernel function using NN. On one

---

hand, this mitigates the issue that the number of parameters increases significantly with that of frequency modes and dimensionality PDEs. On the other hand, this approach avoids AFNO's uniform transformation, ensuring richer expressiveness in the Fourier domain.
Concretely, by the rule of Fourier transformation, there is:
R(k) k(()e Rimor-k da.
(4)
Note that the output of It is matrix, so we can use Rp.q. P, q E 21, scalar-valus valued function yielding one element the matrix output. Our AM-FNO directly uses NNs to define the real and imaginary parts of the function Rp.g to maximize the modeling flexibility. Formally, there is
d. to denote the complex
(k) == Ne (k) + -
(5)
The detailed implementations of the two NNs have no essential difference, so we only discuss one of them in the following.
4.2 Kolmogorov-Arolld Networks
Kolmogorov-Arnot Networks (KANs) have been empirically shown to show promise in function approximation 21 According to theoretical analysis, KANs are usually defined as:
2/+1
KAN(x) = Σ m (z)))
(6)
where 7/j. Tb.1 : + R denote the learnable basis functions. In practice, we can generalize the above definition and set a KAN layer as (with æ as a H-dim variable):
(h,1.)) (-
m.2(.) m222())
12
x'=
x.
())(()
())
We can specify the function by learnable coefficients and multiple local B-spline basis functions 21
<T w(r(x) spline(x)), spline(x) Σ By(x)
where r(z) represents basis function (typically sigmoid linear unit or SiLU B and w is a factor controlling the magnitude.
utilize two two-layer KANs each layer of AM-FNO to define the real and imaginary parts of Rp.g- The inputs to the network are all frequencies of (9()) after FFT. In fact, we can share weights among the KANs associated with Ry.q with different and q. Empirical results (Table 4 indicate superior performance compared the corresponding MLP implementation.
4.3 Multi-Layer Perceptrons
In practice, KANs require extensive training time. To address this, we propose an alternative parameterization aimed at improving the performance of Multi-Layer Perceptrons (MLPs). Despite its universal approximation ability. MLPs empirically suffer from compromising performance. The spectral bias of vanilla MLPs, i.c., their tendency to favor low-frequency functions, may limit their capacity represent more complex functions. Motivated by the success of leveraging orthogonal we propose to augment the MLP with
basis functions for function approximation 12 14 27].
orthogonal embedding functions construct AM-FNO.
Specifically, we can embed the frequency mode input using set of orthogonal functions before the MLP transformation. The Fourier basis is natural choice due to its capacity capture high-frequency components, thereby enhancing the gighfrreueecy representation of vanilla MI.Ps. However, our empirical results show that Chebyshev basis functions can perform better (sce Table 4 Of note,

---

N
N2
N2
7
p
t/s
Operator layer
Operator layer?
Operator layer
11+1
FFT
MM
R
IFFT
FFN
ky KAN
MLP
T.
Figure 2: AM-FNO structure for 2D PDEs: The input function is mapped to a higher-dimensional space. Stacked operators and activation functions applied for function propagation. Within the operator layers, linear transformation R is applied to (1) after FFT, followed by bewd-forward network (FFN) after the Inverse Fast Fourier Transform (IFFT). The values of R result from KAN or multiplying the MLP transformations of selected one-dimensional orthogonal basis functions (w denotes linear weights.). Finally, the function is projected to the solution dimension space.
only the first Tmma orthogonal basis functions in the family for the parameterization cannot employ infinite parameters.
Factorization trick for high-dimensional PDEs. For the Fourier-transfomeed kernel, the dimension- ality the frequency modes matches that of the PDE being solved. To approximate functions with d-dimensional inputs, common approach is to construct d-dimensional orthogonal functions based on mic-dmmnsional basis functions. Specifically, for maax one-dimensional basis functions in each of the d dimensions, the complete space contains n high-dimensional basis functions, resulting in exponential complexity for modeling.
Motivated by the dimension factorization in the integral operator 31 sions and construct the final kernel by the products the kernel approximated each dimension. Such process is illustrated in Figure 2 In this way, the total parameter count scales linearly w.r.t. dimension of the PDEs.
we separate the input dimen-
4.4 AM-FNO Architecture
In FNO, a stacked structure is employed to approximate the entire mapping G, as illustrated below:
= P (1))
e())
where L maps the input function a to the hidden state h (0), and P maps the hidden state (E) to the output function both a point-wise manner. Q (0) - ([+1) is the operator layer responsible for the iterative update:
2(h0) ( + (C( +
(10)
where I. represents the kernel integral operator in Equation 3 and b represents the bias. FNO employs pointwise linear mapping with W enable the propagation of high- fegh-freeuency information, but this is empirically limited, validated by the results in Table
Given the non-truncated Fourier transform of the kernel function R in AM-FNO, employ the operator layer architecture 31,, which replaces the linear map W with residual connection [I]:
N(Nh) = ( + (W + b1) + b2).
(11)
we also incorporate activation functions between the operator layers for enhanced flexibility. The resultant model structure is:
As shown in Figure 2
= P oo.. (1 L.
(12)
5

---

Table I: Overview of benchmarks including their spatial dimensions d, spatial resolution M, temporal resolution Nr, and training data Nerain and test data Niest
Benchmark
PDE
Geometry
d
M
Ni
Nunin
Niss
Darey
Darcy flow
Regular grid
2
x 85
1000
200
NS-2D
Navier-Stokes
Regular grid
2
x 64
20
1000
200
Pipe
Navier-Stokes
Structured mesh
2
129 x 129
1000
200
Airfoil
Fuler equation
Structured mesh
2
221 x 51
1000
200
Elasticity
Elastic Wave
Point cloud
2
972
1000
200
CFD-ID
Compressible Ntoker-Stokes
Regular grid
1
128
1800
200
CFD-2D
Compressible Ntvier-Stokes
Regular grid
2
64 x 64
21
1800
200
5
Experiment
In this section, we validate the effectiveness of our proposed method by conducting extensive experiments challenging benchmarks governed by typical solid and fluid PDEs.
5.1
Experimental Setup
Benchmarks. We evaluate the performance of AM-FNO on six well-established benchmarks. These benchmarks include Burger, Darcy, and NS-2D, which are presented in regular grids with varying dimensions [18]. We extend our experiments to assess the method's ' performance different geometries, including Pipe, Airfoil, and Elasticity benchmarks [17] Additionally, incorporate the compressible fluid dynamics (CFD) 1D and 2D benchmarks (30] which involves more high-frequency information. The benchmarks summarized in Table 1
Baselines. We conduct a comparative evaluation of our neural operator against seven baseline methods. These baselines include well-recognized approaches such as FNO [[8] and its variants Geo-FNO [17, U-FNO (33, F-FNO [31 and AFNO 77. Additionally, we consider other models, including OFormer 166 and LSM 34 Notably, LSM represents the latest state-of-the-art (SOTA) neural operator among the baselines.
Implementation details. We train all models for 500 epochs using the AdamW optimizer 23 with a cosine annealing scheduler [22] The initial learning rate 10-3, and the weight decay is set 10 Our models consist of 4 layers with width of 32 and process all the frequency modes of training data. The Gaussian Error Linear Unit (GELU) is used as the activation function [3.. For AM-FNO (KAN), the number of spline grids is selected from {24, 32, 48). while for AM-FNO (MLP), the number of basis functions set to 32 or 48. AM-FNO (MLP) utilizes Chebyshev basis functions as the orthogonal basis functions, as elaborated in the appendix. The batch size is selected from {4, 8, 16, 32), and the experiments are conducted on single 4090 GPU. The evaluation metric and training loss based on the l2 relative error in Equation ( unless otherwise specified. We employ the transformation method from geo-FNO [17] to map between irregular input domains and uniform meshes for the Elasticity benchmark point clouds. More details about the baselines can be found in Appendix A
5.2 Main Results
The main results are shown in Table 2 performance on all six benchmarks with various PDEs, geometries, and dimensions. AM-FNOs exhibit a significant average performance improvement of 22% from KAN implementation and 31 % from MLP implementation compared to the top pop-eerforning bascline. We provide a comparison of GPU memory and training time in Appendix D showing that although AM-FNO retains all frequency modes, achieves comparable memory usage and training time to other FNOs.
Our models consistently achieve state-of-the-art (SOTA)
AM-FNOs significantly reduce prediction error on Darcy and NS-2D benchmarks, standard bench- marks with strong ow-frequency components. Specifically, AM-FNO (KAN) reduces the error by 39% (2.73e-3) and 11% (1.40e-2), while AM-FNO (MLP) reduces the error by 40% (2.80e-3) and 30% (3.69e-2). This improvement can be attributed to AM-FNO's 's kernel parameterization, which effectively captures the low-frequency information. Additionally, AM-FNO demonstrates robustness across irregular geometries, with reductions of 5% (3.30e-4), 32% (1.66e-3) and 7 (%(1.e--)) from AM-FNO (KAN), and 12% (7.50e-4), 34% (1.76e- and 10% (2.20e-3) from AM-FNO (MLP) in

---

Table 2: Comparison of the primary findings across six benchmark tests with six baseline methods. Lower scores signify superior performance, with the best outcome highlighted in bold and the second- best outcome underlined. The presence of x indicates that the corresponding baseline is incapable of addressing the benchmark.
Model
Darcy
NS-2D
Pipe
Airfoil
Elasticity
CFD-ID
CFD-2D
FNO
1.08e-2
1.56e-1
2.93e-2
5.36e-3
AFNO
3.17e-2
2.17e-1
1.72e-2
9.88e-3
4.57e-2
6.72e-2
Geo-FNO
1.08e-2
1.56e-1
6.70e-3
1.38e-2
2.29e-2
2.93e-2
5.36e-3
OFormer
1.24e-2
1.71e-1
9.59e-3
1.83e-2
U-FNO
1.24c-2
1.22c-1
5.76c-3
1.05e-2
2.26c-2
2.44c-2
4.52c-3
R-FNO
9.92e-3
1.74e-1
5.99e-3
1.00e-2
3.16e-2
2.54e-2
7.86e-3
LSM
7.01e-3
1.64e-1
5.20e-3
6.39e-3
2.25e-2
7.62e-2
Ours (KAN)
4.28e-3
1.08e-1
3.54e-3
6.06e-3
2.10e-2
1.83e-2
2.70e-3
Ours (MLP) 4.21e-3
8.51e-2
3.44e-3
5.64e-3
2.03e-2
1.47e-2
2.16e-3
prediction enror on Airfoil, Pipe and Elasticity benchmarks. For CFD-ID and CFD-2D benchmarks characterized by stronger figh-freuuency components, AM-FNO (KAN) achieves improvements of 25% (6. 10e-3) and 40% (1.83e-3), while AM-FNO (MLP) achieves improvements of 40% (9.70e-3) and 52% (2.36e-3). The promotion highlights the effectiveness of our method of handling high- frequency components. We also find that AM-FNO (MLP) outperforms AM-FNO (KAN) across all benchmarks, likely due the enhanced expressiveness of the orthogonal embedding.
5.3 Frequency-Basel Error Analysis
CFD-1D. To assess the performance across various frequency modes, we calculate the error of different frequency modes after FFT on CFD-ID benchmark and incorporate FNO without truncation (FNO+) for comparison. The results are visualized in Figure 3
AM-FNO
(MLP)
FNO
FNO
1.2 Famor
UNO
Truncation
10
20
30
Frequency mode
Figure 3: Comparison of L2 norm error different frequency modes on CFD-1D benchmark.
As shown, the errors primarily stem from the first few frequency modes and decrease as the frequency increases. The baselines exhibit similar errors in the truncated frequency range, whereas models demonstrate significantly lower errors. Our models maintain an advantage over the baselines in the initial modes within the truncated frequency range. As the frequency increases, the strength of the high-frequency components diminishes, and all the errors become negligible.
CFD-2D. We further evaluate the performance across various frequency modes with the metrics outlined in (00 (detailed Appendix B and present the results on CFD-2D benchmark in Table 3
The results showcase that FNO demonstrates lower train enror but higher test error than FNO and U-FNO. Notably, FNO exhibits similar prediction errors to FNO the high-frequency range, suggesting potential overfitting to the training data. We propose that the substantial complexity parameterization of + may render sensitive to high-frequency details, limiting effectiveness of the additional parameters in handling such components. The fl.2 error of U-FNO across all frequency ranges is lower than FNO, which can be attributed the enhanced expressiveness enabled by the additional U-Net architecture. AM-FNO (MLP) achieves the lowest training error and fL2 error
7

---

Table 3: Comparison of the enror in different frequency regions CFD-2D benchmarks. Each complex-valued parameter is considered as in the parameter count (Param). Train error (Train Err.) and test emrrr (Test ErT.) evaluated using the relative enror at each time step. fL2 signifies the l2 relative error in Fourier space (fRMSE) pertaining to the low, middle, and high-frequency regions. FNO refers to FNO without frequency truncation.
CFD-2D
Model
Param (M)
Train Err.
Test Err.
fL.2 low
fL. mid
fL2 high
FNO FNO +
2.37
2.57e-3
5.36e-3
1.51e-3
7.66e-1
1.49e-1
18.39
1.63e-3
5.81e-3
1,82e-3
7.31e-1
1.42e-1
U-FNO
2.66
2.00e-3
4.52e-3
1.26e-3
6,55e-1
1.28c-1
Ours (KAN)
2.21
1.79c-3
2.70e-3
7.78c-4
4.19e-1
7.65e-2
Ours (MLP)
2.29
1.34e-3
2.16e-3
6,22e-4
3.55e-1
5.76e-2
Table Comparison of the 12 relative error for different components of AM-FNO (MLP) on Darcy, Airfoil, and Pipe benchmarks. Chebyshev basis functions are substituted with triangular basis functions (TBF) and non-orthogonal polynomial basis functions (PBF). A version of the model without orthogonal embedding (Non) is included for comparison. The training time and memory requirements are derived from the Airfoil benchmark.
Param (M)
Mem (MB)
Time (s/epoch)
Designs
Darey
Airfoil
Pipe
TBF
1.14
1890
2.61
4.13e-3
7.60e-3
3.80e-3
PBF
1.13
1890
2.55
1.77e-2
1.30e-2
1.03e-2
Non
1.10
1826
2.70
1.29e-2
7.21e-3
7.84e-3
Ours (MLP)
1.14
1890
2.52
4.21e-3
5.64e-3
3.44e-3
Ours (KAN)
1.56
2230
4.70
4.28e-3
6.06e-3
3,54e-3
across all frequency ranges, while AM-FNO (KAN) achieves the second lowest fL2 error. Specifically, AM-FNO (MLP) and AM-FNO (KAN) achieve 57% (7.04e-2) and (5.1ee2) reduction in the high-frequency range. This outcome can be attributed to our amortized parameterization, which significantly reduces model complexity while maintaining adequate expressiveness to approximate the courier-transforme kernel function.
5.4 Ablation Experiments
We conduct detailed ablation study to assess the effectiveness of different components and hyperpa- rameters of our models.
Necessity of Orthogonal Embedding. We study the impact of orthogonal embedding on Darcy, Airfoil, and Pipe benchmarks. presents the findings. Although AM-FNO (KAN) outperforms the MLP version without embedding (Non), its efficiency is significantly lower than that of versions utilizing MLPs. Removal of the orthogonal embedding leads notable performance decline across all three benchmarks. Meanwhile, replacing the orthogonal functions with non-orthogonal polynomial basis functions results in the highest prediction error among the baseline models. These outcomes showcase the efficacy and indispensability of the orthogonal embedding. To validate the robustness of the embedding, we use triangular basis functions and observe comparable errors on the Darcy and Pipe benchmarks, but higher errors the Airfoil benchmark. We attribute this difference to the accuracy of function approximation achieved by Chebyshev basis functions.
Influence of Some Hyperparameters We conduct experiments to assess how prediction error varies with different numbers of basis functions, hidden sizes of KANs, and grid sizes of splines (linearly scaled with local spline count) in KANs. Figure shows shows the results. The left figure illustrates that error decreases with an increasing number basis functions in orthogonal embedding on both benchmarks. This reduction is particularly evident initially on the Airfoil benchmark, followed by diminishing returns. this case, employing 24 basis functions achieves favorable balance between efficiency and accuracy. This performance enhancement can be attributed to the increased

---

Daney
Taney Airfoil
0.012
Dany
12 Rolatvo Lirer
1.2 Relaivve Taror
Rolatie Birer
0,000
0.02
0,008
0.00
Number Basds Functions
Hidden Layer Size K.AN
Size ofKAN
Figure 4: 12 relative error varies w.r.t. the number of basis functions (Left), hidden layer size of KANs (middle), and grid size of KANs (right) on Darcy and Airfoil benchmarks.
Figure 5: Comparison of zero-shot super-reso resolution absolute errors on NS-2D benchmark. The top row displays the ground truth, FNO, U-FNO, AM-FNO (KAN), and AM-FNO (MLP) predictions from left to right. The bottom row illustrates their errors.
expressiveness from the additional orthogonal basis functions. In the middle figure, while the error decreases with increasing hidden size, we observe decline in performance compared to KANs with larger grid sizes but smaller hidden sizes. This may suggest that the expressive power of KANs is primarily derived from the number of local spline functions (grid size). The right figure displays trend similar to the left one: noticeable decrease initially, followed by less pronounced reduction later. In this study, we suggest increasing the grid size to enhance performance rather than focusing on adjusting the hidden layer size. We also present the performance of AM-FNOs, retaining the same frequency modes as other FNOs, in Table AM-FNOs consistently outperform baseline models, underscoring the advantages of our amortized parameterization over the standard FNO approach.
5.5 Zero-Shot Super-Resolution
A notable characteristic of neural operators is their ability to generalize across various discretizations. We conduct experiments training on lower resolution data and evaluate on higher resolution data on
NS-2D benchmark. We visualize the results in Figure 5 and provide the numerical results in Table 5
U-FNO exhibits significant perfor- mance degradation when evaluated on higher resolution. This decline can be attributed to the convolutional opera- tion the U-Net architecture, which possesses fixed receptive field and cannot effectively generalize across different discretizations. In contrast, FNO and AM-FNOs demonstrate ability to generalize across different discretizations. Notably, AM-FNOs achieve superior perfor- mance at 64 resolution compared to baselines trained with the same resolution (see Table 2 which underscores the data efficiency of AM-FNOs.
FNO
U-FNO
Ours (KAN)
Ours (MLP)
32 32
1.32e-1
1.39c-1
9.84e-2 9.96e-2
8.79e-2
64 x 64
1.28e-1
2.18c-1
8.73e-2
Table 5: Comparison of l2 relative error across different resolutions on NS-2D benchmark, with all models trained with 32 x 32 resolution.

---

Conclusion
This paper proposes AM-FNOs to improve Fourier neural operator (FNO)'s efficiency addressing PDEs without frequency truncation. Our approach utilizes Kolmogorov-Aroold Networks (KANs) and Multi-Layer Perceptrons (MLPs) with orthogonal embedding functions to mitigate exponential complexity and overfitting to high-frequency noise. Comprehensive experiments across various datasets demonstrate the effectiveness AM-FNOs compared to baseline approaches.
Limitations. This work attempts to enhance FNO's 's handling of high-freegn high-frequency information but has the following limitations. The benchmarks used are idcalized physical systems, excluding real-world complex problems. Mcanwhile, although AM-FNOs reduce the parameter count, the extremely high-dimensional PDEs still pose challenges for FNOs due the complexity of FFT.
Acknowledgements
This work supported by NSF of China (No. 62306176), Natural Science Foundation of Shanghai (No. 232R 1428700), CCF-Zhipu Large Model Innovation Fund (No. CCF-Zhipu202412), and CCF-Baichuan--Btech Foundation Model Fund.
References
[1] Kaushik Bhattacharya, Bamdad Hosseini, Nikola Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics, 7:121-157, 2021.
[2] Shuhao Cao. Choose a transformer: Fourier or galerkin. Advances in neural information processing systems, 34:24924 24940, 2021.
[3] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
[4] David Elliott, DF Paget, GM Phillips, and PJ Taylor. Error of truncated chebyshev series and other near minimax polynomial approximations. Journal of approximation theory, 50(1):49-57, 1987.
[5] Somdatta Goswami, Minglang Yin, Yue Yu, and George Em Karniadakis. A physics-informed variational deeponet for predicting crack path in quasi-brittle materials. Computer Methods in Applied Mechanics and Engineering, 391: 14587, 2022.
[6] Steven Guan, Ko-Tsung Hsu, and Parag V Chitnis. Fourier neural operator networks: A fast and general solver for the photoacoustic wave equation. arXiv preprint arXiv:2108.09:: 2021.
[7] John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint
2021.
[8] Xiaoxiao Guo, Wei Li, and Francesco Torio. Convolutional neural networks for steady flow ap- proximation. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 481 1940, 2016.
[9] Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying. Hang Su, Anima Anandkumar, Jian Song, and Jun Zhu. Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training. arXiv preprint arXiv:2403.03354. 2024.
[10] Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, and Jun Zhu. Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learning, pages 12556-12569. PMLR, 2023.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
10

---

[12] Mingguo He, Zhewei Wei, and Ji-Rong Wen. Convolutional neural networks on graphs with chebyshev approximation, revisited. Advances neural information processing systems, 35:7264-7276, 2022.
[13] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint 1606 08415, 2016.
[14] Lina Karam and James H McClellan. Complex chebyshev approximation for fir filter design. IEEE Transactions Circuits and Systems II: Analog and Digital Signal Processing, 42(3):207- 216, 1995.
[15] Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. The Journal of Machine Learning Research, 22(1):13237- -33322, 2021.
[16] Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning, arXiv preprint arXiv:2205.13677 2022.
[17] Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural oper- ator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207.05209, 2022.
[18] Zongyi Li, Nikola Kovachki, Kamyar Azizzadeneshelli Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations, 2020.
[19] Zongyi Li, Nikola Kovachki, Kamyar Azizzadeneshelii Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint 0rXiv:2003.03483 2020.
[20] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755-6777 6766, 2020.
[21] Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruchle, James Halverson, Marin Soljačić, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-arool networks. arXiv preprint STfi:22044199566 2024.
[22] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint /608. 2016.
[23] Uyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05101, 2017.
[24] Lu Lu, Pengzhan Jin, Guofei Pang. Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218-22) 22022222222222222222 2021.
[25] Nicholas H Nelsen and Andrew M Stuart. The random feature model for input-output maps between banach spaces. SIAM Journal on Scientific Computing, 43(5):A3212-A3243 2021.
[26] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.1 112122222.12214.. 2022.
[27] S Qian, YC Lee, RD Jones, CW Bames,, and K Lee. Function approximation with an orthogonal basis net. In 1990 IJCNN International Joint Conference on Neural Networks, pages 605-619. IEEE, 1990.
[28] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
11

---

[29] Jonathan D Smith, Kamyar Azizzadenesheli, and Zachary E Ross. Eikonet: Solving the eikonal equation with deep neural networks. IEEE Transactions on Geoscience and Remote Sensing. 59(12):10685-
2020.
[30] Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596-1611, 2022.
[31] Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. In The Eleventh International Conference on Learning Representations, 2023.
[32] Lloyd N Trefethen. Approximation Theory and Approximation Practice, Extended Edition. SIAM, 2019.
[33] Gege Wen, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M Benson. U-fno-an an enhanced fourier neural operator-basee deep-learning model for multiphase flow. Advances in Water Resources, 163:1041800 2022.
[34] Haixu Wu, Tengge Hu, Huakun Luo, Jianmin Wang, and Mingsheng Long. Solving high- dimensional pdes with latent spectral models. In International Conference Machine Learning, 2023.
[35] Jiawei Zhao, Robert Joseph George, Yifei Zhang, Zongyi Li, and Anima Anandkumar. Incre- mental fourier neural operator. arXiv preprint arXiv:2211.15788 2022.
[36] Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415 2018.
A Hyperparametters and Details for Models
AM-FNOs. For AM-FNO (MLP), Chebyshev polynomials are chosen due to their favorable theoreti- Formally, they are defined as:
cal accuracy 32
For & E [-1, 1], To (k) = 1, Ti = k; T T (I)) 2K (k)
(13)
n n21 1.
The polynomial demonstrates an increase in the degree of its highest power term concerning k as n increases. It can be alternatively formulated as:
T, (k) cosscarccos(k)), for k [-1,
(14)
For AM-FNO (KAN), we fix the grid size during training to ensure a consistent parameter count. The order of the spline is fixed as 3.
FNO and its Variants. We employ 4 layers with modes set to 12 and widths set to for FNO and its variations (Geo-FNO, U-FNO, F-FNO). U-FNO incorporates the U-Net path in the last two layers. For AFNO, we set the width 512 and use layers to maintain comparable parameter count. Notably, Geo-FNO reverts to the vanilla FNO when applied to benchmarks with regular grids, resulting in equivalent performance for Darcy and NS-2D benchmarks.
LSM. The model is employed with 8 basis operators and 4 latent tokens. The width of the first scale set to 32, with a downsampling ratio of 0.5.
OFormer The depth of the encoder is fixed at 6, while the hidden dimension is set to 96.
B
Metrics
Following 30]. we use fL. error to quantify errors in different frequency ranges, It is computed as: [F(Upred) F l|2
(15)
[((urue))l
where F denotes the FFT and the frequency range is restricted to kimin S k S kimax- For fl.2 low, Amin = 0 and kmax 4; for fL2 mid, Kmin and kmax 12; for fL2 high, Kmin 12 and kimox co.
12

---

C
Comparison of GPU Memory, Training Time, and Parameter Counts.
Table 6: Comparison of GPU memory, training time per epoch, and parameter counts on Darcy benchmark.
Model
AM-ENO(MLP)
AM-HNOIKAN)
FNO
U-FNO
OFormer
LSM
F-PN
AFNO
Time
2.1
16.5
1.1
Memory (M)
1850
2066
1212
1444
16090
1894
1126
11288
Params. (M)
1.1
1.5
2.4
0.2
D Repeated results on NS-2D and CFD-2D benchmarks.
Table 7: Comparison of the l2 relative error on NS-2D and CFD-2D benchmark.
Benchmark
AM-FNO(MLP)
AM-FNO(KAN)
NS-2D
8.53e-2 + 7.48e-4
1.04e-1 + 3.27e-3
CFD-2D
2.21e-3 + 4.13e-5
2.75e-3 + 8.96e-5
E
Abltation Study
We investigate the impact of the dimensional factorization trick and the model architecture. The results shown in Table &
Table 8: Comparison of the l2 relative error for different components of AM-FNO (MLP) on Darcy, Airfoil, and Pipe benchmarks. The version with the vanilla FNO architecture (Vanilla) and the version without dimensional factorization (No-DF) are included. The training time and memory requirements derived from the Airfoil benchmark.
Param (M)
Mem (MB)
Time (s/epoch)
Designs
Darcy
Airfoil
Pipe
Vanilla
1.11
1274
2.28
5.25e-3
8.03e-3
3.89e-3
No-DF
1.10
1938
2.61
5.24e-3
5.89e-3
3.83e-3
Ours (MLP)
1.14
1890
2.52
4.21e-3
5.64e-3
3.44e-3
Ours (KAN)
1.56
2230
4.70
4.28e-3
6.06e-3
3.54e-3
The results indicate that the version without dimensional factorization performs comparably, Di- mensional factorization be unnecessary for handling low-dinensional PDEs. Meanwhile, the employed architecture achieves an average eтror reduction of 20%.
We also report the performance of AM-FNOs retaining the same number of frequency modes as other FNOs (12) on Darcy, Airfoil, and Elasticity benchmarks. The results, shown in Table D indicate that AM-FNOs consistently outperform the baselines, highlighting the advantages of our amortized parameterization.
F Discussion about KAN and MLP.
Our findings in Section demonstrate that implementing MLPs with orthogonal embeddings results in superior accuracy and efficiency. We attribute this success to the efficacy of the embedding technique employed. Meanwhile, the compatibility of the KAN architecture with widely used optimizers, such as AdamW, remains questionable. However, AM-FNO (KAN) offers several benefits over AM-FNO (MLP). First, as illustrated in Figure 4 the expressiveness of KANs primarily stems from the grid size. To uphold a constant parameter count, the grid size remains unchanged. However, the architecture is inherently extensible during training, which could potentially enhance accuracy. Second, KANs provide level of interpretability, discussed in (1), which holds significant value within this domain. Third, there is need to select orthogonal basis functions for AM-FNO (KAN).
13

---

Table 9: Comparison of the l2 relative enror on Darcy, Airfoil, and Elasticity benchmarks.
BeuchmarkAM-FNNOMMLP AM-FNO(KAN)
Geo-FNO
U-FNO
OFormer
LSM
F-FNO
Darey 4.72e-3
4.78c-3
1.08c-2
1.24c-2
1.24e-2
7.01e-3
9.92c-3
Airfoil
6.26c-3
6.62e-3
1.38c-2
1.05e-2
1.83e-2
6.59c-3
1.00e-2
Elasticity
2.10c-2
2.10c-2
2.29c-2
2.26e-2
2.25c-2
3.16e-2
G Impact Statements
This study introduces neural operator specifically designed for the effective solution of partial differential equations (PDEs), potentially contributing to advancements scientific and engineering domains. Positioned foundational research in machine learning, the immediate identification of negative consequences not evident, and the current risk misuse remains low.
14

---

NeurIPS Paper Checklist
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope?
Question:
Answer:[Yes Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 0
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com / casssnnnnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Section 5 and Appendix A
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the high computational cost of training all benchmarks, conducted each experiment only once.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section S
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://awaragcccca//a/////aaaaaa
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "87feucQmWS", "reviewer": "Reviewer_DLAH", "summary": "The paper introduces the AM-FNO to address high-frequency truncation in the original FNO, which can damage the performance for PDE data with substantial high-frequency information. AM-FNO utilizes MLP or KAN to approximate the kernel function value in Fourier space for all frequency modes. For MLP based AM-FNO, orthogonal embedding functions are applied to enhance MLP's performance. Factorization are applied to reduce the total number of basis functions. Experiments on various PDE datasets show that AM-FNO consistently outperforms baseline models. The efficacy of different components in AM-FNO is valedated through ablation experiments.", "questions": "1. The paper states in line 110 that 'using an NN guarantees the kernel function to evolve more smoothly as the frequency mode changes, due to the smoothness of NN'. However, given the well-known low-frequency bias of MLPs [4], this smoothness might actually be the reason for MLP's suboptimal performance without orthogonal basis functions. A possible explanation for the success of orthogonal basis functions is that they introduce higher frequency components into the MLP’s input, similar to Fourier features, thereby reducing the MLP's smoothness. Could the authors clarify whether a smoother or less smooth MLP is preferable in this context?\n\n2. Taking a 2D PDE as an example, a real-valued function transforms into a centrally symmetric complex function in Fourier space. Does the NN in AM-FNO ensure this central symmetry, such that inputs k_x,k_y produce the same output as -k_x, -k_y?\n\n3. Does AM-FNO require more GPU memory or more training time compared to the original FNO? (It appears that the Vanilla in Table 4 is not the same as the original FNO in Table 2, given the differences in their reported errors.)\n\n[4] Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., ... & Courville, A. (2019, May). On the spectral bias of neural networks. In International conference on machine learning (pp. 5301-5310). PMLR.", "limitations": "The authors addressed the limitations of the AM-FNO method.\n\nNo potential negative societal impact is noted.\n\nTo enhance the paper, please consider address the outlined weaknesses and questions, particularly the role of orthogonal basis functions. Given their significant benefits, a more detailed analysis explaining why these functions are effective would strengthen the paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents an original and intuitive method using MLP or KAN to approximate the kernel function values in Fourier space, contrasting with the original FNO's approach that requires separate linear functions for each frequency mode, which often leads to a high parameter count in high-dimensional PDEs.\n\n2. The manuscript is overall well-written, with clear descriptions of the methods used. \n\n3. AM-FNO has been tested across multiple datasets and consistently outperforms various baselines. Detailed abltation experiments are provided to validate each component's contribution.", "weaknesses": "1. It seems that orthogonal basis functions play a crucial role in boosting AM-FNO's performance. According to Table 4, MLP-based AM-FNO without these functions might even perform worse than the standard FNO. However, the use of orthogonal basis functions in the design is not convincingly motivated. The justification provided—that 'vanilla MLPs lack effective inductive bias for function approximation' (line 135)—is ambiguous. This explanation leaves the impression that orthogonal basis functions are used as an arbitrarily trick to enhance MLP's performance.\n\n2. Using Fourier series as latent features is a widely used method to improve MLP's convergence [1, 2]. The use of orthogonal embedding functions in AM-FNO is very close to how Fourier features work. It's better to discuss this common Fourier features method in the paper and test it performance following the setting of Table 4.\n\n3. The discussion and evaluation lack consideration of AFNO [3], a closely related baseline. AFNO features a key design of weight sharing, where a single MLP approximates the kernel function for all frequency modes in Fourier space. While AFNO's MLP does not take frequency mode as input like AM-FNO does, its motivation and implementation are quite similar to those of AM-FNO.\n\n\n[1] Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., ... & Ng, R. (2020). Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33, 7537-7547.\n\n[2] Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2021). Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1), 99-106.\n\n[3] Guibas, J., Mardani, M., Li, Z., Tao, A., Anandkumar, A., & Catanzaro, B. (2021). Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer DLAH for the acknowledgment of our method and empirical contributions. Below, we respond to the questions.\n\nW1: The motivation for orthogonal basis functions.\n\nSorry for the lack of clarity. We make the following clarifications. As shown in Table 4, the performance degrades compared to the version with KAN when using an MLP to directly approximate the mapping between frequencies and kernel values. We attribute this degradation to the MLP's difficulty in capturing the complex, non-linear nature of the kernel function. Embedding scalar inputs with predefined functions is a well-established technique in machine learning, as exemplified by time embeddings in diffusion models. This approach allows MLPs to more effectively transform linear inputs into expressive, non-linear representations. Inspired by this, we employ orthogonal functions to embed the frequencies, mapping them into a high-dimensional and non-linear feature space. This method introduces an inductive bias that enhances the efficiency of approximating kernel functions. This method is supported by the theory of orthogonal functions and empirical results and has been studied in function approximation [1]. Further discussion will be provided in the revised version. Thanks for your question.\n\nW2: Discussion about the Fourier series.\n\nWe experimented with triangular basis functions (TBF), which are used as basis functions in the Fourier series, as shown in Table 4. While TBF yielded better results for the Darcy benchmark, they performed worse compared to Chebyshev basis functions in other cases. We hypothesize that TBFs better capture the periodic structure of the Darcy benchmark and will discuss this further in the revised version.\n\nW3: Discussion about AFNO.\n\nThanks for your suggestion. We evaluated AFNO on two benchmarks, as shown in the table below. The results indicate that AFNO has lower prediction accuracy compared to AM-FNO. We hypothesize that AFNO's lower accuracy is due to using the same MLP to evolve all frequency modes of the input functions, potentially limiting expressiveness, whereas AM-FNO transforms different frequency modes separately. Further discussion on AFNO will be included in the revised version.\n\n|Benchmark|AM-FNO(MLP)|AM-FNO(KAN)|AFNO|\n|------|--------|---------|--------|\n|Darcy|4.21e-3|4.28e-3|3.17e-2|\n|Airfoil|5.64e-3|6.06e-3|9.88e-3|\n\nQ1: The smoothness about MLP.\n\nThe term \"more smoothly\" in our context indicates that our method achieves smoother evolution compared to FNO, which parameterizes the kernel value point-by-point. This pointwise parameterization may ignore correlations between frequency modes, leading to a less smooth kernel. In contrast, our method uses an MLP to approximate the mapping between frequency modes and their corresponding kernel values, resulting in a smoother representation. As mentioned in W1, directly using an MLP might struggle to capture the complex structure of the kernel, but the orthogonal embedding helps address this issue.  However, we believe the improved performance is due to the increased expressiveness to model kernel functions, rather than just a matter of smoothness.\n\nQ2: The central symmetry in Fourier space\n\nYes. We ensure this central symmetry by obtaining the value at (-k_x, -k_y) from the NN prediction at (k_x, k_y) directly.\n\nQ3:  GPU memory and training time comparison.\n\nWe provide a comparison of the training time and memory consumption for AM-FNO and FNO (with all modes retained) in the table below, tested on the 2D Darcy benchmark with a resolution of $421 \\times 421$. The results indicate that AM-FNO (MLP) exhibits both reduced memory usage and shorter training times compared to FNO, which is attributable to its lower complexity. Although AM-FNO (KAN) demonstrates increased training time due to its architectural design, it still benefits from lower memory consumption. We conjecture that this advantage is particularly evident when solving PDEs with high resolution and high dimensions. A comprehensive analysis will be included in the updated version of the paper.\n\n|Model|Memory|Train Time|\n|-----|------|------|\n|AM-FNO(MLP)|**9.7G**|**43.1s**|\n|AM-FNO(KAN)|13.5G|83.8s|\n|FNO|14.9G|45.6s|\n\nWe are sincerely grateful for your valuable insights, which we firmly believe will significantly enhance the quality of our manuscript. If you have any more questions or find some mistakes, please feel free to correct us.\n\n[1] S Qian, YC Lee, RD Jones, CW Barnes, and K Lee. Function approximation with an orthogonal basis net. In 1990 IJCNN International Joint Conference on Neural Networks, pages 605–619. IEEE, 1990.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ttzY0YNh6S", "reviewer": "Reviewer_Z566", "summary": "This paper presents the AMortized Fourier Neural Operator (AM-FNO), which utilizes an amortized neural representation of the kernel function. It allows accommodating a variable number of frequency modes while using a fixed number of parameters compared to the Vanilla Fourier Neural Network.", "questions": "Q1) What is your rationale for employing an orthogonal basis? Have you experimented with applying depthwise convolution in FFT space, followed by non-linear transformations and another round of depthwise convolution?\n\nQ2) Could you please clarify why KAN consistently performs worse than MLP? Given that the dataset is noise-free, shouldn't KAN be expected to outperform MLP?\n\nQ3) The FFNO number reported does not correspond to the number in the original paper.\n\nQ4) Have you attempted using a low-rank approximation of the kernel weight in Fourier space within FNO?\n\nQ5) Difference between the proposed method and spectral neural operator.\n\nQ6) Why does having orthogonal embedding assist in AM-FNO? Have you experimented with employing the standard FFN directly? What type of orthogonal embeddings were used for the experiment?", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1) Amortized neural parameterization of the kernel function using MLP and KAN amortized neural parameterization of the kernel function.\n\nS2) The approach explores high-frequency components without increasing the parameters of the Fourier Neural Operator.", "weaknesses": "W1) Baselines seem limited, and GNOT, Transsolver, ONO, UNET, etc., need to be included.\n\nW2) The benchmarks have been tested consistently on a structured grid but have yet to be tested on an unstructured mesh.\n\nW3) The proposed approach resembles the spectral neural operator which uses the Fourier and Chebyshev series,\n\nW4) Using low-rank approximation to approximate the Fourier kernel or incorporating depthwise convolution with non-linearity in baselines will enhance comprehension of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Z566 for the valuable feedback. Below, we respond to the questions. **Please note that the additional tables and references are included in the Global Rebuttal due to character limits.**\n\nW1: More baselines.\n\nOur method aims to enhance Fourier neural operators (FNOs), a significant subclass of neural operators. Therefore, we primarily use FNOs as our baselines, and our method demonstrates superior performance. Additionally, we have included widely used baselines (OFormer) and a competitive baseline (LSM) to validate our method's effectiveness. In response to your suggestion, we include some mentioned baselines, as shown in Table B. It indicates that our method outperforms the additional baselines as well. We will include these results in the revised version.\n\nW2: Benchmarks on unstructured mesh.\n\nThanks for your suggestion. We evaluate our method on the Elasticity benchmark presented on point clouds [1] in Table C. To handle irregular geometry, we implemented the widely used Geo-FNO method to map the irregular meshes to and from uniform meshes [1]. Our results show that our method outperforms other neural operators that utilize the same learnable mapping. However, it performs worse than transformer-based neural operators, which naturally process input functions on irregular geometry as a sequence. We hypothesize that the learnable mapping introduces errors. We will include these results in the revised version.\n\nW3: Spectral Neural Operator Resemblance.\n\nWe would like to clarify that our method is **fundamentally different from SNO**. SNO utilizes Chebyshev polynomials to represent functions with finite sets of coefficients and learns the mapping between these coefficients. This results in band-limited operators that are restricted to generating frequencies within a fixed representation space.\nIn contrast, FNOs learn the direct mapping between functions and compute the integral operator in Fourier space for high inference speed. Empirically, FNOs demonstrate the ability to extrapolate to higher frequencies not seen during training, which SNOs lack due to their fixed representation limits [2]. As a variant of FNO, while AM-FNO employs orthogonal basis functions to approximate the kernel function—a common practice in function approximation—the neural operator itself is fundamentally distinct from SNO.\n\nW4: Low-rank approximation or depthwise convolution with non-linearity.\n\nFNO with low-rank approximation is similar to one of our baselines, F-FNO, which factorizes the kernel across different dimensions. The results in Table 2 (**in the paper**) demonstrate that AM-FNO outperforms F-FNO, highlighting the superiority of our amortized parameterization over the point-by-point parameterization in F-FNO. \n\nRegarding the non-linear convolution, AFNO [3] employs a non-linear convnet to evolve features at every frequency mode. We evaluate AFNO on two benchmarks, and the results, as shown in Table B, indicate that AFNO has lower accuracy. We hypothesize that this is because AFNO uses the same MLP to evolve all frequency modes of the input, potentially limiting expressiveness. In contrast, AM-FNO transforms different frequency modes separately. We will include further discussion on AFNO in the revised version.\n\nQ1: Our rationale for employing orthogonal basis functions stems from the observed performance degradation when using an MLP to directly approximate the mapping between frequencies and kernel values, as shown in Table 4 (**in the paper**). The MLP struggles to capture the complex, non-linear nature of the kernel function. Embedding scalar inputs with predefined functions is a well-established technique in machine learning, as exemplified by time embeddings in diffusion models. This approach helps MLPs convert linear inputs into expressive, non-linear forms effectively. Inspired by it, we employ orthogonal functions to embed the frequencies, mapping them into a high-dimensional and non-linear feature space. This introduces an inductive bias that enhances the efficiency of approximating kernel functions. Regarding the depthwise convolution in FFT space, please refer to our response in W4.\n\nQ2: As reported in [4], KAN does not consistently outperform MLP. As shown in Table 4 (**in the paper**), AM-FNO with KAN outperforms AM-FNO using MLP directly and performs worse than AM-FNO using orthogonal basis functions to embed the input. This indicates the effectiveness of our embedding technique. Meanwhile, KAN has the advantage of being extendable during training by increasing the number of local basis functions. For a fair comparison, we kept the parameter count constant and did not extend KAN during training. Given the results in Figure 4 (**in the paper**), it is plausible that AM-FNO (KAN) can outperform if we increase the number of basis functions during training.\n\nQ3: The F-FNO results were reproduced using the training settings and similar hyperparameters as other FNOs, as described in Section 5.1 and Appendix B, to ensure a fair comparison. The discrepancy with the original paper's results may stem from differences in training techniques. For instance, the original F-FNO paper employs methods like enforcing the first-order Markov property and adding Gaussian noise. Our training settings align with standard practices in this domain to maintain consistency across evaluations.\n\nQ4: Please refer to W4.\n\nQ5: Please refer to W3.\n\nQ6: Please refer to Q1 and Q2. We primarily used Chebyshev basis functions for the orthogonal embeddings in our experiments. In our ablation study, we also replaced them with triangular basis functions and non-orthogonal basis functions to evaluate their impact.\n\nWe hope this response clarifies any misunderstandings and addresses your concerns. If you have any further questions or identify any mistakes, please do not hesitate to let us know. We sincerely hope that you will reconsider and potentially increase the score for our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WtEw06prks", "reviewer": "Reviewer_NuVs", "summary": "Typically, FNOs require a large number of parameters when addressing high-dimensional PDEs or when a high threshold for frequency truncation is needed. To overcome this challenge, the authors introduce the Amortized Fourier Neural Operator (AM-FNO). Their method uses an amortized neural parameterization of the kernel function to handle an unlimited number of frequency modes with a fixed number of parameters. The authors provide two implementations of AM-FNO: one based on the Kolmogorov–Arnold Network (KAN) and the other using Multi-Layer Perceptrons (MLPs) with orthogonal embedding functions.", "questions": "I'm not familiar with the Kolmogorov–Arnold Network (KAN), which is new and controversial as far as I know, so I will not comment on it.\n\n1. It is unclear to me how this can reduce the number of parameters. Suppose for one channel in FNO, the size of the (complex) kernel is $k$. Doesn't the MLP used to generate this kernel must contain more parameters? Otherwise, let's say you want a kernel of size $k$. The output size of your MLP is $k$, then the $W$ matrix in your output layer should be of size $k' \\times k$, where $k'$ is the size of the output of the previous layer.\n\n2. Assume that you can use an MLP with fewer parameters to generate a large kernel. Doesn't this also limit the expressivity of the kernel you can learn? Essentially, what you are doing is trying to reduce the dimension of the kernel, but Fourier is already very effective in this. If your data is translationally invariant or your PDE is translationally equivariant (which is an assumption of FNO), then the optimal PCA basis consists of Fourier vectors, and you can relate this to the Eckart-Young theorem.\n\n3. In Table 2, why is the number of parameters and running time not reported? Are the experiments repeated several times to reduce the effect of randomness? Without this information, Table 2 means nothing to me.\n\nI apologize if the authors think my comments are too harsh. If I have indeed misunderstood certain parts of this paper, I am open-minded for a discussion and willing to adjust my views during the rebuttal.", "limitations": "Ｔhe authors have addressed the limitations．", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Improved performance: AM-FNOs, as shown by the authors, consistently achieve better performance across multiple benchmarks.", "weaknesses": "1. The curse of dimensionality (CoD) is not lessened by any extent. For each Fourier layer in FNO, we have to perform FFT and IFFT, which have a complexity of \\( O(n \\log n) \\). Even if we use the full spectrum, the pointwise multiplication would only take \\( O(n) \\). It remains unclear to me whether it is meaningful to tackle the CoD issue in the number of parameters.\n2. The proposed method seems to have some flaws. See Questions.\n3. The presentation of this paper is very poor. For example, the way FNO is presented is non-standard and seems to be more from the view of the actual implementation. For example, in Section 3.2, $d_h$, correct me if I'm wrong, is the number of channels (i.e., width in FNO implementation). You should clearly state this. In line 98, $R(k): \\mathcal{E} \\rightarrow \\mathbb{C}^{\\left(d_h \\times d_h\\right)}$, you have $d_h$ as the number of input channels and $d_h$ as the number of output channels. That's why your codomain is $\\mathbb{C}^{\\left(d_h \\times d_h\\right)}$, and these two do not necessarily equal, although in the implementation from the FNO paper they are equal. If you choose to present FNO from an actual implementation perspective, you should clearly explain everything, especially for readers who are not familiar with the actual implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer NuVs for the valuable feedback. Below, we respond to the questions.\n\nW1: The reason for tackling the CoD issue in the number of parameters.\n\nIn FNO, the kernel is parameterized independently for each frequency mode, resulting in complexity for the Fourier integral operator of $O((d_h^2 k)^{D}))$, where $d_h$ represents both the input and output channel dimensions, $k$ denotes the number of retained modes, and D indicates the spatial dimensionality. For high-resolution and high-dimensional data, this leads to significant memory consumption, as a large number of modes need to be retained. For example, one 64-width FNO layer for 2D PDEs can exceed 130 million parameters with $k=256$. This severely limits practical applications, such as large-scale pre-trained models. \n\nWe provide a comparison of the training time and memory consumption for AM-FNOs and FNO ( with all modes retained) in the table below, tested on the 2D Darcy benchmark with a resolution of $421 \\times 421$. The results indicate that AM-FNO (MLP) exhibits both reduced memory usage and shorter training times compared to FNO, which is attributable to its lower complexity. Although AM-FNO (KAN) demonstrates increased training time due to its architectural design, it still benefits from lower memory consumption. We conjecture that this advantage is particularly evident when solving PDEs with high resolution and high dimensions, attributed to AM-FNO's reduced complexity. A comprehensive analysis will be included in the updated version of the paper.\n\n|Model|Memory|Train Time|\n|-----|------|------|\n|AM-FNO(MLP)|**9.7G**|**43.1s**|\n|AM-FNO(KAN)|13.5G|83.8s|\n|FNO|14.9G|45.6s|\n\nW2: Paper presentation.\n\nThank you for your feedback on the presentation. We apologize for any confusion caused by the brief introduction of FNO due to page limitations. We will provide a clearer explanation in the updated version to ensure understanding, especially for readers not familiar with the implementation details.\n\nQ1: Parameter reduction\n\nWe would like to clarify that in AM-FNO, the MLP is used to map each frequency mode (after embedding) to its corresponding kernel value, meaning the number of parameters depends on the number of channels rather than the number of retained modes. For instance, if an MLP with $m$ basis functions processes input and outputs a kernel of size $k$, the parameter count is approximately $2m^2 + 2m$ (assuming one hidden layer and one channel). In contrast, FNO requires $k$ parameters for the kernel, which can be expensive for high-resolution and high-dimensional data (more modes should be retained).\n\nQ2: MLP limits the expressivity.\n\nWhile Fourier methods are theoretically efficient, they require a large number of parameters when handling high-dimensional and high-resolution data, which can be computationally prohibitive. Using an MLP to generate the kernel might indeed limit expressivity compared to parameterizing every frequency mode as in FNO theoretically. We regard this as a tradeoff. However, our empirical results demonstrate that AM-FNO outperforms FNO covering all frequency modes (see Table 3). We believe this is because the smoother transformations facilitated by the MLP improve optimization efficiency, leading to better performance.\n\nQ3: Main experiments.\n\nMany results in Table 2 are sourced from the original papers, making a direct comparison of running times potentially unfair due to hardware differences. All the baselines in Table 2 have comparable numbers of parameters, with their hyperparameters detailed in Appendix B. Below, we provide a parameter comparison on Darcy benchmark to validate this. The results of neural operators are generally consistent despite randomness, which is why many popular baselines do not report repeated results [1,2]. We repeated our experiments three times on CFD-2D and NS-2D benchmarks, as shown in the table below, demonstrating the stability of our method.\n\n| |AM-FNO(MLP)|AM-FNO(KAN)|FNO|U-FNO|OFormer|LSM|\n|------|--------|---------|------|-------|---------|--------|\n|Params.(M)|1.1|1.5|2.4|1.3|2.7|4.8|\n\n|Benchmark|AM-FNO(MLP)|AM-FNO(KAN)|\n|------|--------|---------|\n|NS-2D|8.53e-2 ± 7.48e-4|1.04e-1 ± 3.27e-3|\n|CFD-2D|2.21e-3 ± 4.13e-5|2.75e-3 ± 8.96e-5|\n\nThank you for your openness and willingness to discuss further. We hope this discussion clarifies any misunderstandings and addresses your concerns. If you have any further questions or identify any mistakes, please do not hesitate to let us know.\n\n[1] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2020). Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895.\n\n[2] Hao, Z., Wang, Z., Su, H., Ying, C., Dong, Y., Liu, S., ... & Zhu, J. (2023, July). Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learning (pp. 12556-12569). PMLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C6VXFQyw0j", "reviewer": "Reviewer_okt7", "summary": "This paper introduces Amortized Fourier Neural Operators (AM-FNOs), a novel approach to improve Fourier Neural Operators (FNOs) for solving PDEs. The key contributions to at least me are:\n\n1. An amortized neural parameterization of the kernel function in FNOs to accommodate arbitrarily many frequency modes using a fixed number of parameters.\n2. Two implementations of AM-FNO: one based on Kolmogorov-Arnold Networks (KAN) and another using Multi-Layer Perceptrons (MLPs) with orthogonal embedding functions.\n3. Theoretical analysis of the approximation capabilities of AM-FNOs.\n4. Extensive empirical evaluation demonstrating significant performance improvements over existing neural operator baselines across diverse PDE benchmarks.", "questions": "I just have a few, maybe it wont be answered in the rebuttal phase, but just want to know if the authors have done any studies on this:\n1. How does the performance of AM-FNOs compare to multi-scale approaches like U-FNO for PDEs with significant multi-scale behavior?\n2. The paper focuses on the forward problem of solving PDEs. How well might AM-FNOs perform on inverse problems or parameter estimation tasks?\n3. How does the computational complexity of AM-FNOs compare to standard FNOs and other baselines, particularly for high-dimensional PDEs?\n4. The paper mentions the potential of KANs for interpretability. This is evident from the original KAN paper as you can recover symbolic expressions. Could the authors elaborate on how this interpretability could be leveraged in the context of PDE solving? Would the recovered be a close enough analytic solution ( which often in PDEs we dont have any).", "limitations": "The authors have described their limitations!", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novelty: The paper presents a novel approach to address a significant limitation of FNOs - the trade-off between model complexity and the ability to represent high-frequency details. The amortized parameterization is an innovative solution to this problem.\n2. Theoretical foundation: The authors provide a solid theoretical analysis of their approach, including a theorem on the approximation properties of orthogonal basis functions. This adds depth to the empirical results and helps understand why the proposed method works.\n3. Comprehensive experiments: The evaluation is thorough, covering six diverse PDE benchmarks and comparing against multiple state-of-the-art baselines. The inclusion of both in-distribution and out-of-distribution tests, as well as zero-shot super-resolution experiments, strengthens the claims of generalization ability.\n4. Performance improvements: The reported improvements in accuracy (up to 35% average reduction in relative error) are substantial and consistent across different PDE types, which is impressive given the diversity of the benchmarks.\n5. Ablation studies: The paper includes detailed ablation studies that provide insights into the importance of different components of the proposed method, such as the orthogonal embedding and the dimensional factorization trick.", "weaknesses": "1. Inadequate baseline tuning: A major weakness is that the baselines were not properly tuned. The authors use default hyperparameters or settings from previous papers for the baseline models, which may not be optimal for the specific benchmarks used in this study. This raises questions about the fairness of the comparisons and the true extent of AM-FNO's improvements.\n2. Limited discussion on scalability: The paper focuses on 1D and 2D PDEs. A more in-depth discussion on the scalability of the approach to higher-dimensional problems would be valuable.\n3. Computational efficiency: While the paper discusses the parameter efficiency of AM-FNOs, it doesn't provide a comprehensive analysis of the computational efficiency in terms of training time and inference speed compared to baseline methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer okt7 for recognizing the novelty and empirical contributions of our method. Below, we address the questions raised.\n\nW1: Inadequate baseline tuning.\n\nWe appreciate the reviewer's concern regarding baseline tuning. To ensure a fair comparison, we have adjusted all models in Table 2 to maintain comparable parameters, with hyperparameters detailed in Appendix B. Specifically, we standardized the width and depth across all FNOs baselines (including AM-FNO) to assess the effectiveness of our amortized parameterization. Furthermore, for the critical hyperparameter of FNO, the number of retained modes, we increased it to cover all modes as demonstrated in Table 3. Even under these conditions, our methods consistently outperformed the baselines.\n\nW2: Limited discussion on scalability.\n\nCurrently, our study focuses on 1D and 2D PDEs. As demonstrated in Table 3, covering all frequency modes with FNO requires nearly 9 times the number of parameters, which highlights its limitations even in 2D PDEs. We evaluate our method on a 3D benchmark, Plasticity [1], and the results, as shown in the table below, illustrate its efficiency and scalability. We will provide a more detailed discussion in the revised version.\n\n|Benchmark|AM-FNO(MLP)|AM-FNO(KAN)|Geo-FNO|\n|------|--------|---------|---------|\n|Plasticity|3.04e-3|6.11e-3|7.4e-3|\n\nW3:Computational efficiency.\n\nIn the table below, we compare the training times, inference speeds, and GPU memory usage for AM-FNOs and FNO on the 2D Darcy benchmark with a resolution of $421 \\times 421$. The results indicate that AM-FNO (MLP) exhibits both reduced memory usage and shorter training times compared to FNO, which is attributable to its lower complexity. Although AM-FNO (KAN) demonstrates increased training time due to its architectural design, it still benefits from lower memory consumption. We conjecture that this advantage is particularly evident when solving PDEs with high resolution and high dimensions. During inference, AM-FNOs (with kernels generated by MLP or KAN being precomputed) exhibit a similar speed to FNO, as both methods rely on similar kernel calculations. We will include a more thorough discussion in the revised version of the paper.\n\n|Model|Memory|Train Time|Inf Time|\n|-----|------|------|------|\n|AM-FNO(MLP)|**9.7G**|**43.1s**|2.4s|\n|AM-FNO(KAN)|13.5G|83.8s|2.2s|\n|FNO|14.9G|45.6s|2.2s|\n\nQ1 & Q2: Multi-scale, inverse problem and parameter estimation tasks.\n\nWe have evaluated AM-FNOs on the Airfoil benchmark, which has some multi-scale features, and achieved state-of-the-art results. A more detailed assessment of AM-FNO on benchmarks with significant multi-scale behavior will be addressed in future work. Our focus is on the forward problem for PDEs. Its effectiveness on inverse problems and parameter estimation is yet to be explored and will be also considered in future research.\n\nQ3: Computational complexity.\n\nPlease refer to W3.\n\nQ4: The interpretability of KAN.\n\nThank you for your insightful question. In AM-FNO, the black-box nature of the MLP between Fourier integral operators limits interpretability. However, leveraging KANs for symbolic expression recovery is promising and could enhance interpretability in future work. This is an area we plan to explore further.\n\nWe sincerely appreciate your valuable insights and corrections. We will revise our manuscript accordingly. If you have any further questions or identify any mistakes, please feel free to correct us.\n\n[1] Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207.05209,2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1jXP8LvB1J", "reviewer": "Reviewer_kDgT", "summary": "This paper tries to alleviate one of the issues in a so-called \"Fourier Neural Operator (FNO)\" which is a machine learning model to estimate a solution of partial differential equations (PDEs) based on the concept of \"Operator learning\". FNO takes into account the embedding of input field information in Fourier space, which partially contributes to its very weak dependence on the resolution. However, to reduce numerical cost, FNO abandons the frequency modes exceeding a predefined threshold, which limits its expressiveness power in the high-frequency region. To alleviate this issue, this paper proposed \"Amortized Fourier Neural Operator (AM-FNO)\", which utilizes a neural kernel function to accommodate arbitrarily many frequency modes with a fixed number of parameters, resulting in more expressive power with moderate model size.", "questions": "1.  In sec 3.1 above of Eq. (1), what does it mean: \"functions a_i and u_i\"? Do the authors want to indicate that each numerical cell at i owns independent functions?\n\n2. In Eq. (5), the authors use NN(k) for both real and imaginary parts. Do the authors want to indicate the real and imaginary parts share the same weight? (it seems not). It would be better to describe them as NN(k; \\theta_real) and NN(k; \\theta_imaginary). \n\n3. Below Eq. (8), lines 129-130, the description seems to contradict the result in Table 4 (KAN is worse than MLP). Do the authors indicate the case \"Non\"? Seemingly, the description is insufficient. \n\n4. Why have the authors introduced the well-known \"Stone-Weierstrass theorem\" [1,2] (or Weierstrass approximation theorem) as Theorem 4.1 without citing it? Does the authors' version include non-trivial improvement? In addition, the function f should not be an \"arbitrary function\" but a \"continuous function\". \n\n          [1] Stone, Marshall Harvey. \"Applications of the theory of Boolean rings to general topology.\" Transactions of the American Mathematical Society 41.3 (1937): 375-481.\n          [2] Stone, Marshall H. \"The generalized Weierstrass approximation theorem.\" Mathematics Magazine 21.5 (1948): 237-254.\n\n5. (suggestion) AM-FNO(KAN) can be moved into the appendix to increase the space to explain other important information because AM-FNO(KAN) is consistently worse than AM-FNO(MLP).", "limitations": "The authors addressed the limitation in the manuscript.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed AM-FNO allows to accommodate arbitrarily many frequency modes with a fixed number of parameters, resulting in more expressive power with moderate model size. Numerical experiments indicate that it improves not only the high-frequency regime but most regimes (mainly the low-frequency regime, surprisingly (Fig. 3))\n\n2. The experiments utilize various datasets covering diverse PDEs with 1D and 2D spatial coordinate systems. \n\n3. AM-FNO keeps a near-resolution independent nature of FNO even though considering high-frequency information.", "weaknesses": "The following major weaknesses are the factors forcing me to give a relatively lower score on this paper: \n\nMajor weakness: \n\n1. Each experiment in the paper is conducted only once, which prohibits readers (and reviewers) from distinguishing whether the obtained improved performance is either due to the true-effectiveness of the proposed approach or just a lucky statistical fluctuation of optimizer and initial model weight. In particular, the training sample number is 1000, which is relatively small and can result in large statistical fluctuation. The justification in \"Checklist\" says that this is due to \"computational cost\", which does not validate anything but sounds like the authors lazily submit this version of the paper just for the deadline (I hope not). Following the ML convention, averaging over three to five times experiments with the standard deviation information to the result table are necessary before acceptance. I feel this crucial defect unnecessarily decreases the worth of this paper, though the proposed method seems important and very interesting. \nAt least, NS-2D and CFD-2D results in Table 2 should be given with standard deviation values because of their solution complexity, which can cause strong statistical fluctuations. \n\n2. No information on the validation dataset is provided. Explain how to record the best score. If not using the validation dataset, provide a clear explanation of not overfitting to the test dataset.\n\nMinor: \n1. \"related works\" only introduces the work relating to the neural operators, which covers a part of ML for PDE. The authors should introduce the other ML models for PDE and explain why picked up the neural operator and FNO in this paper (maybe in Appendix). \n\n2. There are several undefined symbols and terms, in particular, in Section 3, such as d_a, d_u, d (seemingly already observed in FNO paper..), and FNN in the caption of Figure 2. \n\n3. The description of \"Factorization trick for high-dimensional PDEs\" seems too short. I encourage the authors to provide a more comprehensive explanation with mathematical descriptions either in the main-body or appendix.  \n\n4. No discussion on the training/inference time and memory consumption required for AM-FNO in comparison to FNO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer kDgT for the detailed feedback. Below, we respond to the questions.\n\nW1: Experimental repetition.\n\nThe results for neural operators are relatively stable. Consequently, many results for widely used baselines are taken directly from the original papers, which also do not include repeated experiments [1,2]. Reproducing experiments for every baseline and benchmark multiple times is computationally intensive. To address this concern, we have conducted additional experiments with three repetitions for our method on NS-2D and CFD-2D benchmarks as shown in the table below. The standard deviations are minimal compared to the average values. We will include these results, along with standard deviation values, in the revised manuscript to better assess the statistical stability of our approach. We appreciate your feedback and will incorporate these improvements into the revised version.\n\n|Benchmark|AM-FNO(MLP)|AM-FNO(KAN)|\n|------|--------|---------|\n|NS-2D|8.53e-2 ± 7.48e-4|1.04e-1± 3.27e-3|\n|CFD-2D|2.21e-3 ± 4.13e-5|2.75e-3 ± 8.96e-5|\n\nW2: Validation dataset.\n\nThe primary objective of our experiments is to evaluate the effectiveness of our models across various benchmarks and to provide a comparative analysis with existing baselines. To ensure fairness and consistency, we use a fixed set of training settings in line with other baseline studies [1,2,3]. We do not perform hyperparameter tuning specifically for individual benchmarks in Table 2, as detailed in Appendix B. This approach aligns with standard practices in operator learning and helps mitigate the risk of overfitting by preventing excessive optimization for any single benchmark.\n\nW3: Related works.\n\nWe have briefly discussed our choice of neural operators over other neural network-based methods in the introduction. Due to space constraints, a more detailed comparison with other ML models for PDEs will be included in the revised version, potentially in the appendix.\n\nW4: Undefined symbols.\n\nThank you for pointing this out. In Section 3, $d_a$ and $d_u$ represent the dimensions of different functions, and FFN refers to the feed-forward neural network. We will provide detailed definitions for these terms in the revised version.\n\nW5: More description of the factorization trick.\n\nThe \"Factorization trick for high-dimensional PDEs\" was briefly introduced as it serves primarily as a supplementary technique rather than a core component of our method. We will provide a more detailed mathematical description and explanation in the appendix.\n\nW6: Training/inference time and memory consumption.\n\nWe present a comparison of training/inference times and memory consumption for AM-FNO and FNO on the 2D Darcy benchmark with a resolution of $421 \\times 421$ in the table below. The results indicate that AM-FNO (MLP) exhibits both reduced memory usage and shorter training times compared to FNO, which is attributable to its lower complexity. Although AM-FNO (KAN) demonstrates increased training time due to its architectural design, it still benefits from lower memory consumption. We conjecture that this advantage is particularly evident when solving PDEs with high resolution and high dimensions. During inference, AM-FNOs (with kernels generated by MLP or KAN being precomputed) exhibit a similar speed to FNO, as both methods rely on similar kernel calculations. We will provide a more detailed discussion of these aspects in the revised version of the paper.\n\n|Model|Memory|Train Time|Inf Time|\n|-----|------|------|------|\n|AM-FNO(MLP)|**9.7G**|**43.1s**|2.4s|\n|AM-FNO(KAN)|13.5G|83.8s|2.2s|\n|FNO|14.9G|45.6s|2.2s|\n\nQ1: $a_i$ refers to different input functions indexed by $i$, while $u_i$ denotes the corresponding output functions.\n\nQ2: Thanks for your suggestion. We will describe them differently in the revised version.\n\nQ3: We apologize for the confusion. The caption states, \"A version of the model without orthogonal embedding (Non) is included for comparison,\" which refers to the model that directly approximates the kernel with MLP, as opposed to the KAN approach. \n\nQ4: Citation of \"Stone-Weierstrass theorem\".\n\nThe Stone-Weierstrass theorem specifically addresses polynomial approximation of continuous functions, while our theorem generalizes approximation using orthogonal function bases [4], not limited to orthogonal polynomials. We will add the appropriate reference and clarify this distinction in the revised manuscript.\n\nQ5: Moving the KAN part.\n\nThank you for your suggestion. While AM-FNO(KAN) does not outperform AM-FNO(MLP), it offers significant advantages, such as extendability during training and interpretability, as discussed in Appendix E. These benefits highlight the value of the KAN approach, which we believe justifies its inclusion in the main text.\n\nWe hope this response clarifies any misunderstandings and addresses your concerns. If you have any further questions or identify any mistakes, please do not hesitate to let us know. We sincerely hope that you will reconsider and potentially increase the score for our paper.\n\n[1]Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2020). Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895.\n\n[2]Li, Z., Huang, D. Z., Liu, B., & Anandkumar, A. (2023). Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Research, 24(388), 1-26.\n\n[3]Li, Z., Meidani, K., & Farimani, A. B. (2022). Transformer for partial differential equations' operator learning. arXiv preprint arXiv:2205.13671.\n\n[4]https://www.math.uni-hamburg.de/home/gunesch/calc1/chapter11.pdf (Theorem 11.13)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude for the thoughtful reviews. We are pleased that the reviewers found our paper to be **overall well-written, with clear descriptions of the methods** (Reviewer DLAH), that our method is **original and intuitive** (Reviewer DLAH), **novel** (Reviewer okt7), our theoretical analysis is **solid** (Reviewer okt7), and that our experimental results **consistently outperform various baselines/achieve better performance** (Reviewer DLAH, Reviewer NuVs).

1) To address the concerns of Reviewers DLAH, NuVs, and okt7, we provide a comparison of GPU memory usage and inference/training times between our method and FNO, tested on the 2D Darcy benchmark with a resolution of 421 $\times$ 421.

2) We report the repeated results on NS-2D and CFD-2D benchmarks to address the concern of Reviewer NuVs and kDgT.

3) We add more baselines including AFNO, GNOT and ONO, and more benchmarks (3D Plasticity and Elasticity in point cloud [1]) to address the concern of Reviewer DLAH, Z566, okt7.

**Table A: Comparison of GPU Memory Usage and Inference/Training Times.**

|Model|Memory|Train Time|Inf Time|
|-----|------|------|------|
|AM-FNO(MLP)|**9.7G**|**43.1s**|2.4s|
|AM-FNO(KAN)|13.5G|83.8s|2.2s|
|FNO|14.9G|45.6s|2.2s|

**Table B: Comparison on Darcy and Airfoil benchmarks with additional baselines.**

|Benchmark|AM-FNO(MLP)|AM-FNO(KAN)|GNOT|ONO|AFNO|
|------|--------|---------|--------|--------|--------|
|Darcy|**4.21e-3**|4.28e-3|1.05e-2|7.20e-3|3.17e-2|
|Airfoil|**5.64e-3**|6.06e-3|7.57e-3|5.60e-3|9.88e-3|

**Table C: Comparison on Elasticity benchmark.**

|Benchmark|AM-FNO(MLP)|AM-FNO(KAN)|Geo-FNO|LSM|GNOT|ONO|
|------|--------|---------|--------|--------|--------|--------|
|Elasticity|2.03e-2|2.10e-2|2.29e-2|2.25e-2|8.6e-3|1.18e-2|

**Table D: Repeated results on NS2d and CFD2d benchmarks.**

|Benchmark|AM-FNO(MLP)|AM-FNO(KAN)|
|------|--------|---------|
|NS-2D|8.53e-2 ± 7.48e-4|1.04e-1± 3.27e-3|
|CFD-2D|2.21e-3 ± 4.13e-5|2.75e-3 ± 8.96e-5|

**Reference**

[1] Li, Z., Huang, D. Z., Liu, B., & Anandkumar, A. (2023). Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Research, 24(388), 1-26.

[2] Li, Z., Zheng, H., Kovachki, N., Jin, D., Chen, H., Liu, B., ... & Anandkumar, A. (2024). Physics-informed neural operator for learning partial differential equations. ACM/JMS Journal of Data Science, 1(3), 1-27.

[3] Guibas, J., Mardani, M., Li, Z., Tao, A., Anandkumar, A., & Catanzaro, B. (2021). Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587.

[4] Yu, R., Yu, W., & Wang, X. (2024). Kan or mlp: A fairer comparison. arXiv preprint arXiv:2407.16674.

Meta-review: The paper introduced Amortized Fourier Neural Operators (AM-FNO), which aim to alleviate the problem of limited frequency modes in FNOs. The key contribution of the work is to introduce a neural network-based function to map frequency to the Fourier transform of the kernel function. This is not as straightforward, and the authors carefully map these values using two neural networks (identical, but these map to real and imaginary parts respectively) and also augment the NN with orthogonal embedding functions. This parametrization leads to improvements both in terms of accuracy and efficiency. Therefore, we recommend an acceptance. 

We recommend that the authors to incorporate the edits suggested by the reviewers including the following.

1. Consider defining "." in eq (3)
2. line 111 -- "On the other hand, this guarantees the kernel function to evolve more smoothly as the frequency mode changes, due to the smoothness of NN". Consider clarifying how smoothness is ensured since smoothness is measured by differentiability and NN in general are non-smooth. Next, "more smoothly" does not convey technical meaning, since functions are either smooth or they are not (a binary). Do you intend to use the continuity of the function? 
3. Consider clarifying what kind of activations are used in the experiments.
4. Line 135 -- Consider adding support for or clarify the statement "vanilla MLPs lack effective inductive bias for function approximation." Since this is a contradiction to universal approximation property (stated in the previous statement and also , Hornik et al. (1989), Cybenko (1989), Funahashi (1989). Also, clarify or provide support for "vanilla MLPs".
5. Consider adding a line about the reason for calling the method "Amortized".
6. line 145 and 137 -- "we can implement the NN in defining Rp,q as a linear transformation upon a set of orthogonal functions" -- Since the discussion here is about MLPs this seems like a major switch in context since the entire discussion is centered around NNs. Moreover, Fourier basis can serve as orthogonal basis. Consider clarifying this. Also, consider making it clear (via mathematical expressions) what "augment" in "we propose to augment the MLP with orthogonal embedding functions to construct AM-FNO.", and including a clear rationale for employing these (some of the justification included in the rebuttal phase were not very direct and clear).
7. A description of the hyper-parameters and baseline settings should be included in the manuscript for reproducibility. 
8. Consider correcting the discussion about the computational complexity in the manuscript based on the discussion with the reviewers in the rebuttal phase. 
9. Any new data and tables which were presented during the rebuttals should be added to the manuscript/supplementary with the corresponding discussion.


Hornik et al. (1989). Multilayer feedforward networks are universal approximators. Neural networks, 2(5), 359-366.
Cybenko (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and
systems, 2(4):303–314, 1989.
Funahashi (1989) Funahashi, K.-I. On the approximate realization of continuous mappings by neural networks. Neural networks, 2(3):183–192, 1989.
Decision: Accept (poster)