Amortized Fourier Neural Operators

Authors: zipeng  xiao
siqi  kou
zhongkai  hao
bokai  lin
zhijie  deng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:34:04

Abstract: Fourier Neural Operators (FNOs) have shown promise for solving partial differential equations (PDEs).
Typically, FNOs employ separate parameters for different frequency modes to specify tunable kernel integrals in Fourier space, which, yet, results in an undesirably large number of parameters when solving high-dimensional PDEs. 
A workaround is to abandon the frequency modes exceeding a predefined threshold, but this limits the FNOs' ability to represent high-frequency details and poses non-trivial challenges for hyper-parameter specification. 
To address these, we propose AMortized Fourier Neural Operator (AM-FNO), where an amortized neural parameterization of the kernel function is deployed to accommodate arbitrarily many frequency modes using a fixed number of parameters. 
We introduce two implementations of AM-FNO, based on the recently developed, appealing Kolmogorov–Arnold Network (KAN) and Multi-Layer Perceptrons (MLPs) equipped with orthogonal embedding functions respectively. 
We extensively evaluate our method on diverse datasets from various domains and observe up to 31\% average improvement compared to competing neural operator baselines.

Main paper: 
 introduction: 
 Neural operators (NOs) have been extensively studied for their potential in accelerating the solving of partial differential equations (PDEs) in science and engineering fields [19, 20, 18, 24, 25, 2] . In contrast to approaches limited to specific discretizations or PDE instances [8, 36, 28, 29] , NOs characterize the solving operator of a family of PDEs across different discretizations and hence enjoy higher efficiency and usability, e.g., for weather forecasting [26] and material analysis [5] .Fourier neural operator (FNO) [18] and its variants [17, 33, 31, 35] stand out as a significant subclass of NOs, which explore the convolution theorem and Fast Fourier Transform (FFT) to efficiently performs kernel integral, a central module for the learning operator of PDEs. Typically, FNO separately parameterizes the values of the Fourier-transformed kernel function for different frequency modes, and hinges on frequency truncation-abandons the parameters corresponding to frequencies exceeding some threshold-to reduce modeling costs, particularly for high-dimensional PDEs.Frequency truncation can be problematic when solving PDE systems with intense high-frequency components. To address this, IFNO dynamically adjusts the threshold for frequency truncation during training, though it still experiences exponential parameter growth with increasing dimensionality [35] . This complexity can result in substantial memory consumption and hinder the development of largescale pretrained models [9] . Conversely, AFNO [7] employs a shared MLP to transform the outcomes of FFT, but the uniform treatment of frequency modes may constrain expressiveness and lead to suboptimal performance.Figure 1 : Comparison between FNO and AM-FNO: FNO assigns each value at the discretized frequencies of the Fourier-transformed kernel function as a learnable parameter, while AM-FNO utilizes neural network parameterization (MLP or KAN) to approximate the mapping between frequencies and function values. The frequencies are embedded using a set of orthogonal basis functions before being processed by the MLP.We address this by developing AMortized Fourier Neural Operator (AM-FNO), where we introduce extra neural networks (NNs) to specify the kernel integral operator to amortize the modeling cost. As illustrated in Figure 2 , AM-FNO is simple and intuitive-learnable NN transformations are leveraged to directly define the Fourier-space kernel function to accommodate arbitrarily many frequency modes at the cost of a fixed number of parameters. An amortized parameterization also provides an inherent regularization mechanism for resisting overfitting to potential high-frequency noise.The NN transformation in AM-FNO can be flexibly defined. One natural choice is the Kolmogorov-Arnold Network (KAN) due to its superior accuracy in function fitting [21] . Doing so, we, for the first time, reveal the potential of KAN for the operator learning of PDEs. Considering the widely criticized inefficiency issues of KAN in both time and memory consumption, we also investigate the regular Multi-Layer Perceptrons (MLPs) for amortized parameterization. We empirically identify the necessity of embedding the frequency modes with orthogonal basis functions before MLP transformation.We experiment on challenging benchmarks governed by diverse typical PDEs, covering six standard PDE benchmarks [18, 17, 30] . AM-FNO (KAN) and AM-FNO (MLP) achieve an average 22% and 31% reduction in relative error on these benchmarks, respectively. We also analyze the error of different frequency modes, and the results show that our models outperform in all frequency ranges. Additionally, we perform zero-shot super-resolution on three benchmarks to assess the generalization ability of AM-FNO across discretizations. We observe that AM-FNOs outperform the baselines and achieve lower error even compared to FNO trained on the test resolution. The results reflect that AM-FNOs have the promising potential to substantially reduce both the data collection and training costs for solving high-resolution PDE problems.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer:[Yes]Justification:Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Neural Operators. Neural operators have attracted considerable interest for their capacity to map infinite-dimensional function spaces [24, 19, 1] , thereby facilitating solutions across diverse discretizations without retraining. The pioneering work DeepONet [24] introduces a trunk and branch network architecture grounded in the universal approximation theorem for operators. Transformerbased neural operators represent a notable line of work in the domain. Galerkin Transformer [2] proposes self-attention operators that theoretically correspond to a learnable kernel integral operator and projection. OFormer [16] introduces an architecture with input and query encoders for querying arbitrary output locations. GNOT [10] proposes a heterogeneous normalized attention layer to encode different input information.FNO and its Variants. Fourier neural operator (FNO) [18] represents a novel approach with exceptional efficiency and accuracy due to its ability to learn kernel integral operators in the Fourier domain. The theoretical proof establishes its capability to approximate arbitrary continuous operator [15] . Tailored for addressing multiphase flow problems, U-FNO [33] augments its representation in higher frequency information through U-Net paths. Geo-FNO [17] extends the applicability of FNO beyond uniform grids by employing a learnable mapping from irregular domains to uniform latent meshes. F-FNO [31] reduces the model parameters and addresses performance degradation with increasing layers by factorizing the integral operator and enhancing the architecture. AFNO [7] transforms the function values after FFT with a shared MLP for each frequency, effectively reducing the parameter count. FNOs have made notable contributions across various challenging tasks [6, 25, 33, 26] . However, minimizing model complexity while effectively managing high-frequency information and ensuring generalization across different discretizations poses a persistent challenge in FNOs. Additionally, to the best of our knowledge, there has been no attempt to incorporate KANs with neural operators.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 This section presents the foundations of operator learning and the integral operator used in FNO.Consider the input function space A = A(D; R da ) and the target function space U = U(D; R du ) defined on a bounded and open set D ⊂ R d . Operator learning seeks to learn a θ-parameterized operator G θ to approximate the ground-truth mapping G : A → U specified by a PDE. This learning process is based on a finite set of function observations {a i , u i } N i=1 , where functions a i and u i are discretized on meshes {x j ∈ D} M j=1 . The optimization problem is:EQUATIONwhere the regular mean-squared error (MSE) is extended with a normalizer ∥u i ∥ 2 to handle scale variations across benchmarks, denoted as l 2 relative error. The relative error can also be substituted with other loss functions.A learnable kernel integral is a central module for defining mappings among functions. Specifically, we denote the hidden state of the input function in the l-th transformation stage as h (l) (y) : D → R d h , where d h represents the dimensionality, assumed to be consistent across all stages. The kernel integral operator makes the following transformation:(K(h (l) ))(x) = D κ(x, y)h (l) (y)dy, ∀x ∈ D (2)However, the integral is not computation-friendly within deep learning frameworks. To address this, FNO [18] assumes the kernel is shift-invariant, i.e., κ(x, y) = κ(x-y), and leverages the convolution theorem to efficiently compute the integral in the Fourier domain:EQUATIONwhere F and F -1 denote the FFT and its inverse (IFFT), and R(k) : E → C (d h ×d h ) represents the Fourier-transformed complex-valued kernel function with k ∈ E denoting a frequency mode. Typically, FNO individually parameterizes the values of R(k) for a fixed range of frequency modes (denoting the number as k T ) to avoid high modeling costs. This, yet, limits the exploration of highfrequency details in the function and poses non-trivial challenges for hyper-parameter specification.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification:Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
method: 
 This section elaborates on amortized FNO (AM-FNO), which amortizes an arbitrary number of frequency modes by sharing a fixed number of parameters.Instead of parameterizing the kernel function point-by-point, we propose to build the mapping between frequency and the values of the Fourier-transformed kernel function using an NN. On one hand, this mitigates the issue that the number of parameters increases significantly with that of frequency modes and dimensionality of PDEs. On the other hand, this approach avoids AFNO's uniform transformation, ensuring richer expressiveness in the Fourier domain.Concretely, by the rule of Fourier transformation, there is:EQUATIONNote that the output of R is a matrix, so we can use R p,q , p, q ∈ {1, 2, . . . , d h } to denote the complex scalar-valued function yielding one element of the matrix output. Our AM-FNO directly uses NNs to define the real and imaginary parts of the function R p,q to maximize the modeling flexibility. Formally, there isEQUATIONThe detailed implementations of the two NNs have no essential difference, so we only discuss one of them in the following.Kolmogorov-Arnold Networks (KANs) have been empirically shown to show promise in function approximation [21] . According to theoretical analysis, KANs are usually defined as:EQUATIONwhere η ′ j , η j,l : R → R denote the learnable basis functions. In practice, we can generalize the above definition and set a KAN layer as (with x as a H-dim variable):EQUATIONWe can specify the function by learnable coefficients and multiple local B-spline basis functions [21] :EQUATIONwhere r(x) represents a basis function (typically sigmoid linear unit or SiLU [3] ) and w is a factor controlling the magnitude.We utilize two two-layer KANs in each layer of AM-FNO to define the real and imaginary parts of R p,q . The inputs to the network are all frequencies of h (l) after FFT. In fact, we can share weights among the KANs associated with R p,q with different p and q. Empirical results (Table 4 ) indicate superior performance compared to the corresponding MLP implementation.In practice, KANs require extensive training time. To address this, we propose an alternative parameterization aimed at improving the performance of Multi-Layer Perceptrons (MLPs). Despite its universal approximation ability, MLPs empirically suffer from compromising performance. The spectral bias of vanilla MLPs, i.e., their tendency to favor low-frequency functions, may limit their capacity to represent more complex functions. Motivated by the success of leveraging orthogonal basis functions for function approximation [12, 14, 27] , we propose to augment the MLP with orthogonal embedding functions to construct AM-FNO.Specifically, we can embed the frequency mode input using a set of orthogonal functions before the MLP transformation. The Fourier basis is a natural choice due to its capacity to capture high-frequency components, thereby enhancing the high-frequency representation of vanilla MLPs. However, our empirical results show that Chebyshev basis functions can perform better (see Table 4 ). Of note, wea i L σ • • • Operator layer 1 Operator layer 2 σ σ Operator layer L P u i N 1 N 2 N L h (l) i FFT IFFT R FFN + h (l+1) i w x1 σ w x2Amortized neural parameterizationσ w y1 w y2 × k x k y k x k y T n (•) KAN MLPFigure 2 : AM-FNO structure for 2D PDEs: The input function a is mapped to a higher-dimensional space. Stacked operators and activation functions are applied for function propagation. Within the operator layers, a linear transformation R is applied to h (l) after FFT, followed by a feed-forward network (FFN) after the Inverse Fast Fourier Transform (IFFT). The values of R result from KAN or multiplying the MLP transformations of selected one-dimensional orthogonal basis functions (w denotes linear weights.). Finally, the function is projected to the solution dimension space.use only the first n max orthogonal basis functions in the family for the parameterization as we cannot employ infinite parameters.Factorization trick for high-dimensional PDEs. For the Fourier-transformed kernel, the dimensionality of the frequency modes matches that of the PDE being solved. To approximate functions with d-dimensional inputs, a common approach is to construct d-dimensional orthogonal functions based on one-dimensional basis functions. Specifically, for n max one-dimensional basis functions in each of the d dimensions, the complete space contains n d max high-dimensional basis functions, resulting in an exponential complexity for modeling.Motivated by the dimension factorization in the integral operator [31] , we separate the input dimensions and construct the final kernel by the products of the kernel approximated in each dimension. Such a process is illustrated in Figure 2 . In this way, the total parameter count scales linearly w.r.t. the dimension of the PDEs.In FNO, a stacked structure is employed to approximate the entire mapping G, as illustrated below:EQUATIONwhere L maps the input function a to the hidden state h (0) , and P maps the hidden state h (L) to the output function u, both in a point-wise manner. Q : h (l) → h (l+1) is the operator layer responsible for the iterative update:EQUATIONwhere K represents the kernel integral operator in Equation ( 3), and b represents the bias. FNO employs a pointwise linear mapping with W to enable the propagation of high-frequency information, but this is empirically limited, validated by the results in Table 8 .Given the non-truncated Fourier transform of the kernel function R in AM-FNO, we employ the operator layer architecture in [31] , which replaces the linear map W with residual connection [11] :EQUATIONAs shown in Figure 2 , we also incorporate activation functions between the operator layers for enhanced flexibility. The resultant model structure is: EQUATION 
experiment: 
 In this section, we validate the effectiveness of our proposed method by conducting extensive experiments on challenging benchmarks governed by typical solid and fluid PDEs.Benchmarks. We evaluate the performance of AM-FNO on six well-established benchmarks. These benchmarks include Burger, Darcy, and NS-2D, which are presented in regular grids with varying dimensions [18] . We extend our experiments to assess the method's performance in different geometries, including Pipe, Airfoil, and Elasticity benchmarks [17] Additionally, we incorporate the compressible fluid dynamics (CFD) 1D and 2D benchmarks [30] , which involves more high-frequency information. The benchmarks are summarized in Table 1 .Baselines. We conduct a comparative evaluation of our neural operator against seven baseline methods. These baselines include well-recognized approaches such as FNO [18] and its variants Geo-FNO [17] , U-FNO [33] , F-FNO [31] and AFNO [7] . Additionally, we consider other models, including OFormer [16] and LSM [34] . Notably, LSM represents the latest state-of-the-art (SOTA) neural operator among the baselines.Implementation details. We train all models for 500 epochs using the AdamW optimizer [23] with a cosine annealing scheduler [22] . The initial learning rate is 10 -3 , and the weight decay is set to 10 -4 . Our models consist of 4 layers with a width of 32 and process all the frequency modes of training data. The Gaussian Error Linear Unit (GELU) is used as the activation function [13] . For AM-FNO (KAN), the number of spline grids is selected from {24, 32, 48}, while for AM-FNO (MLP), the number of basis functions is set to 32 or 48. AM-FNO (MLP) utilizes Chebyshev basis functions as the orthogonal basis functions, as elaborated in the appendix. The batch size is selected from {4, 8, 16, 32}, and the experiments are conducted on a single 4090 GPU. The evaluation metric and training loss are based on the l 2 relative error in Equation ( 1), unless otherwise specified. We employ the transformation method from geo-FNO [17] to map between irregular input domains and uniform meshes for the Elasticity benchmark on point clouds. More details about the baselines can be found in Appendix A.The main results are shown in Table 2 . Our models consistently achieve state-of-the-art (SOTA) performance on all six benchmarks with various PDEs, geometries, and dimensions. AM-FNOs exhibit a significant average performance improvement of 22% from KAN implementation and 31% from MLP implementation compared to the top-performing baseline. We provide a comparison of GPU memory and training time in Appendix D, showing that although AM-FNO retains all frequency modes, it achieves comparable memory usage and training time to other FNOs.AM-FNOs significantly reduce prediction error on Darcy and NS-2D benchmarks, standard benchmarks with strong low-frequency components. Specifically, AM-FNO (KAN) reduces the error by 39% (2.73e-3) and 11% (1.40e-2), while AM-FNO (MLP) reduces the error by 40% (2.80e-3) and 30% (3.69e-2). This improvement can be attributed to AM-FNO's kernel parameterization, which effectively captures the low-frequency information. Additionally, AM-FNO demonstrates robustness across irregular geometries, with reductions of 5% (3.30e-4), 32% (1.66e-3) and 7%(1.5e-3) from AM-FNO (KAN), and 12% (7.50e-4), 34% (1.76e-3) and 10% (2.20e-3) from AM-FNO (MLP) in prediction error on Airfoil, Pipe and Elasticity benchmarks. For CFD-1D and CFD-2D benchmarks characterized by stronger high-frequency components, AM-FNO (KAN) achieves improvements of 25% (6.10e-3) and 40% (1.83e-3), while AM-FNO (MLP) achieves improvements of 40% (9.70e-3) and 52% (2.36e-3). The promotion highlights the effectiveness of our method of handling highfrequency components. We also find that AM-FNO (MLP) outperforms AM-FNO (KAN) across all benchmarks, likely due to the enhanced expressiveness of the orthogonal embedding.CFD-1D. To assess the performance across various frequency modes, we calculate the error of different frequency modes after FFT on CFD-1D benchmark and incorporate FNO without truncation (FNO + ) for comparison. The results are visualized in Figure 3 . As shown, the errors primarily stem from the first few frequency modes and decrease as the frequency increases. The baselines exhibit similar errors in the truncated frequency range, whereas our models demonstrate significantly lower errors. Our models maintain an advantage over the baselines in the initial modes within the truncated frequency range. As the frequency increases, the strength of the high-frequency components diminishes, and all the errors become negligible.CFD-2D. We further evaluate the performance across various frequency modes with the metrics outlined in [30] (detailed in Appendix B) and present the results on CFD-2D benchmark in Table 3 .The results showcase that FNO + demonstrates lower train error but higher test error than FNO and U-FNO. Notably, FNO + exhibits similar prediction errors to FNO in the high-frequency range, suggesting potential overfitting to the training data. We propose that the substantial complexity parameterization of FNO + may render it sensitive to high-frequency details, limiting the effectiveness of the additional parameters in handling such components. The fL2 error of U-FNO across all frequency ranges is lower than FNO, which can be attributed to the enhanced expressiveness enabled by the additional U-Net architecture. AM-FNO (MLP) achieves the lowest training error and fL2 error across all frequency ranges, while AM-FNO (KAN) achieves the second lowest fL2 error. Specifically, AM-FNO (MLP) and AM-FNO (KAN) achieve 55%(7.04e-2) and 50%(5.15e-2) reduction in the high-frequency range. This outcome can be attributed to our amortized parameterization, which significantly reduces model complexity while maintaining adequate expressiveness to approximate the Fourier-transformed kernel function.We conduct a detailed ablation study to assess the effectiveness of different components and hyperparameters of our models.Necessity of Orthogonal Embedding. We study the impact of orthogonal embedding on Darcy, Airfoil, and Pipe benchmarks. Table 4 presents the findings. Although AM-FNO (KAN) outperforms the MLP version without embedding (Non), its efficiency is significantly lower than that of versions utilizing MLPs. Removal of the orthogonal embedding leads to a notable performance decline across all three benchmarks. Meanwhile, replacing the orthogonal functions with non-orthogonal polynomial basis functions results in the highest prediction error among the baseline models. These outcomes showcase the efficacy and indispensability of the orthogonal embedding. To validate the robustness of the embedding, we use triangular basis functions and observe comparable errors on the Darcy and Pipe benchmarks, but higher errors on the Airfoil benchmark. We attribute this difference to the accuracy of function approximation achieved by Chebyshev basis functions.We conduct experiments to assess how prediction error varies with different numbers of basis functions, hidden sizes of KANs, and grid sizes of splines (linearly scaled with local spline count) in KANs. Figure 4 shows the results. The left figure illustrates that the error decreases with an increasing number of basis functions in orthogonal embedding on both benchmarks. This reduction is particularly evident initially on the Airfoil benchmark, followed by diminishing returns. In this case, employing 24 basis functions achieves a favorable balance between efficiency and accuracy. This performance enhancement can be attributed to the increased expressiveness from the additional orthogonal basis functions. In the middle figure, while the error decreases with increasing hidden size, we observe a decline in performance compared to KANs with larger grid sizes but smaller hidden sizes. This may suggest that the expressive power of KANs is primarily derived from the number of local spline functions (grid size). The right figure displays a trend similar to the left one: a noticeable decrease initially, followed by a less pronounced reduction later. In this study, we suggest increasing the grid size to enhance performance rather than focusing on adjusting the hidden layer size. We also present the performance of AM-FNOs, retaining the same frequency modes as other FNOs, in Table 9 . AM-FNOs consistently outperform baseline models, underscoring the advantages of our amortized parameterization over the standard FNO approach.A notable characteristic of neural operators is their ability to generalize across various discretizations.We conduct experiments training on lower resolution data and evaluate on higher resolution data on NS-2D benchmark. We visualize the results in Figure 5 and provide the numerical results in Table 5 . U-FNO exhibits a significant performance degradation when evaluated on higher resolution. This decline can be attributed to the convolutional operation in the U-Net architecture, which possesses a fixed receptive field and cannot effectively generalize across different discretizations. In contrast, FNO and AM-FNOs demonstrate the ability to generalize across different discretizations. Notably, AM-FNOs achieve superior performance at 64 × 64 resolution compared to baselines trained with the same resolution (see Table 2 ), which underscores the data efficiency of AM-FNOs. 
conclusion: 
 This paper proposes AM-FNOs to improve Fourier neural operator (FNO)'s efficiency in addressing PDEs without frequency truncation. Our approach utilizes Kolmogorov-Arnold Networks (KANs) and Multi-Layer Perceptrons (MLPs) with orthogonal embedding functions to mitigate exponential complexity and overfitting to high-frequency noise. Comprehensive experiments across various datasets demonstrate the effectiveness of AM-FNOs compared to baseline approaches.Limitations. This work attempts to enhance FNO's handling of high-frequency information but has the following limitations. The benchmarks used are idealized physical systems, excluding real-world complex problems. Meanwhile, although AM-FNOs reduce the parameter count, the extremely high-dimensional PDEs still pose challenges for FNOs due to the complexity of FFT.A Hyperparameters and Details for Models AM-FNOs. For AM-FNO (MLP), Chebyshev polynomials are chosen due to their favorable theoretical accuracy [32, 4] . Formally, they are defined as:EQUATION)The polynomial demonstrates an increase in the degree of its highest power term concerning k as n increases. It can be alternatively formulated as:EQUATIONFor AM-FNO (KAN), we fix the grid size during training to ensure a consistent parameter count. The order of the spline is fixed as 3.FNO and its Variants. We employ 4 layers with modes set to 12 and widths set to 32 for FNO and its variations (Geo-FNO, U-FNO, F-FNO). U-FNO incorporates the U-Net path in the last two layers. For AFNO, we set the width to 512 and use 4 layers to maintain a comparable parameter count. Notably, Geo-FNO reverts to the vanilla FNO when applied to benchmarks with regular grids, resulting in equivalent performance for Darcy and NS-2D benchmarks.LSM. The model is employed with 8 basis operators and 4 latent tokens. The width of the first scale is set to 32, with a downsampling ratio of 0.5.OFormer The depth of the encoder is fixed at 6, while the hidden dimension is set to 96.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Section 5 and Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Due to the high computational cost of training on all benchmarks, we conducted each experiment only once.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See Section 5.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: See Appendix G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Following [30] , we use fL2 error to quantify errors in different frequency ranges. It is computed as:EQUATION)where F denotes the FFT and the frequency range is restricted to k min ≤ k ≤ k max . For fL2 low, D Repeated results on NS-2D and CFD-2D benchmarks. kWe investigate the impact of the dimensional factorization trick and the model architecture. The results are shown in Table 8 . The results indicate that the version without dimensional factorization performs comparably. Dimensional factorization can be unnecessary for handling low-dimensional PDEs. Meanwhile, the employed architecture achieves an average error reduction of 20%.We also report the performance of AM-FNOs retaining the same number of frequency modes as other FNOs on the Darcy benchmark. The results, shown in Table 9 , indicate that AM-FNOs consistently outperform the baselines, highlighting the advantages of our amortized parameterization.F Discussion about KAN and MLP.Our findings in Section 5 demonstrate that implementing MLPs with orthogonal embeddings results in superior accuracy and efficiency. We attribute this success to the efficacy of the embedding technique employed. Meanwhile, the compatibility of the KAN architecture with widely used optimizers, such as AdamW, remains questionable. However, AM-FNO (KAN) offers several benefits over AM-FNO (MLP). First, as illustrated in Figure 4 , the expressiveness of KANs primarily stems from the grid size.To uphold a constant parameter count, the grid size remains unchanged. However, the architecture is inherently extensible during training, which could potentially enhance accuracy. Second, KANs provide a level of interpretability, as discussed in [21] , which holds significant value within this domain. Third, there is no need to select orthogonal basis functions for AM-FNO (KAN). This study introduces a neural operator specifically designed for the effective solution of partial differential equations (PDEs), potentially contributing to advancements in scientific and engineering domains. Positioned as foundational research in machine learning, the immediate identification of negative consequences is not evident, and the current risk of misuse remains low.NeurIPS Paper ChecklistJustification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper introduced Amortized Fourier Neural Operators (AM-FNO), which aim to alleviate the problem of limited frequency modes in FNOs. The key contribution of the work is to introduce a neural network-based function to map frequency to the Fourier transform of the kernel function. This is not as straightforward, and the authors carefully map these values using two neural networks (identical, but these map to real and imaginary parts respectively) and also augment the NN with orthogonal embedding functions. This parametrization leads to improvements both in terms of accuracy and efficiency. Therefore, we recommend an acceptance. 

We recommend that the authors to incorporate the edits suggested by the reviewers including the following.

1. Consider defining "." in eq (3)
2. line 111 -- "On the other hand, this guarantees the kernel function to evolve more smoothly as the frequency mode changes, due to the smoothness of NN". Consider clarifying how smoothness is ensured since smoothness is measured by differentiability and NN in general are non-smooth. Next, "more smoothly" does not convey technical meaning, since functions are either smooth or they are not (a binary). Do you intend to use the continuity of the function? 
3. Consider clarifying what kind of activations are used in the experiments.
4. Line 135 -- Consider adding support for or clarify the statement "vanilla MLPs lack effective inductive bias for function approximation." Since this is a contradiction to universal approximation property (stated in the previous statement and also , Hornik et al. (1989), Cybenko (1989), Funahashi (1989). Also, clarify or provide support for "vanilla MLPs".
5. Consider adding a line about the reason for calling the method "Amortized".
6. line 145 and 137 -- "we can implement the NN in defining Rp,q as a linear transformation upon a set of orthogonal functions" -- Since the discussion here is about MLPs this seems like a major switch in context since the entire discussion is centered around NNs. Moreover, Fourier basis can serve as orthogonal basis. Consider clarifying this. Also, consider making it clear (via mathematical expressions) what "augment" in "we propose to augment the MLP with orthogonal embedding functions to construct AM-FNO.", and including a clear rationale for employing these (some of the justification included in the rebuttal phase were not very direct and clear).
7. A description of the hyper-parameters and baseline settings should be included in the manuscript for reproducibility. 
8. Consider correcting the discussion about the computational complexity in the manuscript based on the discussion with the reviewers in the rebuttal phase. 
9. Any new data and tables which were presented during the rebuttals should be added to the manuscript/supplementary with the corresponding discussion.


Hornik et al. (1989). Multilayer feedforward networks are universal approximators. Neural networks, 2(5), 359-366.
Cybenko (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and
systems, 2(4):303–314, 1989.
Funahashi (1989) Funahashi, K.-I. On the approximate realization of continuous mappings by neural networks. Neural networks, 2(3):183–192, 1989.

Reviews: review: paper_summary: paper_summary: The paper introduces the AM-FNO to address high-frequency truncation in the original FNO, which can damage the performance for PDE data with substantial high-frequency information. AM-FNO utilizes MLP or KAN to approximate the kernel function value in Fourier space for all frequency modes. For MLP based AM-FNO, orthogonal embedding functions are applied to enhance MLP's performance. Factorization are applied to reduce the total number of basis functions. Experiments on various PDE datasets show that AM-FNO consistently outperforms baseline models. The efficacy of different components in AM-FNO is valedated through ablation experiments.
strength_weakness: strength_weakness: 1. It seems that orthogonal basis functions play a crucial role in boosting AM-FNO's performance. According to Table 4, MLP-based AM-FNO without these functions might even perform worse than the standard FNO. However, the use of orthogonal basis functions in the design is not convincingly motivated. The justification provided—that 'vanilla MLPs lack effective inductive bias for function approximation' (line 135)—is ambiguous. This explanation leaves the impression that orthogonal basis functions are used as an arbitrarily trick to enhance MLP's performance.

2. Using Fourier series as latent features is a widely used method to improve MLP's convergence [1, 2]. The use of orthogonal embedding functions in AM-FNO is very close to how Fourier features work. It's better to discuss this common Fourier features method in the paper and test it performance following the setting of Table 4.

3. The discussion and evaluation lack consideration of AFNO [3], a closely related baseline. AFNO features a key design of weight sharing, where a single MLP approximates the kernel function for all frequency modes in Fourier space. While AFNO's MLP does not take frequency mode as input like AM-FNO does, its motivation and implementation are quite similar to those of AM-FNO.


[1] Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., ... & Ng, R. (2020). Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33, 7537-7547.

[2] Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2021). Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1), 99-106.

[3] Guibas, J., Mardani, M., Li, Z., Tao, A., Anandkumar, A., & Catanzaro, B. (2021). Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587.
questions: questions: 1. The paper states in line 110 that 'using an NN guarantees the kernel function to evolve more smoothly as the frequency mode changes, due to the smoothness of NN'. However, given the well-known low-frequency bias of MLPs [4], this smoothness might actually be the reason for MLP's suboptimal performance without orthogonal basis functions. A possible explanation for the success of orthogonal basis functions is that they introduce higher frequency components into the MLP’s input, similar to Fourier features, thereby reducing the MLP's smoothness. Could the authors clarify whether a smoother or less smooth MLP is preferable in this context?

2. Taking a 2D PDE as an example, a real-valued function transforms into a centrally symmetric complex function in Fourier space. Does the NN in AM-FNO ensure this central symmetry, such that inputs k_x,k_y produce the same output as -k_x, -k_y?

3. Does AM-FNO require more GPU memory or more training time compared to the original FNO? (It appears that the Vanilla in Table 4 is not the same as the original FNO in Table 2, given the differences in their reported errors.)

[4] Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., ... & Courville, A. (2019, May). On the spectral bias of neural networks. In International conference on machine learning (pp. 5301-5310). PMLR.
limitations: limitations: The authors addressed the limitations of the AM-FNO method.

No potential negative societal impact is noted.

To enhance the paper, please consider address the outlined weaknesses and questions, particularly the role of orthogonal basis functions. Given their significant benefits, a more detailed analysis explaining why these functions are effective would strengthen the paper.
paper_summary: paper_summary: This paper presents the AMortized Fourier Neural Operator (AM-FNO), which utilizes an amortized neural representation of the kernel function. It allows accommodating a variable number of frequency modes while using a fixed number of parameters compared to the Vanilla Fourier Neural Network.
strength_weakness: strength_weakness: W1) Baselines seem limited, and GNOT, Transsolver, ONO, UNET, etc., need to be included.

W2) The benchmarks have been tested consistently on a structured grid but have yet to be tested on an unstructured mesh.

W3) The proposed approach resembles the spectral neural operator which uses the Fourier and Chebyshev series,

W4) Using low-rank approximation to approximate the Fourier kernel or incorporating depthwise convolution with non-linearity in baselines will enhance comprehension of the proposed method.
questions: questions: Q1) What is your rationale for employing an orthogonal basis? Have you experimented with applying depthwise convolution in FFT space, followed by non-linear transformations and another round of depthwise convolution?

Q2) Could you please clarify why KAN consistently performs worse than MLP? Given that the dataset is noise-free, shouldn't KAN be expected to outperform MLP?

Q3) The FFNO number reported does not correspond to the number in the original paper.

Q4) Have you attempted using a low-rank approximation of the kernel weight in Fourier space within FNO?

Q5) Difference between the proposed method and spectral neural operator.

Q6) Why does having orthogonal embedding assist in AM-FNO? Have you experimented with employing the standard FFN directly? What type of orthogonal embeddings were used for the experiment?
limitations: limitations: Yes, the authors adequately addressed the limitations.
paper_summary: paper_summary: Typically, FNOs require a large number of parameters when addressing high-dimensional PDEs or when a high threshold for frequency truncation is needed. To overcome this challenge, the authors introduce the Amortized Fourier Neural Operator (AM-FNO). Their method uses an amortized neural parameterization of the kernel function to handle an unlimited number of frequency modes with a fixed number of parameters. The authors provide two implementations of AM-FNO: one based on the Kolmogorov–Arnold Network (KAN) and the other using Multi-Layer Perceptrons (MLPs) with orthogonal embedding functions.
strength_weakness: strength_weakness: 1. The curse of dimensionality (CoD) is not lessened by any extent. For each Fourier layer in FNO, we have to perform FFT and IFFT, which have a complexity of \( O(n \log n) \). Even if we use the full spectrum, the pointwise multiplication would only take \( O(n) \). It remains unclear to me whether it is meaningful to tackle the CoD issue in the number of parameters.
2. The proposed method seems to have some flaws. See Questions.
3. The presentation of this paper is very poor. For example, the way FNO is presented is non-standard and seems to be more from the view of the actual implementation. For example, in Section 3.2, $d_h$, correct me if I'm wrong, is the number of channels (i.e., width in FNO implementation). You should clearly state this. In line 98, $R(k): \mathcal{E} \rightarrow \mathbb{C}^{\left(d_h \times d_h\right)}$, you have $d_h$ as the number of input channels and $d_h$ as the number of output channels. That's why your codomain is $\mathbb{C}^{\left(d_h \times d_h\right)}$, and these two do not necessarily equal, although in the implementation from the FNO paper they are equal. If you choose to present FNO from an actual implementation perspective, you should clearly explain everything, especially for readers who are not familiar with the actual implementation.
questions: questions: I'm not familiar with the Kolmogorov–Arnold Network (KAN), which is new and controversial as far as I know, so I will not comment on it.

1. It is unclear to me how this can reduce the number of parameters. Suppose for one channel in FNO, the size of the (complex) kernel is $k$. Doesn't the MLP used to generate this kernel must contain more parameters? Otherwise, let's say you want a kernel of size $k$. The output size of your MLP is $k$, then the $W$ matrix in your output layer should be of size $k' \times k$, where $k'$ is the size of the output of the previous layer.

2. Assume that you can use an MLP with fewer parameters to generate a large kernel. Doesn't this also limit the expressivity of the kernel you can learn? Essentially, what you are doing is trying to reduce the dimension of the kernel, but Fourier is already very effective in this. If your data is translationally invariant or your PDE is translationally equivariant (which is an assumption of FNO), then the optimal PCA basis consists of Fourier vectors, and you can relate this to the Eckart-Young theorem.

3. In Table 2, why is the number of parameters and running time not reported? Are the experiments repeated several times to reduce the effect of randomness? Without this information, Table 2 means nothing to me.

I apologize if the authors think my comments are too harsh. If I have indeed misunderstood certain parts of this paper, I am open-minded for a discussion and willing to adjust my views during the rebuttal.
limitations: limitations: Ｔhe authors have addressed the limitations．
paper_summary: paper_summary: This paper introduces Amortized Fourier Neural Operators (AM-FNOs), a novel approach to improve Fourier Neural Operators (FNOs) for solving PDEs. The key contributions to at least me are:

1. An amortized neural parameterization of the kernel function in FNOs to accommodate arbitrarily many frequency modes using a fixed number of parameters.
2. Two implementations of AM-FNO: one based on Kolmogorov-Arnold Networks (KAN) and another using Multi-Layer Perceptrons (MLPs) with orthogonal embedding functions.
3. Theoretical analysis of the approximation capabilities of AM-FNOs.
4. Extensive empirical evaluation demonstrating significant performance improvements over existing neural operator baselines across diverse PDE benchmarks.
strength_weakness: strength_weakness: 1. Inadequate baseline tuning: A major weakness is that the baselines were not properly tuned. The authors use default hyperparameters or settings from previous papers for the baseline models, which may not be optimal for the specific benchmarks used in this study. This raises questions about the fairness of the comparisons and the true extent of AM-FNO's improvements.
2. Limited discussion on scalability: The paper focuses on 1D and 2D PDEs. A more in-depth discussion on the scalability of the approach to higher-dimensional problems would be valuable.
3. Computational efficiency: While the paper discusses the parameter efficiency of AM-FNOs, it doesn't provide a comprehensive analysis of the computational efficiency in terms of training time and inference speed compared to baseline methods.
questions: questions: I just have a few, maybe it wont be answered in the rebuttal phase, but just want to know if the authors have done any studies on this:
1. How does the performance of AM-FNOs compare to multi-scale approaches like U-FNO for PDEs with significant multi-scale behavior?
2. The paper focuses on the forward problem of solving PDEs. How well might AM-FNOs perform on inverse problems or parameter estimation tasks?
3. How does the computational complexity of AM-FNOs compare to standard FNOs and other baselines, particularly for high-dimensional PDEs?
4. The paper mentions the potential of KANs for interpretability. This is evident from the original KAN paper as you can recover symbolic expressions. Could the authors elaborate on how this interpretability could be leveraged in the context of PDE solving? Would the recovered be a close enough analytic solution ( which often in PDEs we dont have any).
limitations: limitations: The authors have described their limitations!
paper_summary: paper_summary: This paper tries to alleviate one of the issues in a so-called "Fourier Neural Operator (FNO)" which is a machine learning model to estimate a solution of partial differential equations (PDEs) based on the concept of "Operator learning". FNO takes into account the embedding of input field information in Fourier space, which partially contributes to its very weak dependence on the resolution. However, to reduce numerical cost, FNO abandons the frequency modes exceeding a predefined threshold, which limits its expressiveness power in the high-frequency region. To alleviate this issue, this paper proposed "Amortized Fourier Neural Operator (AM-FNO)", which utilizes a neural kernel function to accommodate arbitrarily many frequency modes with a fixed number of parameters, resulting in more expressive power with moderate model size.
strength_weakness: strength_weakness: The following major weaknesses are the factors forcing me to give a relatively lower score on this paper: 

Major weakness: 

1. Each experiment in the paper is conducted only once, which prohibits readers (and reviewers) from distinguishing whether the obtained improved performance is either due to the true-effectiveness of the proposed approach or just a lucky statistical fluctuation of optimizer and initial model weight. In particular, the training sample number is 1000, which is relatively small and can result in large statistical fluctuation. The justification in "Checklist" says that this is due to "computational cost", which does not validate anything but sounds like the authors lazily submit this version of the paper just for the deadline (I hope not). Following the ML convention, averaging over three to five times experiments with the standard deviation information to the result table are necessary before acceptance. I feel this crucial defect unnecessarily decreases the worth of this paper, though the proposed method seems important and very interesting. 
At least, NS-2D and CFD-2D results in Table 2 should be given with standard deviation values because of their solution complexity, which can cause strong statistical fluctuations. 

2. No information on the validation dataset is provided. Explain how to record the best score. If not using the validation dataset, provide a clear explanation of not overfitting to the test dataset.

Minor: 
1. "related works" only introduces the work relating to the neural operators, which covers a part of ML for PDE. The authors should introduce the other ML models for PDE and explain why picked up the neural operator and FNO in this paper (maybe in Appendix). 

2. There are several undefined symbols and terms, in particular, in Section 3, such as d_a, d_u, d (seemingly already observed in FNO paper..), and FNN in the caption of Figure 2. 

3. The description of "Factorization trick for high-dimensional PDEs" seems too short. I encourage the authors to provide a more comprehensive explanation with mathematical descriptions either in the main-body or appendix.  

4. No discussion on the training/inference time and memory consumption required for AM-FNO in comparison to FNO.
questions: questions: 1.  In sec 3.1 above of Eq. (1), what does it mean: "functions a_i and u_i"? Do the authors want to indicate that each numerical cell at i owns independent functions?

2. In Eq. (5), the authors use NN(k) for both real and imaginary parts. Do the authors want to indicate the real and imaginary parts share the same weight? (it seems not). It would be better to describe them as NN(k; \theta_real) and NN(k; \theta_imaginary). 

3. Below Eq. (8), lines 129-130, the description seems to contradict the result in Table 4 (KAN is worse than MLP). Do the authors indicate the case "Non"? Seemingly, the description is insufficient. 

4. Why have the authors introduced the well-known "Stone-Weierstrass theorem" [1,2] (or Weierstrass approximation theorem) as Theorem 4.1 without citing it? Does the authors' version include non-trivial improvement? In addition, the function f should not be an "arbitrary function" but a "continuous function". 

          [1] Stone, Marshall Harvey. "Applications of the theory of Boolean rings to general topology." Transactions of the American Mathematical Society 41.3 (1937): 375-481.
          [2] Stone, Marshall H. "The generalized Weierstrass approximation theorem." Mathematics Magazine 21.5 (1948): 237-254.

5. (suggestion) AM-FNO(KAN) can be moved into the appendix to increase the space to explain other important information because AM-FNO(KAN) is consistently worse than AM-FNO(MLP).
limitations: limitations: The authors addressed the limitation in the manuscript.

Comments: title: Rebuttal Summary
Reply to Reviewer NuVs
Reply to Reviewer NuVs
Reply to Reviewer DLAH
Reply to Reviewer DLAH
Reply to Reviewer Z566
Reply to Reviewer DLAH
Reply to Reviewer DLAH
Reply to reviewer Reviewer NuVs
Reply to Reviewer Z566
Reply to Reviewer DLAH
Reply to Reviewer okt7
Reply to Reviewer Z566
Reply to Reviewer DLAH
Response to authors
Reply to Reviewer kDgT
reply
Reply to Reviewer kDgT
reply
Reply to Reviewer kDgT
reply
Reply to Reviewer kDgT
Reply to Reviewer NuVs
reply
Reply to Author's Rebuttal
comment: As the rebuttal period ends, we would like to briefly summarize some key points from the authors' perspective.

Firstly, we would like to express our gratitude for the thoughtful reviews and discussions. We are pleased that the reviewers found our paper to be **overall well-written**, **with clear descriptions of the methods** (Reviewer DLAH), that our method is **original and intuitive** (Reviewer DLAH) and **novel** (Reviewer okt7), our theoretical analysis is **solid** (Reviewer okt7), and that our experimental results **consistently outperform various baselines/achieve better performance** (Reviewer DLAH, Reviewer NuVs).

Secondly, we would like to conclude all the efforts we made during the rebuttal phase.

- Experiments with additional baselines: Include experiments with additional baselines, particularly AFNO.

- Experiments on additional benchmarks: Expand to include higher-dimensional settings and unstructured meshes.

- Efficiency comparison: Provide a comprehensive analysis including training time, inference time, and GPU memory usage.

- Clarify the differences between SNO and our method.

- Clarify our motivation about the orthogonal embedding.

- Experiment with our model using truncated modes or one layer.

- Explain why we did not report the repeated results.

- Explain why we did not use a validation dataset.


Based on the discussions so far, we have addressed most concerns raised by reviewers. 
Specifically, Reviewer DLAH has acknowledged that we have addressed all the major concerns and is convinced that our design "is valuable in terms of both accuracy and efficiency".
Reviewer Z566 also recognized that the concerns were addressed and agreed that our method improves performance.

To further improve our work and address the reviewer's concerns, we would make the following revisions:

- We will revise our motivation and incorporate spectral bias to explain the orthogonal embedding.  (Reviewer DLAH & Z566 & NuVs)

- We will include additional baselines (AFNO, DeepONet, SNO, etc.) and introduce more benchmarks involving higher dimensions and unstructured meshes. (Reviewer DLAH & Z566 & okt7)

- We will provide a more rigorous version of the theorem concerning orthogonal basis functions. (Reviewer kDgT)

- We will provide a more comprehensive analysis of efficiency and frequency truncation and discuss its limitations. (Reviewer DLAH & Z566 & NuVs & okt7)

We want to express many thanks to reviewers for asking questions or even challenging us. We will carefully consider all suggestions. This engagement in the rebuttal phase has been invaluable for improving our paper and deepening our understanding of our work.
Thank you for making such a constructive discussion with us!

We totally understand your concerns about method efficiency and have tried our best to demonstrate that our approach may not suffer from issues from that aspect during the rebuttal. Of course, we will add the mentioned comparison between large FNO and large AM-FNO in the revision. Additionally, we will provide a comprehensive comparison with DeepONet, particularly focusing on efficiency.

At last, we also clarify that alleviating the parameter count issue is a side product of this paper. A more evident effect of the proposed amortized parameterization is to foster the frequency modes to communicate with each other, leading to enhanced predictive performance. We will carefully revise the paper to weaken the argument on addressing COD in our revised version.

Thanks again!
> Geo-FNO is equivalent to the standard FNO in this context.

Your aim is to resolve or lessen the CoD issues in FNO. It would be more interesting to see a comparison between a large FNO and a large AM-FNO regarding the improvement in 1) memory usage, 2) training and inference time, and 3) FLOPs. However, I understand that due to the limited time available during the rebuttal, it may be almost impossible to obtain such results.

>  While FFT may become a limitation in very high-dimensional settings, FNO still outperforms in this context.

In Section 5.1, you mentioned DeepONet as a baseline, but I do not see it in the results (Table 2 or any other tables). I'm trying to find the L2 error information on this. Moreover, in my experience, DeepONet is usually faster than an FNO model with a similar number of parameters on normal uniform rectangular domain data (implementation follows directly from [1], a torch adaptation can also be found from the implemetation of [2]). However, I do understand that implementation details and hardware details may differ.


Given that most of my concerns and questions have been addressed, and recognizing the authors' efforts to include additional results within the short rebuttal period, I am willing to increase my score. However, since my major concern remains unresolved and the authors agree it should be acknowledged as a limitation, I will not be making any further adjustments to the score. 

[1] A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data

[2] Physics-Informed Neural Operator for Learning Partial Differential Equations
Thanks for your feedback and improved score. Below, we further respond to your questions.

1） Due to the time constraints of Rebuttal, we directly used the official Geo-FNO code to compare the speed of Geo-FNO and DeepONet [1]. Note that DeepONet requires function values at all input points and a query coordinate but outputs only the value at the coordinate, while FNO outputs values for all points. To align batch sizes, we used a batch size of $16$ for Geo-FNO and $16 \times 972$ (972 is the number of sensor points) for DeepONet. As shown in the table below, huge batch sizes for DeepONet can pose significant challenges for data transfer and parallel processing, resulting in reduced efficiency. While FFT may become a limitation in very high-dimensional settings, FNO still outperforms in this context. We also compare FLOPs for computing an entire function (972 points) to better illustrate the efficiency of the two models.

| Model    |   Params.(M) | Train Time (s/epoch) |   Inf Time (s/epoch)   | FLOPs（Calculation for 972 Points） |
|----------| ---------|---------| ------- |------- |
| DeepONet | 1.0 | 13.9 | 2.5 | 0.99B |
| Geo-FNO  | 1.5 | 2.1 | 0.2 | 0.11B |

Regarding U-Net, while it may be more efficient in many scenarios, its local convolution in the spatial domain limits performance when testing across different discretizations, which is significant for neural operators [2]. 

Thank you for your questions on efficiency. We recognize that the complexity of FFT may pose a limitation for FNOs and will include a detailed discussion on COD in our revised version.

2）The Plasticity benchmark is presented on a structured mesh, where the indexing induces a canonical coordinate map and enables the direct application of FFT  without the learnable mapping [3]. Consequently, Geo-FNO is equivalent to the standard FNO in this context. We refer to it as Geo-FNO to maintain consistency with the original Geo-FNO paper. Thank you for pointing this out.

We hope the above response can further address your concern. If you have any further questions, we are pleased to discuss them.

[1] https://github.com/neuraloperator/Geo-FNO

[2] Wen, G., Li, Z., Azizzadenesheli, K., Anandkumar, A., & Benson, S. M. (2022). U-FNO—An enhanced Fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources, 163, 104180.

[3] Li, Z., Huang, D. Z., Liu, B., & Anandkumar, A. (2023). Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Research, 24(388), 1-26.
> We understand your reasonable concerns about the complexity of FFT; however, FNOs still demonstrate superior speed compared to other neural operators. We will include a discussion on the limitations of FFT in the revised version.

FNO does not always demonstrate superior speed. How can you fairly compare the speed of FNO with that of DeepONet or UNets?

Moreover, since it performs convolution, i.e., pointwise multiplication in the spectral domain? The complexity of the number of parameters or FLOPs is just $O(n)$, while FFT takes $O(n\log n)$, where $n$ is the number of sensor points. I feel this is an important aspect to think about when you think of the complexity of the number of parameters.

> While our models require more training time than the standard FNO due to the use of MLPs or KANs, the difference is relatively minor, as shown in the table presented before(1.2s for AM-FNO (MLP) vs. 0.9s for FNO). In terms of inference time, we can precompute the kernel using the trained MLPs or KANs before inference, resulting in the same computational complexity as the FNO.

Ok, this makes sense to me. You should include this in the paper as part of the discussion.

> The MLP is different for every dimension. We present the result on 3D Plasticity benchmark, where we kept 101, 31, 20 modes for every dimension. As shown, our method outperform Geo-FNO.

I see. However, can you clarify why you chose Geo-FNO as a baseline? Geo-FNO tackles non-rectangular geometries. If your test data is rectangular, then why use Geo-FNO? If it's not, then how do FNO, let alone AM-FNO, even work?


Since some concerns and questions have been addresed, I have changed my rating.
Thank you very much for the valuable feedback and the improved score. We will thoroughly consider your suggestions and revise our paper accordingly.
I've raised my score to 6 since the authors have addressed all my major concerns.
We appreciate your valuable suggestions regarding the motivation for our study. We will revise the abstract and introduction to incorporate your feedback. We will also include empirical results and a discussion on frequency truncation in the revised version.

Thank you sincerely for the constructive discussion, which will help enhance our work. We will carefully consider the points mentioned and make revisions accordingly.
I see, and I'm now convinced that AM-FNO's design is valuable in terms of both accuracy and efficiency.

However, the authors should consider revising the abstract and introduction. Based on the available experiments, the major benefit of AM-FNO appears to come from the MLP parameterization, not from using full frequency modes. Yet, the current motivation in the abstract and introduction focuses primarily on the issue of high-frequency truncation in FNO. Since there's evidence that increasing the number of modes in FNO may actually damage its performance, using full frequency modes doesn't provide a strong enough motivation. I suggest focusing the motivation on the curse of dimensionality, highlighting AM-FNO is designed to reduce the number of parameters in FNO. This motivation holds whether truncation is used or not and is more consistent with the experimental results.

I also recommend adding a discussion and ablation study on AM-FNO's truncation frequency across other datasets in this paper, similar to the test on Darcy flow. This is crucial for helping readers understand why AM-FNO performs better. The authors should clarify that AM-FNO doesn't necessarily require full frequency modes, as experiments demonstrate that AM-FNO with high-frequency truncation is actually a very competitive method in both accuracy and efficiency.

I appreciate the authors' efforts to address my concerns. I would be happy to raise my score if the authors could revise the paper based on our discussion.
Thank you very much for the valuable feedback and improved score. We will carefully consider your suggestions and revise our paper accordingly.
We clarify that AM-FNOs use 85×43 modes, whereas FNO uses 2×12×12 modes. AM-FNO requires MLPs to generate the kernel and perform matrix-vector multiplication with a larger kernel. However, both operations can be parallelized for high efficiency.

We present the performance of AM-FNOs **trained with $24 \times 12$ modes** (requested by Reviewer Z566) on Darcy benchmark, as shown in the table below. Even with truncated modes, AM-FNOs still significantly outperform FNO, demonstrating the effectiveness of MLP parameterization. While maintaining all modes can further improve performance, the impact is relatively modest on this benchmark. 

| Modes  | AM-FNO(MLP)  | AM-FNO(KAN) |   FNO   |
|----------| ---------|---------| ------- |
| 24*12 | 4.72e-3 | 4.78e-3 | 1.08e-2 |
| 85*43 | 4.21e-3 | 4.28e-3 | - |

If you have any further questions, we are pleased to discuss them.
Thanks for addressing my concerns. It seems both MLP inductive bias and adding modes are boosting the performance of AM-FNO. After reviewing the global and each reviewer's question responses, I have decided to raise my score based on new experimental results and discussion. I have changed my score from 4 to 5 and hope to see the discussion and all the new results in a revised version of the paper.
So, is it correct that in each Fourier layer, AM-FNO processes 85 * 43 = 3,655 modes while FNO processes 23 * 12 = 276 modes? And yet, AM-FNO only takes about 1.33 times the training time and 1.5 times the memory to handle approximately 13 (3,655 / 276) times the modes? If that's true, I'd say AM-FNO's efficiency is impressive.

Could you also comment on why AM-FNO outperforms FNO? Is it mainly because (1) AM-FNO doesn't truncate high-frequency modes or (2) AM-FNO uses MLP for all modes? I believe you could gain insights by running a test with AM-FNO using the same truncation mode as FNO. If time doesn’t allow for this test, any comments based on your understanding would also be helpful.
The results were tested on the Darcy benchmark (with an $85 \times 85$ resolution) using a batch size of 16. The hyperparameters were consistent with those used in the main experiments presented in our paper (Table 2).

If you have any further questions, we are pleased to discuss them.
Thank you for providing these results. However, they don’t seem to align with the table you previously shared in the rebuttal or with Table 6 in your paper. Could you confirm which dataset is used for this test and what the data resolution is?
Thanks for your feedback.

1) We understand your reasonable concerns about the complexity of FFT; however, FNOs still demonstrate superior speed compared to other neural operators. We will include a discussion on the limitations of FFT in the revised version.

2) While our models require more training time than the standard FNO due to the use of MLPs or KANs, the difference is relatively minor, as shown in the table presented before(1.2s for AM-FNO (MLP) vs. 0.9s for FNO). In terms of inference time, we can precompute the kernel using the trained MLPs or KANs before inference, resulting in the same computational complexity as the FNO.

3) The MLP is different for every dimension. We present the result on 3D Plasticity benchmark, where we kept 101, 31, 20 modes for every dimension. As shown, our method outperform Geo-FNO.

| Benchmark  | AM-FNO(MLP)  | Geo-FNO |
| ---------- | -----------  | ------- |
| Plasticity | 3.04e-3      | 7.4e-3  |

4) Thanks for your suggestion. We will add the mentioned baselines in the revised version.

If you have any further questions, we are pleased to discuss them.
Thanks for your feedback.

Q2) We have reported the essential hyperparameters of our models in Section 5.1. We will provide a more detailed description in our revised version.

Q3) We present the performance of our models with 12 modes compared to FNO with the same number of modes on Darcy benchmark, as shown in the table below. The results indicate that our models continue to significantly outperform the FNO, demonstrating the effectiveness of our parameterization.

| Benchmark  | AM-FNO(MLP)  | AM-FNO(KAN) |   FNO   |
|----------| ---------|---------| ------- |
| Darcy      | 4.72e-3 | 4.78e-3 | 1.08e-2 |

Q) The results of AM-FNOs and FNO with one layer on Darcy are shown below.

| Benchmark  | AM-FNO(MLP)  | AM-FNO(KAN) |   FNO   |
|----------| ---------|---------| ------- |
| Darcy      | 1.85e-2 | 1.98e-2 | 2.17e-2 |

If you have any further questions, we are pleased to discuss them.
Thanks for your prompt response.

We understand your concern regarding GPU memory usage and training time. Below, we provide a comparison of GPU memory and training time between our models and baselines, including FNO with 12 modes, tested on the same benchmark. While our models require more training time and memory than the standard FNO due to the use of MLPs or KANs, the difference is relatively minor. We appreciate your suggestion and will include a more detailed discussion on the efficiency of our method.

| Model| AM-FNO (MLP)| AM-FNO (KAN) | FNO | U-FNO | OFormer | LSM | F-FNO |
|------|------|------|------|------|------|------|------|
|Train Time (s/epoch) |  1.2  | 2.1  | 0.9   |  1.6  | 16.5 |   2.1 |  1.1  |
|Memory (M) | 1850 |  2066  |  1212  | 1444 | 16090   |  1894  | 1126  |
|Params. (M) |  1.1  |  1.5  |  2.4  |  2.6  |  1.3  |  4.8  |  0.2 |

If you have any further questions, we are pleased to discuss them.
Thank you for the response.

Q2) It would be better if you could also report hyperparameters for the proposed method.

Q3) With increased modes, FNO is computationally heavy and might sometimes underperform. However, can you report the numbers of AM-FNO compared with FNO having the same modes? It will help to understand the proposed method better, whether increasing the modes is effective in AM-FNO or the way in which the kernel is parametrized (Kernel Bias)

Q) Can you report the numbers using just one layer of AM-FNO and FNO on specific benchmarks?

I am trying to understand the proposed method better from the point where kernel bias is more critical or increasing the modes. Also, it's fine if the proposed method is not competitive with FNO in terms of computational and time complexity, as the proposed method is trying to address different problems altogether.
Thank you very much for the valuable feedback. We will carefully consider the points discussed and revise our paper accordingly.
Thank you for your response. I understand that the rebuttal period is only one week, and it's not feasible to run many experiments or make significant revisions to the paper. Given the limited time for discussion, I'll focus on my major concern: **GPU memory and training time**.

Using the full frequency modes (421) for FNO in your comparison of GPU memory and training time is problematic. Previous studies often use a truncation mode of around 12 for FNO, for three main reasons: (1) Keeping too many modes increases computational costs considerably. (2) For PDE data such as Darcy flow, with minimal high-frequency components, using so many modes (421) is redundant. (3) Using too many modes can actually damage FNO's performance. 

Therefore, using FNO with a truncation mode of 421 is a very weak baseline in terms of accuracy, GPU memory usage, and time consumption. The current comparison appears to increase FNO's parameters to an unreasonably large number, then claims that AM-FNO uses fewer parameters and GPU memory than this excessively large FNO.

Compared to the commonly used FNO, AM-FNO's architecture is expected to demand more GPU memory and longer training time, which could significantly limit its application for high-dimensional PDE data. Additionally, AM-FNO cannot guarantee it has fewer parameters than the commonly used FNO.

I recommend that the authors provide a more detailed discussion on the efficiency of AM-FNO to avoid presenting potentially misleading results.
Thanks for your feedback.

1) Thanks for your suggestion. We agree that the concept of spectral bias provides valuable insight into our approach, and we will include additional discussion to elaborate on this in the revised version.

2) Sorry for not including this part. We will include the relevant descriptions of the benchmarks in the revised version.

3) In our main experiment (Table 2), we aimed to ensure that all models had a comparable number of parameters. However, the parameter count for FNO with all modes becomes excessively large, making it unfair to compare with other models. Meanwhile, one advantage of our method is that it can capture all modes with a limited number of parameters. We have presented the results of FNO with full modes on CFD-2D (denoted as "FNO+" in Table 3), where AM-FNOs outperform FNO+. We also provide comparisons with FNO+ on the Darcy and Airfoil benchmarks below. We will include additional empirical results on FNO+ and AM-FNOs with the same number of modes in the revised version.

| Model | AM-FNO (MLP) | AM-FNO (KAN) | FNO+ |
|------|------|------|------|
|Darcy| 4.21e-3 | 4.28e-3 | 1.33e-3 |
| Airfoil | 5.64e-3 | 6.06e-3 | 1.32e-2 |

4) The first two lines of code are for initialization (though they aren’t necessary), while the next two lines compute the basis functions. We want to clarify that our code is consistent with our method.

5) We provide a comparison of all models, using the hyperparameters from the main experiment, tested on the same benchmark in the table below. Our methods require more memory and training/inference time compared to other FNOs, due to the additional modes we maintain. However, AM-FNOs still outperform non-FNO neural operators.


| Model| AM-FNO (MLP)| AM-FNO (KAN) | FNO | U-FNO | OFormer | LSM | F-FNO |
|------|------|------|------|------|------|------|------|
|Train Time (s) |  1.2  | 2.1  | 0.9   |  1.6  | 16.5 |   2.1 |  1.1  |
|Inf Time (s) |  0.091  |  0.17  |  0.0068  |  0.087  |  1.5 |  0.15  |  0.075  |
|Memory (M) | 1850 |  2066  |  1212  | 1444 | 16090   |  1894  | 1126  |
|Params. (M) |  1.1  |  1.5  |  2.4  |  2.6  |  1.3  |  4.8  |  0.2 |

If you have any further questions, we are pleased to discuss them.
Thanks for your detailed feedback.

# Motivation

We hadn't initially considered spectral bias a significant reason for explaining our work. Thus we greatly appreciate your suggestion regarding spectral bias and concur that it provides a well-founded perspective for interpreting our method. We will expand the discussion on it in the revised version.

# AFNO

Due to the time constraints of the rebuttal phase, we have only presented results for AFNO on two benchmarks. In the revised version, we will provide a more comprehensive discussion along with additional empirical results pertaining to AFNO.

# GPU memory and training time

The models are all with 4 layers and 32 widths. As mentioned above, FNO retains all frequency modes ($421 \times 211$ due to central symmetry), while AM-FNO (MLP) utilizes 32 orthogonal basis functions, and AM-FNO (KAN) employs 32 local basis functions.

Regarding GPU memory usage, FNO requires significantly more memory to store model weights, gradients, and the optimizer, owing to its large parameter count. Although AM-FNOs necessitate memory to store the hidden state used to derive the kernel, the overall memory requirement remains lower.

In terms of training time, AM-FNOs indeed demand more computation compared to FNO. However, due to the large kernel size in FNO, a substantial amount of memory reads, writes, and gradient computations are required for parameter updates. Consequently, FNO demands more training time, when training with high-resolution and high-dimensional data.

We are grateful for your valuable feedback, which we believe will significantly enhance the quality of our paper. Should you have any further questions, please feel free to reach out.
Thank you for the response.
1) The paper references factorization techniques, but the implementation uses a Chebyshev basis to parametrize the entire Fourier kernel, along with an MLP. As reviewer DLAH noted, this approach is not clearly explained in the current version of the paper, where only smooth function parametrization is mentioned. It is essential to explicitly include this discussion in the paper for clarity and the discussion about the spectral bias in MLP.

2) I couldn't find hyperparameter details for the benchmark datasets in the paper, which is needed to ensure reproducibility.

3) If you are using all Fourier frequency modes in the proposed method, then it's not a fair comparison. I would like to see the performance compared with FNO, where we have used all the frequency modes. Also, it would be great if the author could provide the performance of the proposed method using only the same number of modes as used for FNO.

4) I could find two Tx and Ty in the code. Could you clarify:
 >     self.Tx = torch.zeros(self.n1, H+padding)
        self.Ty = torch.zeros(self.n2, (W+padding)//2+1)
        self.Tx = (torch.cos(self.grade1@torch.acos(self.gridx))).reshape(1, self.n1, H+padding, 1).cuda() 
        self.Ty = (torch.cos(self.grade2@torch.acos(self.gridy))).reshape(1, self.n2, 1, (W+padding)//2+1).cuda() 

5) Could you compare the proposed method with baselines regarding training time, GPU consumption, inference time, and number of parameters?

PS: I am open to discussing the paper and would like to consider increasing the score if my questions are addressed.
Thank you for your efforts to ensure fair comparisons by standardizing parameters across models and exploring full mode coverage are appreciated. The additional results on the 3D Plasticity benchmark demonstrate AM-FNO's scalability potential. Thanks also for including the computational efficiency comparison showing AM-FNO (MLP)'s advantages in memory usage and training time. I will not update my score however I hope that the revised version with more detailed discussions on these aspects will be included:) Thank you!
Overall, I appreaciate your efforts to address my concerns; I will be satisfied if the following points can be addressed/clarified:

>　Firstly, we apologize for the error in the complexity statement; it should be $O\left(d_h^2 k^D\right)$.

My main concern is that while your work aims to reduce the curse of dimensionality in FNO, but the complexity introduced by FFT might undermine these benefits from your method. 

> Our method primarily addresses memory issues, particularly in high-resolution or high-dimensional contexts.

Does your method address the time complexity issue in these scenarios? From what I understand, your approach might actually increase the running or inference time. The primary reason for using neural operators is to achieve fast inference; otherwise, numerical schemes with coarse discretization can offer even better performance with theoretical guarantees.

> Q1

So, are the MLPs the same for every dimension? What happens if the axes aren't on the same scale? For instance, in a 2D domain of $[0, 1000] \times [0, 1]$, where you want to keep 100 modes for the first dimension and only 10 for the second, can your method still work given that MLPs are fixed in size? If not, it seems your method can only be applied when you sample the same number of frequency modes (i.e. $k$) in every dimension, which doesn’t seem practical to me.

For some test data, e.g., from the FNO paper, you can effectively do this. But not pratical in general. 

> However, our method still outperforms non-FNO neural operators.

There are many non-FNO neural operators, e.g., DeepONet, U-Net, SNO [1], the selected baselines such as Oformer and LSM is not what I would have expected to see as a baseline in this venue.

[1] Spectral Neural Operators, V. Fanaskov, I. Oseledets
Thank you very much for the detailed feedback and the improved score. We will carefully consider the points discussed and revise our paper accordingly.
Thank you for the clarification. Now everything becomes clear for me. 
Although the paper should be revised with the above information before publication, I believe the author continue their hard work to polish it. So, I am willing to raise the score. 
Thank you for the responses.
Thank you for your response.

For the **motivation (W1)**, I can see your thoughts on why MLP without the orthogonal basis is not good enough. However, the current motivation remains ambiguous. The statement that 'MLPs struggle to capture the complex, non-linear nature of the kernel function' appears speculative. It would be more convincing if you could provide empirical evidence or a formal argument to support this statement.

This is why I mentioned the **spectral bias (Q1) of MLPs**, a well-known property for MLPs with both empirical and theoretical evidence. Following this concept, a function's complexity and non-linearity are linked to its high-frequency components: the more complex and non-linear the function, the more high-frequency components it has. Then, using the **Fourier basis (W2)** as embedded functions to improve the high-frequency property of MLPs can be well supported by the literature. 

I asked about **smoothness (Q1)** because it describes a function's spectral properties—smoother functions have fewer high-frequency components. In your paper, you aimed to make the kernel function smoother (with fewer high-frequency components) but then added an orthogonal basis to the MLP to represent more complex, non-linear functions (with more high-frequency components). This seems contradictory to me. 

I'm curious why you haven't discussed the spectral bias of MLPs in your paper, given its significance. As I mentioned, previous research on spectral bias and the Fourier basis provides a strong explanation for why an orthogonal basis helps MLPs represent more complex functions. 

For **AFNO (W3)**, it would be beneficial to present its results across all datasets used in this paper (Table 2). Additionally, a more thorough comparison of AM-FNO and AFNO would be helpful, such as including ablation studies or an efficiency analysis of both methods. If AFNO is just another unrelated neural operator, it's fine to simply guess why it performs worse than AM-FNO. However, given the similar motivation and implementation between AFNO and AM-FNO, a deeper analysis is necessary.

Regarding **GPU memory and training time (Q3)**, could you please provide more detailed hyperparameters for the models? I’m having trouble understanding why AM-FNO requires less GPU memory and training time compared to FNO. As I understand it, AM-FNO uses MLP to reduce the number of parameters, but this should increase computation. Whether you use shared or different weights for different modes, each Fourier layer still needs to map k modes from input to output. Given that FNO truncates high-frequency modes, it seems FNO should require less computation than AM-FNO.
Thanks for your response.

# W2:

Sorry for the misunderstanding. Our response above intended to clarify that we have used the final epoch performance in Table 2. Therefore, we believe that Table 2 already meets your requirement to fairly assess the effectiveness of all models.

If you have any further questions, feel free to let us know.
Thank you for your reply. 

W2: 

Please report a part of the revised Table 2 (performance at the final epoch, multi-dimensional PDEs are preferable), to assess the effectiveness of Amortized FNO in comparison to the other models again. I expect that it does not need so much effort but only checking log files...
Thanks for your response again.

**W2:**

Thank you for your explanation. We reported the test score evaluated **at the last epoch** for all the models.

**W3:**

In our revision, we assume that $\mathcal{F}$ is a separable Hilbert space. According to Theorem 9 in [1], there exists a complete orthonormal system (orthogonal basis), which can approximate any arbitrary function $f$ within the space, as stated in Definition 11.9 and Theorem 11.13 in [2]. We will consider introducing it directly within the text in the revised version. Thank you for your suggestion.

If you have any further questions, feel free to let us know.

[1] https://www.math.nagoya-u.ac.jp/~richard/teaching/s2023/SML_Tue_Tai_2.pdf

[2] https://www.math.uni-hamburg.de/home/gunesch/calc1/chapter11.pdf
Thank you for your further explanation. However, I still need a little more information to evaluate properly. 

W2: 

Actually, I do not need an information of other papers but my question is very simple: 

"Is the reported test score evaluated at the last epoch's model weight or the best one during the training?"

If it is the former, no problem, though I recommend to use validation set to evaluate the model's generalization ability more straight-forwardly; 
If the latter, the reported result is overfitted to the test set and all the results are recommended to be re-evaluated (though it is no problem if the best performance is obtained at the last epoch for all the experiment...). 

As far as I checked the source code (naca.py / pipe.py), the test set evaluation is performed every epoch (which is not recommended on test set but only for validation set). And I cannot judge at which epoch's result is reported in the paper. 

If by accident the latter is used, then simulating early stopping from log file is necessary to be done (pick up result, e.g., at 100th epoch) where the test score is still decreasing as epochs, as a fairer comparison. 

W3: 

Thank you for your clarification. Can you please tell us how to revise it? 
If the authors are still going to keep it as a theorem (with citation), the conditions should be mathematically rigorous, e.g., I'm still doubt if it is possible to prove the Theorem 1 with "arbitrary function" f in Hilbert space, which I'm not so sure if only including at least piecewise continuous function (are very mathematically exotic functions excluded?). 
(I personally do not think to claim it as "theorem" because MLP function is anyway continuous function, in which case it is obvious that the function can be expanded by a set of complete orthogonal basis).
Thanks for your prompt and detailed response!

**W2:**

Our paper follows the same evaluation methodology as the original FNO paper: using fixed training setting and training for a fixed number of epochs without early stopping. To ensure a fair comparison with other FNOs, we standardize the choice of model hyperparameters (such as the number of layers and channels) by using common settings for all FNOs including AM-FNOs. In summary, we did not use the training set to train or determine the hyperparameters. We believe that the risk of overfitting to the test set is minimal, and thus the results presented offer a fair comparison.

**W3:**

Sorry for the mistakes. Theorem 1 in the paper presents an alternative formulation of Theorem 11.13 in [4] from the perspective of the norm of the approximation error. The space $\mathcal{F}$ is a Hilbert space, and $f$ is any function in $\mathcal{F}$. We will revise the theorem and include the appropriate reference in the updated version. Thank you for pointing this out.

If you have any further questions, we are pleased to discuss them.
Thanks for your prompt and detailed response!

**W1:**

We appreciate your feedback and would like to address your concerns subsequently.

Firstly, we apologize for the error in the complexity statement; it should be $O(d_h^2k^{D})$.

Secondly, there have been attempts to utilize large pre-trained FNO models. For instance, DPOT [1] explores a model that employs a shared MLP to transform each frequency mode of the input (akin to a convnet with a $1 \times 1$ kernel) to mitigate the memory overhead of kernels. This architecture is similar to AFNO [2], which we have compared in Table B of the global rebuttal. Our results indicate that our method outperforms AFNO on both benchmarks, which we attribute to AFNO’s limited expressiveness due to its uniform transformation across different frequency modes, while our approach treats each mode differently.

Regarding your concerns about FFT, we acknowledge that FFT suffers from the Curse of Dimensionality (CoD). However, FNOs still demonstrate superior training speed (as shown in the table below in Q3) and prediction accuracy compared to other neural operators. Our method primarily addresses memory issues, particularly in high-resolution or high-dimensional contexts, as demonstrated in the table in the rebuttal before. Our approach shows a reduction in memory usage (and shorter training time for AM-FNO (MLP)) even in 2D benchmarks, suggesting potential benefits for large-scale models.

**Q1:**

Continuing with the assumption of one channel and a one-dimensional kernel size of $k$, let’s further simplify the MLP to a linear layer for clarity. To map frequency modes (with shape $[k,1]$ ) to the corresponding kernel values (also with shape $[k,1]$ ), our method first applies basis functions to obtain an embedding with shape $[k,m]$ , where m is the number of basis functions. We then compute the kernel values (real or imaginary part) using a linear layer with $m \times 1$ parameters. Consequently, the complexity of our method depends only on the number of channels and the number of basis functions (specifically $O(D m d_h^2)$ in the linear case compared to $O(d_h^2k^{D})$ in FNO for multi-dimensional PDEs), and avoid becoming excessively large with increasing dimensions and resolution.

**Q3:**

To address your concern, we provide a comparison of training times under our experimental settings for the same benchmark, as shown in the table below. The results indicate that our method has a slower training speed compared to other FNOs, which can be attributed to the additional frequency modes we adopted. However, our method still outperforms non-FNO neural operators. Furthermore, as shown in the previous rebuttal, when all modes are retained, our method can surpass FNO in training time.

| |AM-FNO(MLP)|AM-FNO(KAN)|FNO|U-FNO|Oformer|LSM|
|-----|-----|-----|-----|-----|-----|-----|
|Train Time (s/epoch)| 1.2 | 2.1 | 0.9 | 1.6 | 16.5 | 2.1 |

If you have any further questions, we are pleased to discuss them.

[1] Hao, Z., Su, C., Liu, S., Berner, J., Ying, C., Su, H., ... & Zhu, J. (2024). Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training. arXiv preprint arXiv:2403.03542.

[2] Guibas, J., Mardani, M., Li, Z., Tao, A., Anandkumar, A., & Catanzaro, B. (2021). Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587.
Thank you for the authors' addressing to my comments and questions. 
Other than the following two points, I've been satisfied. 

W2:  Without validation dataset, how was the test set performance measured? Early stopping? Otherwise, I slightly suspect that the result would be over-fitted to the test set, which is crucial as an ML conference paper. Please carefully validate that your results are not over-fitted to the test set, with logical explanations. 

Q4: Thank you for your explanation. Then, reading [4], I wonder what is the difference of the paper's Theorem 1 from Theorem 11.13 in [4]. Besides, is either the function space F not Hilbert space or the arbitrary function f is not at least piecewise continuous? I wonder if Theorem 1 in the paper is truly new finding or citing very classical result of Hilbert space
(if f is well-behaved and F is Hilbert space, any well-behaved (at least piecewise continuous) function can be expanded by an Hermitian operator's eigenfunction, which in general form a complete orthogonal basis. This is also known for more than 100 years). 
Note that if functional space F is not Hilbert space (complete unitary space), I think the paper's theorem 1 is trying to show any orthogonal set can be complete even though in non-Hilbert space, which could be mathematically strange. 

If Theorem 1 is a really new finding, it is great. Otherwise, it just damages the reputation of the NeurIPS paper quality, in particular, from mathematics community. Please carefully specify the new point and newly reduced restriction. Although I'm not a professional researcher of Hilbert space, I can also ask for a help to my Hilbert space researcher friend.
### W1
> This severely limits practical applications, such as large-scale pre-trained models.

First of all, how do you get $\left.O\left(\left(d_h^2 k\right)^D\right)\right)$? Isn't it just $\left.O\left(d_h^2 k^D\right)\right.$

Can you provide a case in which they use large pre-trained FNOs?

Moreover, in your case, FFT would take $O(d_h n \log n)$, where $n \geq k^D$. FFT itself is already very expensive and suffers from the CoD issue.

### Q1
Can you be more specific, such as providing some mathematical expressions? I'm still confused.

### Q3
I do understand that the rebuttal period is only one week, and you might not have time to do this.  However, I believe that running times are important. A simple example is that a U-Net or DeepONet with a similar number of parameters might be much faster than FNO. If you run their models and record the time on the same machine and with the same settings, I believe you can ensure the comparison is consistent and fair.