Dynamic Schwartz-Fourier Neural Operator for Enhanced Expressive Power
Authors: Wenhan Gao
Jian Luo
Ruichen Xu
Yi Liu
Venue: Accepted by TMLR ()

Abstract: Recently, neural operators have emerged as a prevailing approach for learning discretization-invariant mappings between function spaces. A particular example is the Fourier Neural Operator (FNO), which constrains integral kernels to be convolutions and learns the kernel directly in the frequency domain. Due to the capacity of Fourier transforms to effectively reduce the dimensionality and preserve information, FNOs demonstrate superior performance in terms of both efficiency and accuracy. In FNOs, the convolution kernel is fixed as a point-wise multiplication in the frequency domain; however, these translation-invariant kernels might limit the expressiveness of FNOs. For instance, if the underlying system lacks translational symmetries, the kernels learned by the FNO will still exhibit translational invariance, thereby limiting the model's expressive power. We propose a dynamic Schwartz operator that induces interactions between modes to enhance the expressiveness of FNOs. In this work, we introduce a novel approach that equips FNOs with Schwartz operators to learn dynamic kernels, termed Dynamic Kernel Fourier Neural Operators (DSFNOs). By incorporating this dynamic mechanism, our model gains the ability to capture relevant frequency information patterns, facilitating a better understanding and representation of complex physical phenomena. Through experiments, we demonstrate that DSFNOs can improve FNOs on a range of tasks, highlighting the effectiveness of our proposed approach. The code is available at https://github.com/wenhangao21/TMLR25_DSFNO.

Published in Transactions on Machine Learning Research (05/2025)
Dynamic Schwartz-Fourier Neural Operator for Enhanced Expressive Power
Wenhan Gao
wenban.gno@@stongbooo.cco edu
Stony Brook University
Jian Luo Stony Brook Universiby
jan..nao@stongbrooo..tt
Ruichen Xu
rulchen.mu@stonnnrooo.ccm
Stony Brook University
Yi Liu Stony Brook University
yi.liu..4@tonnyyyn edu
Reviewed on OpenReview:
https: // openreview. net/ forum? id- B0E2yjrND8
Abstract
Recently, neural operators have emerged as a prevailing approach for learning discretization- invariant mappings between function spaces. A particular example is the Fourier Neural Operator (FNO), which constrains integral kernels to be convolutions and learns the kernel directly in the frequency domain. Due to the capacity of Fourier transforms to effectively reduce the dimensionality and preserve information, FNOs demonstrate superior perfor- mance in terms of both efficiency and accuracy. In FNOs, the convolution kernel fixed as a point-wise multiplication in the frequency domain; however, these translation--invaiaan kernels might limit the expressiveness of FNOs. For instance, if the underlying system lacks translational symmetries, the kernels learned by the FNO will still exhibit translational in- variance, thereby limiting the model's expressive power. We propose dynamic Schwartz operator that induces interactions between modes to enhance the expressiveness of FNOs. In this work, we introduce a novel approach that equips FNOs with Schwartz operators to learn dynamic kornels, termed Dynamic Kernel Fourier Ncural Operators (DSFNOs). By incorporating this dynamic mechanism, our model gains the ability to capture relevant frequency information patterns, facilitating better understanding and representation of complex physical phenomena. Through experiments, demonstrate that DSFNOs improve FNOs range of tasks, highlighting the effectiveness of our proposed approach. The code is available at httpp:///gt thub. com//enhaagga22//MMMR5S DSFNO
1 Introduction
In various areas of science and engineering, researchers seek to explore how physical systems behave under varying conditions, such as different initial conditions, boundary values, or forcing functions. Traditional numerical methods, such as finite difference, finite volume, finite element, and spectral methods, frequently require excessive time for simulating physical systems with varying parameters. A type of data-driven surrogate model, termed neural operators (NOs), serves Zhang et al.) 2025). An operator learner aims to learn a mapping between the parameter function space and the solution function space. A well-trained neural operator can evaluate different parameter functions, such as different initial conditions, without requiring re-training. Evaluation to a parameter function only nccessitates a forward pass of the network, which can be several orders of magnitude faster than traditional mumerical methods (Li al. 2021). Pioneering works of neural operators include, but are not limited to, DeepONet (Lu et. al.] 2021), which is built upon the universal approximation theorem of operators;
efficient alternative (Kovachki et al. 2023

---

Published in Transactions on Machine Learning Research (05/2025)
(a)
(b)
(c)
Figure 1: Learned filters for FNO and DSFNO on the Darey flow equation: (a) Learned DSFNO filters for input function a1; (b) Learned DSFNO filters for input function 02; (c) Learned FNO filters which are fixed for all functions. The 32 input channel filters in the first kernel integral layer are presented. It can be observed that DSFNO filters can focus on different local details based on different input functions, with bright and dark colors being more distinct and clearly separable. In contrast, since FNO filters are fixed for the entire dataset, they tend to be more global for each filter and do not capture the distinct local details for different input functions as effectively. Additional visualization provided in Appendix A
GNO (Anandkumar et al. 2019). which parametrizes integral operators with message passing; and Model Reduction (Bhattacharya et al. p021, which employs model reduction (PCA) to learn mappings ween infinite-dimensiooall spaces. Another prominent and influential example of neural operators is the Fourier Neural Operator (FNO) (Li al. 2023), which model the solution operator as a composition of kernel integral operators and non-linearities and impose the integral kernel operator as a convolution operator. FNO uses the Convolution Theorem to learn the kernel directly in the frequency domain. Owing to the capacity of spectral analysis to capture information in iiseretizationninaarii demonstrated its capacity to serve as an efficient and accurate surrogate model for physical modeling and
2021) and its variants (Rahman et al. 2023 Tran et al. 2023 Bonev et al.,
manner while reducing the dimensionality of the problem, FNO has
has been applicd to various fields (Pathak et al. 2022: Chen et al. 2023 Jiang al. 2021
Kernel integral operators, including FNO, are molivated by the Green's function or fundamental solution of PDEs (Kovachki et al.] (003).. The Green's 's function kernel used to express solutions of PDEs can depend
on the input parameter function (Evans 2010); under fairly general conditions, we can express the solution, u(x) linear PDE as:
u(x) / Ga( (z))))
where the integral kernel G. can be dependent on the parametric function a(x). In other words, if the parameter function changes, the integral kernel changes as well. In contrast, in the formulation of integral neural operators, once trained, the kernel is usually fixed for all input functions. It motivates us to study dynamic kernels that adapt to the input parameter functions for integral neural operators. Moreover, FNO additionally constrains the kernel to be a convolution kernel to effectively harness the power of Fast Fourier transforms model the kernel integral; however, this constraint further limits the learning capacity of the model. A simple example is when the underlying system lacks translational symmetries; in such cases, the kernels learned by FNO will still exhibit translational invariance, limiting its expressive power. To address these challenges, we propose Dynamic Schwartz-Fourier Neural Operator (DSFNO) that utilizes dynamic Schwartz operators to learn convolution kernels to augment the expressive power of FNOs. Examples of learned dynamic kernels (Fourier filters) are provided in Figure [ (a) and (b) are the learned kernels for different input functions that change as the input changes; (c) is the learned fixed FNO kernel. In summary, our contributions can be summarized as follows: 1) We highlight a critical yet often overlooked issue concerning the limited expressive power of fixed convolution kernels; 2) We propose novel general framework to learn dynamic kernels to capture more diverse and spatially-dependerr dynamics; 3) We propose Schwartz operators that represent general lincar operators between function spaces while facilitating dimensionality reduction for efficiency.
2 Related Work
In recent years, neural PDE solvers have gained much attention as promising alternative to traditional numerical methods for tackling PDE challenges in a wide array of practical engineering and life science
2

---

Published in Transactions on Machine Learning Research (05/2025)
applications, including climate modeling (Pathak et al. 2022), fluid dynamics Zhang et al. 2022 Sirig-
nano et al. (020)) material science (Kumar & Echekki 2024], elc.. Traditionally, the process of solving a PDE entails finding a smooth function that satisfies the derivative constraints presented by the equations; many traditional numerical solvers are developed based on this view, such as the Finite Difference Method
QQarteroni & Valli 1994). From this viewpoint, Pllsics-Informed Neural Networks (PINNs) (Raissi el al. 2019 Li et al.J 2022; Gao & Wang 2023 Wang et al., 2021) have emerged approximate solutions of PDEs. However, PINNs approximate PDE solutions on an instance-by bea-ance-b--intance basis; each time the parameter func- tion changes, PINNs need to be retrained. Another perspective from modern functional analysis is to view differential operators as mappings between function spaces, then the inverses of these mappings naturally serve as the solution operators. Expanding on this perspective, neural operators have been introduced as a technique for addressing families of parametric PDEs (Li et al., 2021 Lu el al. 2021 Kovachki el al.) 20233 Brandstetter et al. 2022 Raonic et al. 2023 Bhattacharya et al. 2021 Gao et al. 2024). After training, inference on a new instance of the parameter function only requires a forward pass of the network, which can be notably faster, oflen by several orders of magnitude, compared to traditional numerical methods.
Among neural operators, Fourier neural operators have garnered considerable attention and popularity. FNO is a typc of iterative kernel integral solver, termed as integral neural operators, and the kernel integral is imposed to be a convolution (Kovachki et al. 2023). Global convolution takes place through Fourier layers in the frequency domain; by utilizing low-frequency modes and truncating high-frequency modes, FNO captures global information while incurring low computational costs. Since FNO serves as the backbone in many models designed for specific applications (Pathak et al., 2022: Chen al. 2023; Jiang al., 2021;
Bonev et al. 2023), it is imperative to enhance FNO itself. Lately, several studies have focused enhancing
the learning capabilities of FNO. Zhao et al. (023 Liu-Schiaffni et al. (0024) enable FNO to incorporate
local receptive fields, allowing for the retention of high-frequency local details. Liu et al. (2023a) expands the application of FNO to non-rectangular domains. Liu et al. (2023b) presents a comprehensive framework for empowering FNO to effectively handle non-uniformly distributed data. Xiao et al. (2024b) amortizes the parameters FNO to accommodate arbitrarily many frequency modes to allow FNO to capture high-
frequency details without having to have intractably many network parameters.
3 Background
3.1
Operator Learning
Operator learning refers to the use of neural networks to approximate an operator between infinite di- mensional function spaces (Kovachki et al. 2023). Neural operators hold significant relevance in scientific computing, especially for solving PDEs. Concretely, consider a parametric PDE of the form:
V(a,u)(n) = f(z), u(x) = 0, I E
I xEΩ
(2)
where Ω C <dd is a bounded open set referred to as the domain of the PDE, A is a, possibly non-linear, differential operator, a is the parametric input function, f is a given fixed function in an appropriate function space determined by the structure of N, and u is the PDE solution we are interested in. The PDE solution operator is defined G(a) u. We are interested in learning the operator G(-) through a given finite collection of obsorvations of input-output pairs {az, uz ) where each aj and uy are functions provided in a discretized form.
3.2 Fourier Neural Operators
Let A' in equation 2 be a linear differential operator, under fairly general conditions on N, the solution operator can be defined as the following kernel integral operator:
u(x) G((a= KK( (y)))d
(3)
where K₂ the kernel function defined on f2 x Ω and depends on the parameter function a. Drawing inspiration from the kernel integral method for the solution operator of PDEs, an integral neural operator comprises multiple layers of kernel integral operations with nonlinear activation functions to learn the solution
3

---

Published in Transactions on Machine Learning Research (05/2025)
operator of diverse PDEs. Each layer is characterized by a fixed non-linearity and fixed kernel integral operator K parameterized by network parameters, defined as
(Ko)(z) ()
(4)
where e(z) is the imput function to this kernel integral layer. FNO additionally enforees translation invariance the kernel, expressed as <(2r,) v) x(x- which is a choice from the perspective of fundamental solutions. Hence, the kernel integral operator transforms into convolution operator. Each Fourier layer in FNO is now characterized by fixed non-lincarity and a fixed convolution operator K. paramcterized by network parameters:
(KC))z) = S x(2 (((x-y)o(ydde
(5)
Convolution can be efficiently computed as element-wise multiplication in the frequency:
(KC)(z)= (Fx
(6)
where F and F-1 are the Fourier transform and its inverse, respectively. FNO directly learns Fx in the Fourier domain as complex weights instead of learning the kernel in the time space.
4 Dynamic Schwartz Fourier Neural Operator
This section introduces DSFNOs, a dynamic kernel adaptation of FNOs. DSFNOs feature two major mod- ifications compared to FNOs: Firstly, the kernel is designed to be dynamic. Secondly, Schwartz operators are proposed to facilitate interactions between different frequency modes. By incorporating these two major components, our model gains the ability discern relevant frequency information patterns, facilitating a better understanding and representation of complex physical phenomena. We start this section by introduc- ing the Neural Kernel Generator (NKG) and the overall DSFNO framework in Section 4.1 and then we will define frequency domain interactions (FDIs), provide arguments for the need of FDIs, and discuss translation symmetry related to FDIs in Section 4.2 Lastly, based on the need for FDIs, we provide the formulation of Schwartz operators for generating dynamic kernels in Section 4.3
4.1 Neural Kernel Generator
Kernel integral methods involve expressing the solution of a PDE a convolution or integral involving kernel function that represents the influence of one point in space on another. For example, in the case of elliptic equations, the solution of a linear, homogeneous PDE with given forcing term can be expressed as a convolution of the Green's function with the forcing term The readers are referred to Evans (2010) for more details on the kernel method for PDEs. In the kernel integral of the kernel method, namely equation B the kernel Ka can depend on the parameler function a; however, in the kernel integral operator in FNO, namely equation [] the kernel neural operator in FNOs and Green's 's function method is just motivation for the design of FNOs. However, this still motivates us to think that dynamic kernel, that adjusts as the input changes, can enhance FNOs' expressiveness.
fixed. This can be justified as the parameter function will be the input to the
The core of DSFNO lies in the idea of utilizing dynamic kernels. In traditional convolution, either in FNOs or CNNs, static kernels are employed, where the weights remain constant throughout the operation once trained. However, dynamic kernels introduce a novel approach by leveraging a Neural Kernel Generator (NKG) dynamically compute convolution kernels based on the input function: Ka = k(a), where is the imput function and is an operator that transforms the input into convolution kernel function. We illustrate FNO layer with dynamic kernel in Figure 2 In practice, the Neural Kernel Generator is transformation mapping truncated Fourier cocfficients of the input function to a band-limited output convolution kernel function. Such transformations can be as simple as a linear layer or as complex as a neural network such as the Spectral Neural Operator (Fanaskov & Oseledets) 2022b). It should be noted that NKG does not operate on infiniteddimensional spaces since we truncate the Fourier modes; however, NKGs are also insoution-invariann as they work with coefficients with respect to function basis. Precisely, in Fourier layer, the spectral convolution is now modified follows:
4

---

Published in Transactions on Machine Learning Research (05/2025)
(Kav) - <(a)(r s)(())d
(7)
which can also be efficiently computed as clement-wise multiplication in the Fourier domain directly:
([av)())=FF(())) (F)(())FFo({)).
(8)
Point-wise Multiplication
Spectral Convolution
F
(x()
F-1
Bias Term :Activation
axx)
WMA
NKG that takes the frequencics of the input and producc a convolution kemel
Fourier Transform and its inverse
W
Figure 2: A Fourier layer with Neural Kernel Generator (NKG): The input function a(x) first goes through Fourier transform to the frequency domain. Next, an NKG will take the frequencies of the input to generate a convolution kernel. Then, the input will be convoluted with the kernel through point-wise multiplication in the spectral domain. Subsequently, the result from the convolution will be inverse transformed back to the physical domain to include the bias term and undergo non-linear activation functions.
Additionally, the NKG can directly produce the coefficients of the kernel function in the Fourier domain, represented Fx(a), instead of generating a function in the physical domain. Consequently, the output of the NKG can be seamlessly convoluted with the input function through element wise multiplication in the Fourier domain, making the overall process highly efficient.
4.2 Frequency Domain Interaction through Neural Kernel Generator
Frequency Domain Interaction (FDI) refers to the way Fourier modes interact with each other. In FNO, in the spectral domain, there is no interaction between different Fourier modes since the resulting coefficients are obtained through point-wise multiplication with static kernels. In NKG, we allow connections between different modes to promote such interactions. We formally define frequency domain interactions as follows: Definition 4.1. Suppose H. is separable Hilbert space with orthonormal basis {ee]ã=o We say that T does not induce frequency domain interaction if for function v E H. there erist fized mappings (ke R R+ _ with E [ < co such that (Tv, ex) = ke((v, ex)) for all E.
In terms of FNO, the orthonormal basis in Definition 4.1 is the (gencralized) Fourier basis. Essentially, FDI means that the update of one Fourier coefficient is dependent on the other Fourier coefficients. If we treat the truncated Fourier coefficients, denoted as V, as input to a linear layer MV with a complex square matrix M, then there is no frequency domain interaction if and only if M is diagonal. If we consider spectral convolution, i.., point-wise multiplication in the spectral domain which can be thought of as M being a square diagonal matrix, the update of one Fourier coefficient is not dependent on others. Without activation functions, an FNO layer consists of spectral convolution and possibly residual conncction; it is clear that such operations do not induce FDIs. An FNO layer includes activation functions to learn nonlinear behaviors and to recover the high frequency modes (Kovachki et al. 2023), , which truncated in the spectral convolution. We show that the nonlinear activation functions will also induce FDIs in order to learn nonlinear, diverse, and complex behaviors of physical systems.
Theorem 4.2. Let v(x) be function in a separable Hilbert space such that v(x) admits Fourier transform, and let (x)) be a non-linear function. Assume that (xx) is analytic with respect x. Then, applying the non- linear activation function pointwise to u(z), denoted mathematically as the function composition a v(x), can induce interactions omong all frequency modes.
5

---

Published in Transactions on Machine Learning Research (05/2025)
In pseudo-spectral methods, frequency domain interac- tions (FD[s) are common in the resulting linear systems (Gottlieb & Orszag, 1977). As demonstrated in The- orem 4.2 (proof in Appendix B), nonlinear activa- tion functions in FNO also induce FDIs. However, these FDIs are restricted to preserve translation equivariance, failing to allow the network to cap- ture spatially varying dynamics effectively. This restriction inspired us to incorporate unrestricted FDIs in the spectral domain, enhancing the ability of neural op- erators to learn complex and evolving behaviors of para- metric PDEs. As illustrated in Figure 3 every Fourier mode is influencod by all others, supporting more intri- cale inter-frequency dynamics.
C
1
C1
2
2
C
cm
CK
CK
Cn
(a) NKG Interaction
(b) FNO Interaction
Figure 3: (a) The Neural Kernel Generator is not restricted to point-wisc multiplication, allow- ing connections between different. Fourier modes. The expressive power is enhanced through FDI, (b) An FNO Layer is based on global convolution, restricting the operations to point-wise multipli- cation in the frequency domain. Spectral convo- lution will not induce FDI in the spectral domain.
Discussion on Translation Equivariance. As FNO relies on convolution, it inherently maintains translational equivariance. However, within the DSFNO framework, where connections between different frequency modes are allowed in the neural kernel generator, this translational equivariance is no longer guaranteed. In Appendix C we elaborate further on the basic concepts of translational equivariance and spectral convolutions. FNO can address
this by employing padding or positional encoding (including grid coordinates additional input channels) to remove incorrect translation symmetries, but restricting the integral operator solely to convolution integrals in FNO still limits the model's expressiveness. Through NKG, frequencies are naturally integrated; this inte- gration removes unwanted translational symmetries in the model while also enhancing the learning capacities of FNOs. We provide a detailed justification for why FDIs in NKG can avoid the rranslational symmet in Appendix C.2 As illustrated in Figure imposing convolutions in general can limit the expressiveness of the model; even for operators with translational symmetries, it might still restrict the space of operators can learn. This mightt not be the case theorelically for FNO because FNO is composilion of linear inlegral operators and non-linear activation functions, and a translationally equivariant linear operator can always be written as concolution-like integral, as simple extension of Theorem 3.1 in Cohen et al. (2019). However, practically, might still limit the expressiveness of FNO. will be of interest to theoretically investigate such aspects and explore how to design equivariant models for general neural operators, such as those Fanaskov & Oseledets (202bb) and Lu et al. (2021)
4.3 Efficient Schwartz Kernel Operator
As illustrated in Section 4.2 we aim to incorporate FDIs into NKGs. We can implement NKGs simple spectral neural operator (SNO) Fanaskov & Oseledets 2022a]: we take truncated Fourier coeffi- cients as input and produce the output coefficients with a fully connected neural network. Obviously, will induce
All Continuous Operators
Equivariant ContinuDus Operators
Convolution based Operators
Deprsatlr howe
as the network is fully connected, FDIs. However, as we have multiple Fourier lay- ers, the computational cost of using SNOs as NKGs be prohibitive. Instead, we effectively im- plement NKG simply as lincar operator. To for- malize our approach, we introduce the Schwartz op- erator in the frequency domain, named in reference the Schwartz kernel theorem, which asserts that any linear operator can be represented as an integral over kernel function. With this founda- tion, present. Theorem 4.3 whose proof be found in Appendix B. demonstrating that any integral
Figure 4: The paradigm of different neural operators. For example, DeepONet and SNO learn a general con- timuous operator while FNO learns an operator with translation equivariance.

---

Published in Transactions on Machine Learning Research (05/2025)
operator can be expressed as a frequency domain interaction. Essentially, the Schwartz kernel theorem allows us to represent any linear operator mapping between two functions as interactions in the frequency domain, given fairly general assumptions. As general integral kernels, Schwartz kernels are more expressive than convolution kernels in FNO, which is a special case of integral kernels.
Theorem 4.3. An integral operator T Ha (R2, + Ha (R2, of the form (Tv)(z) = SR. (a, y)o(y)dy, given that the kernel K is smooth or sufficiently regulor to ensure that its Fourier trans- form is bell-behaved, can be expressed as
(To)()) = (Lr
(9)
where ò (E') is the Fourier transform of v(z), and M (5,5") is complex function describing how the operator modulates the input function in the frequency domain.
In this formulation, the Fourier transform of the output becomes linear transformation of 6(x) governed by M(G.<"). As a result, the coefficient value of any Fourier mode of the output is a weighted sum in the complex domain of all the Fourier modes of the input, and clearly, FDIs are induced through this process. In the case of the discrete Fourier transform, this corresponds without the need impose a diagonal constraint on M. Furthermore, similar the FNO, we can directly learn this complex matrix M in the Fourier domain to promote efficiency. Although direct evaluation of this integral in practice involves O(n²) complexity, where is the size of the discrelization of the functions. This complexity can be prohibitive, especially when compared to the more efficient On log n) complexity of the FNO. However, a significant advantage of this Fourier space formulation lies in the global and compressive properties of Fourier transforms. Similar to the truncation used in FNO, we can restrict the interaction to a subset of low-frequency modes, yielding a complexity of 0(1²), where k is the number of truncated Fourier modes. By doing so, we can still ensure global interactions in the physical space, effectively enhancing imput interactions for more expressive kernel. To further reduce computational complexity, albeit with some compromise in expressiveness, an efficient implementation can be found in Appendix D
the MV formulation discussed carlier,
5 Experiments
In this section, we investigate whether DSFNO enhances the expressive power of FNO in terms of modeling complex dynamics more effectively. Different neural operators exhibit different advantages and demonstrate superior performance across different examples of PDEs; even within the same PDE, the performance influenced by the distribution of the data such as how the initial conditions are generated. While one neural operator may excel, being accurate and effective with a significant margin on one dataset, it might fail to perform well on another. There is no single state-of-the-art neural operator in the domain of neural PDE solvers. Given our primary objective to enhance FNO, we conduct experiments on two 2D examples from Li et al.] 2021): the Incompressible Navier-Stokes equation and the Darcy flow equation. Additionally, we compare DSFNO with FNO on the 2D Shallow Water equation sourced from PDEbench (Takamoto et al., 2023 We also conduct an ablation study to systematically analyze the roles of the two components in DSFNO on the Navier-Stokes Equation.
be
Baselines. We include several meaningful baselines for comparison. First, we compare with (1) FNO et al. 2021). as our proposed DSFNO framework aims to enhance the expressiveness of FNO, which serves as the backbone for many frameworks in diverse applications. (2) FFNO (Tran et al. 2023 is also included as it represents a refined version of FNO with separable spectral layers and improved residual connections. As FFNO has fewer learnable parameters, we compare with FFNO using the same channel width and modes, denoted as FFNOsame denoted FFNO. We test against (3) CNO, key model based on fixed-size local convolutions, as well as (4) DeepONet, an influential model that employs learnable basis functions. For the Darcy flow example, we include two transformer-bawe
well as with version using more modes and larger channel width, simply
models, specifically (5) Galerkin Transformer (GT) (Cao) 2021) and (6) ONO Xiao et al. 2024a), though they are excluded from other examples due to poor performance, likely due to transformers' large data requirements or inefficiency to learn in an auto-rogressive manner rauto-regressive settings are not implemented in the experiments from Xiao et al. (2024a); Cao (2021)). Notably, there has been recent surge in transformer-basee architectures. However, our main focus is on improving the
7

---

Published in Transactions on Machine Learning Research (05/2025)
expressive power of FNO, and comparison with all transformer-based architectures is not within the scope of this work. We provide further discussion transformer-baset models in Appendix E Additionally, we include (?) U-Net from Takamoto et al. (2023) and (8) GNO (Aaandkumrr et al. comparison. Many operators, such as DoepONet (Lu et al. 2021) and GNO
2019) for a comprehensive
Anaandumar et al., 2019),
Cherefore, for reference, we al. 2019: in directly predicting the final solution at the last time step given the initial time step for the Navier Stokes and shallow water equations,
are not configured to run in an autoregressive manner in their original setups.
include the results of DeepONet (Lu et al.) 2021) and GNO (Anandkumar
found that autoregressive results for these models fall significantly below expectations. This approach favors non-autoregressive models, as autoregressive models tend to suffer from error accumulation over time (Lippe et al. 2023). As highlighted in Lanthaler et al. (2024); Gao et al. (2025), neural operators can be significantly affected by discretization mismatch errors. Therefore, we do not conduct super-resolution tests or include super-resolution performance as an evaluation metric.
5.1 Incompressible Navier-Stokes Equation
Simulation Description. The Navier-Stokes equations stand as the cornerstone of fluid mechanics, gov- erning the behavior of viscous fluids under various conditions. We consider the 2D Navier-Stokes equation for a viscous, incompressible fluid in vorticity form from Li et al. 2021
O,w(x,t) t) t(x,t) Vw(x, t) vAw(E, 1) f(x).
V u(u t) = 0, w<x, 0) = wo(x),
where x
T (0,T) is the temporal variable, E C (I0, T]; Haer ((0, ((0,1) for any r > 0 is the velocity field, w X is the vorticity, wo L per 1)) is the initial vorticity, v = 1e-1e - or le function. We are interested learning the operator mapping the vorticity up to time 10 to the vorticity up
(0. is the spatial variable,
is the viscosity coefficient, and f I.prr ((0, 1)2: IR) is the forcing
to later time T = 50 for v le 3 and T = 30 for le in an auto-regressive manner in Li et al. (201)).
Sctup. We follow the exact same setup as in Li et al. (2021) and directly present their autoregressive baseline results for reference including U-Net, TF-Net, and ResNet. For FNO, we report both the original results and the updated results from our experiments, reflecting modifications made by the authors to improve FNO's performance after the paper's publication. Further details about these experiments and the dataset are provided in Appendix F
Table 1: Results on the Incompressible Navier-Stoke Equation: The mean and standard deviation of the average relative & error are reported. The reported inference times are for one time step inference.
Common Information
V le -3 3
le 4
# Parameters (M)
Inference Time (ms)
Test (%)
Test (%) 5.973 + 0.286
DSFNO FNO FFNOsame
1.06
3.05
0.562 + 0.039
0.93
3.01
0.795 = 0.071
7.839 = 0.176
0.09
3.39
1.981 = 0.068
10.523 = 0.262
FFNO
0.44
6.16
1.073 0.025
6.880 0.105
CNO
2.66
20.54
1.957 0.126
10.858 + 0.239
GNO
4.61
124.62
26.973 + 3.273
53.073 + 0.926
FNO* *
0.41
N/A
1.28
8.34
U-Net.
24.95
N/A
2.45
11.90
TF-Net"
7.45
N/A
2.25
11.68
ResNet"
0.26
N/A
7.01
23.11
Original error reported in Li al. 2021 the FNO architecture had been updated after the publication. We report
results the new architecture provided in their repository for the fairness of comparison.
Models that are not implemented in an auto-regressive manner.
same; FENO reduces the number of parameters but does not improve runtime, as it is dominated by the FFT. Therefore, wo conduct experiments using the same channel width and number of modes to ensure comparable inference times.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 2: Results on the Shallow Water Equation. The mean and standard deviation of the average relative l2 error reported. We observe that DSENO can improve the expressiveness of FNO and achieve the best performance among all baselines.
(M)
Infer(ms)
Test (%)
DSFNO
1.97
3.24
0.159 + 0.015
FNO
1.54
3.02
0.216 = 0.006
FFNOsame
11.2
3.43
0.854 = 0.058
FFNO
0.54
6.78
0.344 0.012
CNO
2.66
20.37
0.269 + 0.069
U-Net
7.76
2.67
3.964 + 0.109
GNO
4.61
127.93
10.312 + 0.952
DeepONet
2.40
1.63
1.070 + 0.136
+ Models that are not implemented auto-regressively.
Results and Analysis. Table 1 we present the results for the 2D Navier-Stokes equation. DSFNO outperforms all other baselines in terms of accuracy, which indicates that we can enhance the expressiveness of FNO through making the kernel dynamic with the proposed Schwartz operator for FDIs. It is observed that DSFNO incurs similar inference time FNO, this is because NKG through Schwartz kernels, and the cost is mostly induced by the Fourier transform and its inverses. Additionally, the performance of DeepONet is particularly bad on this PDE example, with an error of over 20%: similar phenomena are observed in other works, such as Raonic et al. (2028). However, we want to point out that DeepONet is effective and accurate on many PDE examples (Lu et al. 2022) and DeepONet can work effectively and efficiently on non-rectangular domains as well; some additional architectural and training changes might significantly improve the performance of DeepONet (Lu et al., 2022 although none of our attempts are successful. Therefore, to not misguide the readers and leave impression that DeepONet is suboptimal, we decide to not include its results in Table [ We provide the results in Appendix F.I.1 Additionally, we provide results on the memory and runtime comparison between DSFNO and FNO in Appendix F.1.1
implement an efficient version of
5.2
The Shallow Water Equation
Simulation Description.
We consider the following form of a system of hyperbolic PDEs from Takamoto
ct al.
2023):
0yh + J,hu + -0yhv = 0,
0,hu + 0z
12h +
2 9rh = hghh,b
(10)
oyhu +
12h+ + 2 gnh
-rhhū,b
with a spatial domain Ω = [-2.5, 2.5]2 being the velocities in horizontal and vertical directions, h describing the water depth and describing a spatially varying bathymetry. hu, hv can be interpreted as the directional momentum components and gr describes the gravitational acceleration. We are interested in learning the operator mapping the initial condition hit 0, to later time hit = o(t=T,xx.0 T,
Setup and Results. We the dataset provided by Takamoto al. 2023 details of which can be found in the Appendix F.1.3. Originally, the data is given in spatial resolution of 128 128 and temporal resolution of 101; we downsampled this data to spatial resolution of 64 64 and a temporal resolution of 25. The shallow water equation is time-dependent,, all models except GNO and DeepONet are trained in an autoregressive manner using the first 5 time steps as inputs and predicting the subsequent 20 time steps.
In Table 3 we present the results for the 2D shallow water equation. Similar the Navier-Stokes example, DSFNO outperforms all baselines in terms of accuracy. It suggests that DSFNO enhances the expressiveness of FNO. Similarly, do not observe any significant increase in computational costs.
9

---

Published in Transactions on Machine Learning Research (05/2025)
5.3 The Darcy Flow Equation
Simulation Description.
We consider the steady-state of the 2D Darcy Flow equation from
Li et al.
(2021) given by:
-V -V. (a(x)V((()) f(x). x E (0, 102 u(xx 0, x E 0(0, 1)2
where a E L ((0,)); is the diffusion coefficient and f e L2 ((0,1);: 1)2; R) is the forcing function that is kept lixed [(x) 1. We are interested in learning the operator mapping the diffusion coeflicient a(x) to the
solution u(z).
Sctup and Results. We use the data generator provided by Li et al. (0021) The input diffusion coefficient field a(x,y) is generated by a Gaussian random field with a piecewise fonction. Details of the data gener- ation can be found in Appendix F F.1.2. The Darcy flow equation we consider does not exhibit translation symmetries, for convolution- based models, we apply padding and concatenate coordinate grids to avoid un- intended translational symmetry in the model. As the imput size of the Darcy flow equation is not a power of 2, for U-Net, we apply additional padding to the transposed convolutional layers ensure correct output sizes. Alternatively, we can apply padding interpolation the input obtain inputs of size 64 x 64, but these options result in slightly worse performance. Therefore, we opt to apply padding in the up-convolution layers.
In Table 3 we present the results for the 2D Darcy flow equation. Similar to what we observe in the Navier- Stokes example, DSFNO out performs all baselines in terms of accuracy. It suggests that DSFNO enhances the expressiveness of FNO. Similarly, we do not observe any significant increase computational costs.
Table 3: Results on the Darcy Flow Equation. The mean and standard deviation of the relative &2 error are reported. We observe that DSFNO can improve the expressiveness of FNO and achieve the best performance among all base-
Table 4: Results of the ablation study. The mean and standard deviation of the average relative l2 error are reported. Clearly, both adaptive kernels and frequency domain inter- actions can enhance the performance of FNO. However, DSFNO significantly improves per- formance by larger margin, indicating that both components are indispensable for design- ing an expressive DSFNO.
lines.
(Par.(M)
Infer(ms)
Test (%)
DSENO
2.77
2.87
0.659 + 0.005
FNO
2.36
2.22
0.788 + 0.005
FFNOsame
0.27
2.73
0.754 0.009
FENO
1.72
3.92
0.676 + 0.023
Model
Test (%)
GT
1.99
9.95
0.913 + 0.076
DSFNO FNO
0.562 + 0.039
ONO
1.01
153.83
1.171 + 0.143
0.795 + 0.071
CNO
2.61
17.83
0.762 + 0.014
Adaptive Kernel Only
0.679 = 0.029
U-Net
7.76
2.33
1.294 + 0.003
FDI Only
0.704 + 0.027
GNO
4.61
98.72
3.392 + 0.981
DeepONet
2.48
1.91
2.987 0.099
5.4 Ablation Study on the Navier Stokes Equation
We perform an ablation study to assess the scparate contributions of kernel adaptivity and frequency domain interactions (FDI) within DSFNO. We systematically analyze two DSFNO variants: one with only the dynamic kernel mechanism (no FDI) and the other with only FDI (no dynamic kernels). In the first variant, as defined in Definition 4.1 the NKG is limited to transformations within individual frequency modes, such as pointwise neural network or simple pointwise multiplication with learnable weights. This study isolates the effects of each component. on model performance, clarifying the impact of dynamic kernels and FDI on predictive accuracy in modeling complex operators.
We present the results in Table both variants, dynamic kernel-only and FDI-only, demonstrate slightly bet- ter performance compared to the original FNO and barely increase the computation time. This indicates that
10

---

Published in Transactions on Machine Learning Research (05/2025)
each component independently contributes to enhancing FNO's effectiveness and expressiveness. Moreover, DSFNO, which integrates both dynamic kernel and FDI, consistently outperforms both individual variants. This comprehensive approach leverages synergies between dynamic kernels and frequency interactions, re- sulting in the highest observed performance improvements. These findings underscore the significance of incorporating both dynamic kernel mechanisms and FDI to optimize and advance the capabilities of neural operators solving complex problems efficiently.
6 Conclusion and Future Work
In this work, we propose design DSFNO architecture based on dynamic kernels. Specifically, by consid- ering dynamic kernel that is adaptive to the imput and also induces frequency domain interactions, DSFNO gains the ability to perceive relevant frequency information patterns, facilitating better understanding and representation of complex physical phenomena. We conduct experiments to evaluate our proposed DSFNO. Results show that Fourier neural operators with dynamic kernel can achieve better performance under similar settings and computational budgets. These results underscore the effectiveness of our proposed ap- proach using dynamic kernels Fourier neural operators. Overall, our findings underscore the potential of dynamic kernels in advancing the capabilities of Fourier neural operators, providing a more robust framework for tackling intricate problems in various scientific and engineering domains.
Future Work. It will be interesting to see if designing a more sophisticated neural kernel generator could further improve DSFNO and yield other significant benefits, such as enabling learning of more complex dynamics, reducing the latent, dimension (channel dimension) of FNO, and speeding up both training and inference by minimizing FFT costs associated with high channel dimensions. Furthermore, as noted in the discussion on translation symmetrics Scction 4.2 there a compelling interest in developing general equivariant methods (ailored for other operators such as SNO (Fanaskov & Oseledets 2022a), and DeepONet (Lu et al. ).21). Additionally, it will be interesting to see if DSFNO FNO in learning foundation models for operator learning (Subramanian et al. 2023
better alternative to
References
Anima Anandkumar, Kamyar Azizzadnnssheli, Kaushik Bhattacharya, Nikola Kovachki, Zongyi Li, Burigede Liu, and Andrew Stuart. Neural operator: Graph kernel network for partial differential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2019. URL
https://openreviiee wttp:/neeetoooootnddfg22aaaaaa
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B. Kovachki, and Andrew M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI Journal of computational mathematics, 7:121 157, 2021. doi: 10.5802/smaiijem..77 URL httpps:/smmi: jamaaii-jmmmaa wwweeeennteeeeeee mewereeeeereeeee.oooe smai- amai-jem.760 74/
Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkunar. Spherical fourier neural operators: learning stable dynamics on the sphere. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Johannes Brandstetter, Daniel F. Worrall, and Max Welling. Message passing neural PDE solvers. Tn International Conference on Learning Representations, 2022. URL houLLatttss///ppeeriiieeeee////s
vSix3HPYKST
Shuhao Cao. Choose transformer: Fourier or Galerkin. In Advances in Neural Information Processing Systems (NeurIPS 2021), volume 34, 2021. URL httpp:///ppaaeeeieeereee///rorrp
Gengxiang Chen, Yingguang Li, Xu liu, Qinglu Meng, Jing Zhou, and Xiaozhong Hao. Residual fourier neural operator for thermochemical curing of composites. Journal of Manufacturing Systems, 70:217 229, 2023. ISSN 0278-6125.
Taco Cohen and Max Welling. Group equivariant convolutional networks In Maria Florina Balcan and Kilian Q Weinberger (eds.), Proceedings of The 33rd International Conference Machine Learning,
11

---

Published in Transactions on Machine Learning Research (05/2025)
volume 48 of Proceedings of Machine Learning Research, pp. 2990-29900 New York, New York, USA, 20-22 Jun 2016. PMI.R. URT. https::/pproceedngg. mor...mm..... wwt.:ppress///ooireocsse cot./c16.eecc.mmmmm
Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns homogeneous spaces. Advances in neural information processing systems, 32, 2019.
Lawrence C. Evans. Partial differential equations. American Mathematical Society, Providence, R.I., 2010.
ISBN 9780821849743 0821849743.
V. Fanaskov and Oscledots. Spectral noural operators, 2022a.
Vladimir Fanaskov and I. Oseledets. Spectral neural operators. ArXiv, abs/2205.1057 2022b. URL
hattps:/app) stt.appiieeaatiiccccaaaaa....... Ittp:ooprr...r/ooouusIDI.I 248986643
Wenhan Gao and Chunmei Wang. Active learning based sampling for high-dimensional nonlinear par-
tial differential equations. Journal of Computational Physics, 475:111848, 2023. ISSN 0021-9991. doi:
https://doi.orr/100.00000 http:////.....././........... 202.....1.2118888. URL http:://ww..sieeccediiiie http://hh...aaaaoaoooooooooo/ooa htt.e///eccemmmeeeeeee/rrrtttooo piJ/8002199912200111
Wenhan Gao, Ruichen Xu, Hong Wang, and Yi Liu. Coordinate transform fourier ncural operators for symmetries in physical modelings. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
URL https::/ net/forum?Id-pMD7A4.
Wenhan Gao, Ruichen Xu, Yuefan Deng, and Yi Liu. Discrelization-invaiianee on the discretization mis- match errors in neural operators. In The Thirteenth International Conference on Learning Representations, 2025. URL
D. Gottlieb and S.A. Orszag, Numerical Analysis of Spectral Methods: Theory and Applications. CBMS- NSF Regional Conference Series in Applied Mathematics. Society for Industrial and Applied Mathematics, 1977. ISBN 9780898710236. URL httpss//bookss /ogggeee/////ooooooee/oaaaaloaa
Peishi Jiang, Nis Meinert, Helga Jordão, Constantin Weisser, Simon Holgate, Alexander Lavin, Björn Lüt- jens, Dava Newman, Haruko Wainwright, Catherine Walker, and Patrick Barnard. Digital twin earth coasts: Developing fast and physics- inffome- surrogate model for coastal floods via neural operators, 2021.
David I. Ketcheson, Kyle Mandli, Aron J. Ahmadia,, Amal Alghamdi, Manuel Quezada de Luna, Matteo Parsani, Matthew G. Knepley, and Matthew Emmett. Pyclaw: Accessible, extensible, scalable tools for wave propagation problems. SIAM Journal on Scientific Computing, 34(4):C210 C231, 2012. doi: 10.1137/110856976.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Avizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1-97, 2023. URL
21-1524.html
Anuj Kumar and Tarek Echekki. Combustion chemistry acceleration with deeponets. Fuel, 365:131212, 2024. ISSN 0016-2361. doi: httpps://oii.ogg///
URL hctts://www.scienceet
Samuel Lanthaler, Andrew M. Stuart, and Margaret Trautner. Discretization error of fourier neural opera- tors, 2024. URL
Zijie Li, Dule Shu, and Amir Barati Farimani. Scalable transformer for PDE surrogate modeling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL htttp:://ppprrevies pot/forrmmiddddyyneee
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL pttps://openrrviiee [d=c@P9MQVtmnD]
12

---

http://..cma.222.1144/... Published in Transactions on Machine Learning Research (05/2025)
Zongyi Li, Hongkai Zheng, Nikola Borislavov Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Andrew Stuart, Kamyar Avizzaeenehheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations, 2022. URL https://openrevieer
Phillip Lippe, Bastiaan S. Veeling, Paris Perdikaris, Richard E. Turner, and Johannes Brandstetter. Pde-
refiner: Achieving accurate long rollouts with neural pde solvers, 2023.
Ning Liu, Siavash Jafarzadeh, and Yue Yu. Domain agnostic fourier neural operatons. In Thirty-seventh Conference on Neural Information Procossing Systems, 2023a. URL https://openreviex https///ppperrrreeeeeetrrtrrrru net/forun?
id=ubap5FkbJa
Songming Liu, Zhongkai Hao, Chengyang Ying, Hang Su, Ze Cheng, and Jun Zhu. Nuno: A general framework for learning parametric pdes with non-uniform data. arXiv preprint arXiv:2305.1868 2023b.
Miguel Liu-Schiaffini, Julius Berner, Boris Bonev, Thorsten Kurth, Kamyar Arizzadenesheli, and Anima Anandkumar. Neural operators with localized integral and differential kernels, 2024.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218 2222,, March 2021. ISSN 2522-5839.
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, April 2022. ISSN 0045-7825. doi:
URL http://dxx. www.ddo..og....o........ 101.0rr//0.1016// 1016/j j. cma. 2022.
114778
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202. 11214, 2022.
Omri Puny, Matan Atzmon, Edward J. Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamn and Yaron Lipman. Frame averaging for invariant and equivariant network design. In International Conference Learning Representations, 2022. URL
A. Quarteroni and A. Valli. Numerical approximation of Partial differential equations, volume 23. Springer, 1994.
Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadenesheli. U-NO: U-shaped neural operators. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL httpss//oppenreviev. net.
f forum?id=33o(9Sco.
M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informet neural networks: A deep learning frame- work for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686 707, 2019. ISSN 0021-9991.
doi: octemmoc//arrccc////////22911111
10.045. URL
Bogdan Raonic, Roberto Molinaro, Tobias Rohner, Siddhartha Mishra, and Emmanuel de Bezenac. Con- volutional neural operators. In ICLR 2023 Workshop on Physics for Machine Learning, 2023. URL
https://openreviees www.:/neeeffffiieeeeeeee nett.foruummtddTTee00tt1
Justin Sirignano, Jonathan F. MacArt, and Jonathan B. Freund. DPM: A deep learning PDE augmentation method with application to large-eddy simulation. Journal of Computational Physics, 423:109811, 2020. ISSN 0021-9991.
Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W. Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httpa///wwwe..eee.eo.ooo/////aaa
13

---

Published in Transactions on Machine Learning Research (05/2025)
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüiger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning, 2023.
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators.
Tn The Fleventh International Conference on Learning Representations, 2023. URL httpps://opennreviee.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial
differential equations with physics-informed deeponcts. Science Advances, 7(40) 2021. doi:
10.1126 (..1126/ccidv.aabie htttstts::/// https://ww..siienee httpe/////aaaaaaa/.1////////////
abb8660
Maurice Weiler, Patrick
Forré,
Erik Verlinde, and
Max
Welling,
Equivariant and Coordinate Independent Convolutional Networks. 2023.
URL
https://
cuurice-weiless sesiler.gitlane got.a..io/cnn,nnn..c..ee. hat__book/Beee http://aaaaaaataaaaaaaaoaaaaaaaa
pdf
Haixu Wu, Tengge Hu, Huakun Luo, Jianmin Wang, and Mingsheng Long, Solving sigh-dimensional pdes with latent spectral models. In International Conference on Machine Learning, 2023.
Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang, and Mingsheng Long. Transolver: A fast transformer solver for PDEs general geometries. In Forty-first International Conference on Machine Learning, 2024. URL https://openrevier:
Zipeng Xiao, Zhongkai Hao, Bokai Lin, Zhijie Deng, and Hang Su. Improved operator learning by or- thogonal attention. In Forty-first International Conference Machine Learning, 2024a. URL https:
//opennreveww.nt/// http://www.....oooo.m/.////////.
Zipeng Xiao, Siqi Kou, Zhongkai Hao, Bokai Lin, and Zhijie Deng. Amortized fourier neural operators. The Thirty-cighth Annual Conference on Neural Information Processing Systems, 2024b. URL https:
Mingrui Zhang, Jianhong Wang, James B Tihomole, and Matthew Piggott. Learning to estimate and refine fhuid motion with physical dynamics. Tn Proceedings of the 39th International Conference on Machine Learning. 2022.
Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, Keir Adams, Maurice Weiler, Xiner Li, Tianfan Fu, Yuchong Wang, Haiyang Yu, YuQQig Xie, Xiang Fu, Alex Strasser, Shenglong Xu, Yi Liu, Yuanqi Du, Alexandra Saxton, Hongyi Ling, Hannah Lawrence, Hannes Stärk, Shurui Gui, Carl Edwards, Nicholas Gao, Adriana Ladera, Tailin Wu, Elyssa F. Hofgard, Aria Mansouri Tehrani, Rui Wang, Ameya Daigavane, Montgomery Bohde, Jerry Kurtin, Qian Huang, Tuong Phung, Minkai Xu, Chaitanya K. Joshi, Simon V. Mathis, Kamyar Azizzadenesheli, Ada Fang, Alán Aspuru-Guzik, Erik Bekkers, Michael Bronstein, Marinka Zitnik, Anima Anandkumar, Stefano Ermon, Pictro Liò, Rose Yu, Stephan Günnemann, Jure Leskovee, Heng Ji, Jimong Sun, Regina Barzilay, Tommi Jaakkola, Connor W. Coley, Xiaoning Qian, Xiaofeng Qian, Tess Smidt, and Shuiwang Ji. Artificial intelligence for science in quantum, atomistic, and continuum systems. Foundations and Trends in Machine Learning, 2025. URL https: /ttpp:://rxxiv www.s/iww.w..rooeeooooooo http:///.wwww/////////////// 08423.
Xuanle Zhao, Yue Sun, Tielin Zhang, and Bo Xu. Local convolution enhanced global fourier neural operator for multiscale dynamic spaces prediction, 2023.
14

---

Published in Transactions on Machine Learning Research (05/2025)
A
Additional Visualization of Learned Kernels
We provide additional visualization of the learned kernels of DSFNO in Fig.
Input Function
Learned Dynamic
Output Function
Inpul Function
Learned Dynamic Kemel
Output Function
Figure Visualization of two different pairs of input and output functions on the Darcy flow PDE with their corresponding learned kernels. Note that there are L Cin X Cout DSFNO kernels in total; we present the dynamic kernels for the in-channels in the first layer.
B Proofs
B.1 Proof of Theorem 4.2
Theorem 4.2. Let v(z) be function in a separable Hilbert space such that v(x) admits a Fourier transform, and let (x) be a non-linear function. Assume that (xx) is analytic with respect x. Then, applying the non- linear activation function pointwise to v(x), denoted mathematically as the function composition a v(x), induce interactions omong all frequency modes.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Proof. Let B(g) F(E( denote the Fourier transform of v(z) given by 8(E) Sa o(x)e 2xien dr. If of is a smooth function of æ, then we can expand as
1 21 d0())²² + o 00) +...
(1)==((0)+
(11)
2!
Then,
1
σ do = =00 + 2! (0))))(())++ +....
(12)
and we perform the Fourier transformation,
F(ooo()))(EE> 2 (0))(())) (0)0(E) + o0 / dEo 0 (Eo) @ (g (5- (o) dEo
2x
II dEa d51 0 (Eo) 0 (E1 Eo) ( 11 dEodE1
(0) 3
2 2x
Here, observe that, without loss of generality, as long there exists > 1, and o o²(0) is non-zero at O, such function composition induces interactions among all Fourier modes. Practically, most non-lincar functions satisfy these constraints. In practice, we empirically observe that, even non-linear functions that do not satisfy these constraints, such as ReLU, will still promote interactions among Fourier modes.
B.2 Proof of Theorem
Theorem 4.3. An integral operator T Ha (4) + H' (R.; of the form (Tv)(z) Sea ((x, v)v(y)dy, given that the kernel K is smooth or sufficiently regular ensure that Fourier trans- form is well-behaved, can be expressed as
(T)) (
(9)
where (g') is the Fourier bransform of v(z), and M (E,5") is complex function describing how the operator modulates the input function in the frequency domain.
Proof. First of all, given the assumption and Sobolev function spaces, it is safe to conclude that all the functions' Fourier transforms are bell-behaved. Take the Fourier transform of both sides of the equation
F((To)(<)) = =F [ K(a, ((a,w)o(uddy)
[rd (Fr K(x, ((c,))o()ddy) ei
Jad v(y) (Fr K( v)
ita (( dy
((E,N))(W)dy
R(E,N) Upr (('))) (('))etsv dy.
Define k(E.v) = Sax K(x, i5= dx as the Fourier transform of K with respect to we have
F((TT)(2)) [ R(E,)))(y)dy
[ R(E.v) ( ( dy.
JRN
[ ((E)) (fre RCCE ((e de'
I (g') M (g.g')
Jed
16

---

Published in Transactions on Machine Learning Research (05/2025)
where v(y) = fga 0 (5') e "dg' is the Fourier transform of and M (5,5") fza transform of (5, y) with respective to yi in other words, (5,5") is the Fourier transform of K K( y) over both and y. This completes the proof by applying the inverse Fourier Transform to both sides.
fe- ((,, y) 'dy is the Fourier
C Preliminaries
C.1 Group Equivariance and Translation Symmetries in FNO
Equivariance is property of an operator, such as the solution operator of PDEs, a neural operator, or a layer within neural operator, such that if the imput transforms, the output transforms correspondingly in predictable way.
Definition C.1 (Equivariance). An operator x is said to be equivariant to a group G actions of G on X' and y, respectively denoted by Lg + x X and Lg y + Y, satisfy
E ( ( f), FEX,
where X' and y are subspaces of a function space. In other words, the operator & commettes with actions of G.
In the context of operator learning, I and ng can be considered as transformations, such as translation or rotation, of the input parameter function and the output solution function, respectively. o on the other hand, can be thought of as the solution operator that maps an input function to its corresponding oulput solution.
Equivariance to various groups, including but not limited to SE(n), E(m), SIM(n), can be achieved by group- convolutions (Cohen & Welling 2016 Weiler et al. 2023) or group/frame averaging (Puny el al. 2022).
Let vh) and k(h) be real-valued functions on group with (,(( = v (g h)), the group convolution is defined as:
(v* (h) [ v(g)x h)) dg.
JgG
where e can be regarded as the imput (feature) function and the convolutio kernel.
Integrability over group and the identification of the suitable measure, dg. are necessary for group con- volution. Tt has been shown that with the measure dg, group convolution consistently maintains group equivariance. For all a E G.
(LaU* (1) /
gEa (a (g h) dg
[
lbe v(b)x ((ab) h) db
JOEG v(b)x (b-l a0-1a-11/ h1) db
(U* (a 2/²)
La (u* R)) (h).
Each Fourier layer in FNO consists of a point-wise nonlinear activation function and a convolution over the domain Rdd;
(Ku)(z)= = f <(x - y))
(13)
it follows immediately that Fourier layers equivariant to the translation group Rd
C.2 Frequency Domain Interaction Breaks Translation Equivariance
While the Fourier Neural Operator (FNO) equivariant to translation, the frequency domain interactions (FDIs) break this equivariance. At high level, the simplest FDIs can be seen weighted summation
17

---

Published in Transactions on Machine Learning Research (05/2025)
in the frequency domain. As shown in Appendix B.2 no longer convolution but full integral Sq k (ar, y) ely)dy in the real space. Thus, it is naturally not equivariant. This can also be seen purely from a frequency domain perspective. A translation in the input function corresponds to a phase shift in the Fourier domain:
weighted summation in the frequency domain is
e (z) f e (E) and v v ((--zo) z0) f iEma ((E),
where v(x) is the original function, v(x vo) the translated function by zo, o(E) the Fourier transform of v(z), IF denotes the Fourier transform operator, and & is the frequency variable in the Fourier domain. The spectral convolution in FNO is point-wise multiplication in the Fourier domain by the Fourier convolution theorem:
k(x) ( ( 0)( and kxx) u ro) E, R(E)e LE00 (E (EE)
-
(k. 0)())
Clearly, FNO preserves translation equivariance because the phase shift e500 remains unaffected by the pointwise multiplication in the Fourier domain. However, we introduce FDIs, the phase shift property is disrupted. Suppose a general integral transformation T is applied to the Fourier domain to have FDIs:
T(6(E)) = -/wwln (())
where (E,E') represents the interaction weights across frequencies. Applying this transformation to the translated function:
T (e (()0 = / (E,E)) is' dE'
Unlike the case of pointwise multiplication, the phase shift i5zo no longer preserved consistently across frequencies due to the integral over g'. Consequently, the transformation distorts the structured phase relationship, breaking translation equivariance.
Efficient Implementation of Frequency Mode Interaction
Group
Group 2
Group 3
Group and Mixing within Each Group
Weighting
Mixing
Mixing
Mixing
Mixing Between Groups:
Mixing
Mixing
Mixing
(2) Doluma.via Mixing
(b)
Figure 6: (a) Efficcentt Frequency Domain Interaction by Group and Mixing: First of all, we group the froquencics and perform mixing within each group. Secondly, perform mixing between different groups. (b) Efficient Linear NKG in 2D: For 2D array of frequencies (Fourier coefficients), first apply kernel point-wise to the frequencies; this is to learn a general pattern for individual frequencies. Then, we perform mixing for each row and column, separately. Row-wise mixing combined with column-wis wise mixing learns the interaction across all frequency modes.
18

---

Published in Transactions on Machine Learning Research (05/2025)
As illustrated in Sec. 4.1 and Sec. 4.2 we aim to incorporate FDIs into NKGs. We can implement NKGs as simple spectral neural operator (SNO) (Fanaskov & Oseledets 2022a): : we take truncated Fourier coefficients as input and produce the output coefficients with a fully connected neural network. Obviously, as the network is fully connected, it will induce FDIs. However, as we have multiple Fourier layers, the computational cost of using SNOs as NKGs can be prohibitive. Instead, we implement NKG simply as a linear layer operating on coefficients: we take the truncated Fourier coefficients, denoted as &, as input to a linear layer M.. Here, we no longer restrict M to be square diagonal in spectral convolution. It is obvious that such linear layer will induce FDIs.
Practically, let's assume we have real-valued discrete signals of resolution N (with N equidistant grid points in the domain). The Fast Fourier Transform (FFT) incurs a complexity of O(N log N). Usually, we truncate the number of Fourier modes to some & < N to perform convolution through the spectral domain, thus the complexity of a linear NKG is O(k²). For a relatively large k, which can go as high as N, O(k2) can still result in significant computational costs.
We propose efficient FDI grouping method. For the k truncated frequency terms, we group them into m groups, with each group containing A. frequency terms, assuming m divides k. A weight matrix W1 of size m x in will be applied to each group to perform mixing within the group. We perform mixing bet ween different groups by mixing elements from all groups. This grouping and mixing process is illustrated in part (a) of Figure G Although this grouping may seem complicated, it is natural to consider when the underlying physical system is in 2D or higher dimensions. We will illustrate this with 2D example, and for higher dimensions, this mixing can even be done recursively to further reduce complexity.
Given the discrete equidistant sensor points on a 2D rectangular domain as a 2D array, the resulting Fourier frequencies will also be a 2D array, and we might truncate to only keep the low wrw-feeuency modes. Let r x denote the size of the truncated frequency array. The efficient linear NKG is performed in three steps, illustrated in part (b) of Figure 6 First, we apply kernel point-wise to the frequencies; this is to learn a general pattern for individual frequencies. Second, we multiply a kernel of size r X to every column of the resulting frequency array; this is to learn the interaction within every column. Lastly, we multiply a kernel of size c x e to every row of the resulting frequency array; this is to learn the interaction within every row. Row-wise mixing combined with columu-wise mixing learns the interaction across all frequency modes. If the original truncated frequencies are of size k; while the full matrix multiplication interaction would cost Okk*), the efficient Frequency Domain Interaction lowers the cost to O(K*). The number of parameters involved is also reduced from O(k-) to O(k²), significantly speeding up the training process and reducing memory usage a smaller computational graph (for backward propagation) is constructed
E Recent Transformer-based Models
For complex problems involving turbulent characteristics and small-scale features, such as the Navier-Stokes equations with a very high Reynolds number, global operations are often prone to oversmoothing (Liu-) Schiaffini et al. 2024). Recently, there has been a surge in transformer-basee neural operators, which leverage the powerful sel sell-atteniion mechanisms to capture long-range dependencies while maintaining flexibility in handling multi-scale features. GT (Caol 2021) is the pioneering work that relates attention with kernel integral in the field of operator learning. Many PDE systems are typically discretized into large-scale meshes with complex geometries, a lot of works have been proposed to mitigate such issues; for example, Farmer (Li et al. [2023] factorizes the kernel integral to scale better with the dimensionality of the PDE problem, Transolver (Wu et al. 2024) splits the discretized domain into a series of learnable slices of flexible shapes, and LSM (Wu et al.) 2023) removes the reliance coordinate space and utilizes an attention- hierarchical projection network to efficiently map high-dimensional data into compact latent space with lincar-time complexity.
While (ransformer- based methods have shown great potential in solving high-dimensional multi-scale PDE problems, FNO and its variants remain prominent due to their efficiency in global feature extraction. DSFNO explores the expressive power of FNO and introduces a new adaptive kernel for improved performance. We believe that the most valuable comparison for DSFNO is against FNO. FNO and DSFNO tend to perform better than transformer-basee models when global patterns are more prominent due to their ability to
19

---

Published in Transactions on Machine Learning Research (05/2025)
efficiently and accurately capture such patterns. However, they can be prone over-smoothing when small- scale features are more significant, whereas transformer-based models excel in such tasks. Such issues with FNO have been noted and can be mitigated in [1]. The same approach can be taken for our proposed DSFNO as well; however, it is beyond the scope of our work. In summary, we believe that improving FNO is another important path towards ad advancing the field of operator learning and is just as crucial as transformer-bassed models. Both of these two mainstream approaches have their distinct advantages, and comparisons within each branch are more insightful.
F
Implementation Details
F.1 Data and Experimental Setups
F.1.1 The Navier- Stokes Equation
We consider the 2D Navier-Stoker equation for a viscous, incompressible fluid in vorticity form on the unit torus from
Li et al. 2021 :
0,0 t) u(r,t) (w(x, = vAw(zz f(x), x ZE E
V u () = O,
TEΩIET
w(z, 0) = wo(
xEΩ
where = (0, 1)² is the spatial domain, T = (0, the temporal domain. The initial condition wo(z) is gencrated according to Wy ~ js where f = V/ (0. 73/33-/////// 497) with periodic boundary conditions. The forcing kept. fixed (xx) 0.1 (sin (2 (I1 + x2)) + cos (2t (I) + x2))).. The equation is solved using the ftreainfunntion formulation with pseudospectral method. First, a Poisson equation is solved in Fourier space to find the velocity field. Then the vorticity is differentiated and the non-linear term is computed in physical space, after which it is dealiased. Time is advanced with a Crank-Nicolson update where the non- linear term does not enter the implicit part. All data are generated on a 256 x 256 grid and are downsampled to 64 x 64. We use a time-step of le- 4 4 for the Crank-Nicoles scheme in the generaeeeeeted process where we record the solution every = 1 time units.
Setup. We use the dataset provided by the authors of Li et al. (2021) on their GitHub repository and present their autoregrossive baseline results, including U-Net, TF-Net, and ResNet, for reference directly. This choice is made because these models are not available in their repository and while we attempl to re-implement these models in an auto-regressive way, the resulting performance falls below what is presented in their paper. Morcover, many operators, such as DeepONet (Lu et al. 2021) and GNO (Anandkumar et al. 2019), do not run in an auto-regressive manner. While we allempt to re-implement these operalor models with an auto-regressive approach, the resulting performance falls significantly below expectations, potentially because of high one-step errors that accumulate through time steps. This could be attributed to either the inherent characteristies of the models or the need for substantial additional engineering work. Therefore, for reference, we include the results on DeepONet (Lu et al. 2021) and GNO (Anandkumar et al. 2019) in predicting directly the final solution at the last time step given the initial time step. This would be in favor of non-autoregressive models as autoregressive models will face the issue of error accumulation through time (Lippe et al. )023). Furthermore, we include updated results on FNO from our experiments, reflecting some modifications made to the model by the authors to improve FNO's results after the paper's publication. The average relative l errors are reported. For auto-regressive models, l2 errors reported are averaged over all time steps. All of our results are averaged over 10 runs for 1e 3 and averaged over 5 runs for / 1e due to the extensive computation requirements. Average inference time is measured on batches of size 20. As FNO and DKO are auto-rogressive, we provide the inference time for only one time step for fair comparison with other non-auto-regrest models. Following the setups in Li et al. (2021) the training dataset contains 1000 and 10000 input-output function pairs for v = 1e - and // = Ie respectively, whereas the testing datasct contains 200 input-output function pairs. The primary objective of our proposed DKO framework to enhance the expressiveness of FNO, which is the backbone model of many frameworks for diverse applications. Our primary comparison baseline will be against. FNO itself.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Results of DeepONet. The results of DeepONet the Navier-Stokes equation are listed in Table Again,
want to emphasize that DeepONet is an efficient and effective neural operator. Similar phenomena where DeepONet might observe suboptimal performance are presented in other works, such as Raonic et al. (2023) This should be investigated further in the AI4PDE community, although none of our attempts have been successful.
Table Results on the Incompressible Navier- Stokes Equation.
Navier-Stokes
Common Information
le
le
Par. (M) Inference Time (ms)
Test (%)
Test (%) 26.381 + 5.244
DeepONet
2.40
1.63
18.865 + 5.143
Models that not implemented auto-regressive manner.
Training Time and Peak Memory Comparison between FNO and DSFNO. We report the training time per epoch and peak GPU memory comparison between FNO and DSFNO on the Navier-Stokes equation with a viscosity constant le - in Table G evaluate their computational efficiency. The training time per epoch is reported on NVIDIA Tesla V100 Volta GPU Accelerator 32GB Graphics Card. The standard deviation is several orders smaller than the mean, we only report the mean.
Table 6: Training Time and Peak Memory.
Model
Training Time (s) Peak Memory (MiB)
FNO
18.76
12034
DSFNO
21.59
12238
F.1.2 The Darcy Flow Equation
The Darey flow equation is fundamental equation in fluid dynamics that describes the flow of fluids through porous media. It is crucial in various fields like hydrogeology, petroleum engineering, soil mechanics, and environmental science. We consider the steady-state of the 2D Darcy Flow equation from Li et al. (2021) given by:
-V. (a(x))Vu(()) f(x). uxx) = 0, x E 0(0, 1)2
x E (0, 1)²
where a E -0 1)) is the diffusson coefficient and e L ((0, 1)2; IR) the forcing function that is kept fixed f(x) = 1. We are interested in learning the operator mapping the diffusion coefficient a(x) to the solution u(z).
The input diffusion coefficient field a(x, y) is generated by Gaussian random field with a piecewise function, namely axx, y) = (H), where μ is a distribution defined by = N (0. (-A + 9I)- The mapping & R + I takes the value 12 for positive values and 3 for negative values, and the push-forward is defined point- wise. Solutions are obtained using a second-order finite difference scheme on 241 241 grid and then downsampled to 49 49.
F.1.3 The Shallow Water Equation
We consider the following form of a system of hyperbolic PDEs from Takamoto et al. 2023
0yh + J,hu + -yhw = 0,
0,hu + O2 12h +
2 grh = grh0,b,
12 + ghh 0yhu + 0y
grh0,b
with a spatial domain = 2-2.5, 2.5 u, u being the velocities in the horizontal and vertical direction, h describing the water depth and describing spatially varying bathymetry. hu, he can be interpreted
21

---

Published in Transactions on Machine Learning Research (05/2025)
Table 7: Architectural
for FNO and DKFNO, For fair comparison, they have the exact
same hyper-parameters.
Width
Modes
Fourier Layers
Navier Stokes
20
12
Darcy Flow
32
12
Shallow Water
20
16
Table 8: Training hyper byper-parameters for all models.
Harameeer
Value
Learning Rate
0.001 for all except 0.0001 for GNO
Weight Decay
le -4 4
Scheduler
CosineAnnealing...
Epochs
500 for all except 2000 for DeepONet
Batch Size
20
the directional ummentum components and 9r describes the gravitational acceleration. We are interested in learning the operator mapping the initial condition h(t 0, br y) to later time h(t bT,=T, y).
The specific simulation they include in their benchmark for the shallow-water equations problem is 2D radial dam break scenario. On square domain Ω = [-2.5, 2.5 The water height is initialized circular bump in the center of the domain
< V 2 + y 2.0, for r
h(t 0. =
1.0, for
2 y²
with the radius randomly sampled from ((0.3, 0.7). For generating the datasets, they simulate this problem using the PyClaw (Ketcheson et al. 2012) Python package which offers a comprehensive finite volume solver.
F.2 Model and Training Details
We adhere to the FNO convention by omitting a validation set and training for a fixed number of epochs without early stopping. However, for CNO and GT, we include a validation set and select the best model based on the validation set, as we observe that the errors fluctuate more for these two baselines. All experiments repeated 10 times (except for the Navier-Stokes equation with randomness and ensure robustness of the results. For each baseline comparison, provide the details follows:
le 4) to reduce
FNO Li ct al.
2021 : Adopted from the code provided by the authors in their GitHub repository
with no change to their architecture.
2023 Adopted from the code provided by the authors in their GitHub repository FFNO Tran et al.
with no change to their architecture.
GT (Coo 2021:: :: We re-implemented the Galerkin Transformer using the authors' implementation from their GitHub repository. Since the PDE problems considered in our work are on uniform grids, we chose the identity mapping instead of GNNs as encoders for efficiency.
ONO (Raonic et al. 2023): : Adopted from the code provided by the authors their GitHub repos- itory with no change to their architecture.
2023 Adopted from the code provided by the authors in their GitHub repos- itory with no change to their architecture.
CNO Raonic et al.
22

---

Published in Transactions on Machine Learning Research (05/2025)
U-Net:
Adopted from the code provided by PDEBench Takamoto et al. 2023 with change to their architecture at all.
DeepONet (Lu al. 2021): Adopted from the Torch version provided by the authors in their GitHub repository. Specifially, for the Branch network, have tested both MLP and CNN-based network. We found that CNN-basod network yiclds better performance, network as the Branch. An MLP is used as the trunk nerw
chose CNN-basod
DSFNO: For a fair comparison, DSFNO follows exactly the FNO architecture, except that the Fourier convolution layers with fixed Fourier filters replaced by the dynamic kernels produced by the efficient Schwartz Kernel Operator described in Sec. 4.3
Architectural Hyperparameters. All the baseline models are adopted from their original work with no change at all to the architectural hyperparameters. We report the hyperparameters for FNO and DSFNO Table 7 Note that the hyperparameters for the Navier-Stokes and Darcy flow equations are followed exactly from the original work (Li ct al., 2021) as well.
Training Hyperparameters. We report the training hyperparameters in Table
23


Reviews:
[{"review_id": "s41T0cazpC", "reviewer": "Reviewer_ebVa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n1) Dynamic Kernel Integral enhancing the expressive power of fixed convolution kernels in FNO.\n\n2) Schwartz operators that represent general linear operators between function spaces while facilitating dimensionality reduction for efficiency.\n\n**Weakness:**\n\n1) \"There is no single state-of-the-art neural operator in the domain of neural PDE solvers.\" Proposed method is not compared with latest baselines such as LSM, Transolver, Amortized FNO etc.\n\n[1] Wu, H., Hu, T., Luo, H., Wang, J., & Long, M. (2023). Solving high-dimensional pdes with latent spectral models. arXiv preprint arXiv:2301.12664.\n\n[2] Wu, H., Luo, H., Wang, H., Wang, J., & Long, M. (2024). Transolver: A fast transformer solver for pdes on general geometries. arXiv preprint arXiv:2402.02366.\n\n[3] Xiao, Z., Kou, S., Zhongkai, H., Lin, B., & Deng, Z. (2024). Amortized Fourier Neural Operators. Advances in Neural Information Processing Systems, 37, 115001-115020.\n\n2) Hyperparameter details, architectural details and experimental details are not provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This Paper proposes a dynamic Schwartz operator that induces interactions between modes to enhance the expressiveness of FNOs. It equips FNOs with Schwartz operators to learn dynamic kernels, termed Dynamic Kernel Fourier Neural Operators (DSFNOs). By incorporating this dynamic mechanism, our model gains the ability to capture relevant frequency information patterns, facilitating a better understanding and representation of complex physical phenomena.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) The idea is quite similar to the following paper [1]. Can you describe the advantages it offers compared to [1]?\n\n[1] Xiao, Z., Kou, S., Zhongkai, H., Lin, B., & Deng, Z. (2024). Amortized Fourier Neural Operators. Advances in Neural Information Processing Systems, 37, 115001-115020.\n\n2) How does the computational complexity (training time) change with the dynamic kernel?\n\n3) In FNO, modeling the Navier-Stokes equations with a viscosity coefficient of 1e-5 is more challenging compared to 1e-4 and 1e-3. Why haven’t you included it as a benchmark as well?\n\n4) The complete architectural details are not provided in either the main paper or the supplementary material.\n\n5) How does the model perform on the super-resolution task? Have you conducted experiments on it?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fI63VgwEPb", "reviewer": "Reviewer_1C32", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The analysis of the weakness of the current design in FNO is reasonable and inspiring.\n\n2. This paper is well-written and the overall performance looks convincing.\n\n## Weaknesses\n\n1. The motivation for including frequency domain interaction is not clear enough.\n\nI am convinced by the argument that FNO needs adaptive kernels, but why does DSFNO conduct interaction in the frequency domain? It should be noticed that different elements in the frequency domain actually represent orthogonal bases, which means that they contain less relative information. I cannot understand the advantages of this operation. Besides, if you delve into the amplitude in the frequency domain, you may find that the low-frequency values will be much larger than the high-frequency ones. It may be meaningless to include the high-frequency components in the interaction. Thus, I think more ablation studies or discussions should be presented.\n\n2. More baselines are expected.\n\nAlthough FNO is a classical method in this community, many Transformer-based models achieve better performance, such as Transolver [1] and Factformer [2]. It should be considered to add them to the experiments or at least discuss the relation w.r.t. these new Transformer-based models.\n\n[1] Transolver: A Fast Transformer Solver for PDEs on General Geometries, ICML 2024\n[2] Scalable Transformer for PDE Surrogate Modeling, NeurIPS 2023\n\n3. About the visualization in Fig 1.\n\nIn general, I appreciate the visualization of learned filters. However, what are these visualizations actually about? I think the authors should provide more implementation details of the visualization process to make readers understand the meaning of each filter.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents DSFNOs to enhance the expressive capability of FNOs. Specifically, the authors argue the translation-invariant design of FNO and propose an efficient way to include interactions into the projection process of the frequency domain, which is accomplished by the adaptive kernel. Experimentally, DSFNO surpasses FNO in solving Shallow Water, Darcy Flow and Navier-Stokes equations with only a little extra computation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Discuss the frequency domain interaction under the context that elements in the frequency domain are orthogonal.\n\n2. Compare with more recent Transformer-based baselines.\n\n3. Give more details for Fig 1.", "broader_impact_concerns": "This paper discusses the deficiency of FNO, which can be inspiring for future research.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gOPGykF852", "reviewer": "Reviewer_ct7k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nFor me the biggest strengths is that the authors present a rigorous theoretical foundation, carefully connecting the proposed methods to fundamental concepts in functional analysis and operator theory. The authors provide clear mathematical justification for their approach through the Schwartz kernel theorem and frequency domain analysis. The experimental evaluation is comprehensive, testing across multiple challenging PDE problems with different physical characteristics (thank you for doing 10 runs). The ablation study is particularly well done, systematically demonstrating the contribution of both the dynamic kernel and frequency interaction components. The paper also presents an efficient implementation strategy that keeps computational complexity manageable, showing practical consideration beyond just theoretical improvements.\n\nBut I have several points of weaknesses that needs to be clarified\n\nThe paper's central claim about the benefits of adaptive kernels requires more thorough justification and analysis. While the authors argue that fixed convolution kernels in FNOs limit expressivity, especially for systems lacking translational symmetry, they don't provide convincing theoretical or empirical evidence for why learning input-dependent kernels necessarily solves this limitation. The mechanism by which the dynamic kernel actually captures relevant patterns in the input function remains somewhat unclear. The authors show improved performance but don't adequately explain the underlying reason - is it truly because of better handling of spatial dependencies, or could it be simply due to increased model capacity? For me its the latter reason as you just are learning the frequency interactions but there have been work doing that especially this work [1] https://openreview.net/attachment?id=a6em980M9x&name=pdf where they amortize the FNO in the Spectral block using MLP/KAN instead of just doing a simple linear transformation.\n\nAgain I do like the results, but while they do show improvement over baselines, they don't demonstrate the kind of dramatic performance gains one might expect from a fundamental architectural innovation. This raises questions about whether the added complexity of dynamic kernel generation is truly justified. The paper also lacks detailed analysis of the computational overhead introduced by the neural kernel generator, including memory requirements and training stability considerations.\n\nAlso theoretically all neural operator models are learning the kernel that is a good approximation to the greens function for the PDE. So for a fixed PDE it makes sense to learn the shared solution operator over different initial conditions for that single PDE. So not sure why one would try to learn a dynamic kernel a lot more, this makes sense however when we move on to training foundation models say pretraining over different PDEs? Again I maybe mistaken here but atleast my intuition says that the advantage of a dynamic kernel isnt that much to learning just a static one for a fixed PDE problem ( like the authors have done in this paper, all single PDE experiments).\n\nLastly, it would be great to include ablation studies of how by varying the number of modes, how the architecture performs. The reason the simple FNO works so well is that the linear transform basically zeros out all the high frequency components, but in the paper that is proposed here, all these are based to the generator and mixed, so you could overfit on the noise etc?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors in this paper introduces the Dynamic Schwartz-Fourier Neural Operator (DSFNO), an enhancement to Fourier Neural Operators (FNOs) with two key innovations that I think are worth highlighting:\n1. Dynamic kernels that adapt to input functions, rather than using fixed convolution kernels as in standard FNOs. This allows the model to better handle systems lacking translational symmetry.\n2. Schwartz operators that enable interactions between different frequency modes, expanding beyond the point-wise multiplication in frequency domain used in FNOs.\n\nThe authors also provide (Some) theoretical analysis showing that:\n1. Traditional FNO activation functions induce limited frequency domain interactions that preserve translation equivariance\n2. Any integral operator can be expressed as frequency domain interactions through the Schwartz kernel theorem\n3. An efficient implementation of frequency mode interactions can reduce complexity from O(k⁴) to O(k³)\n\nOverall its a cool work: )!\n\nThe empirical evaluation demonstrates improved performance over FNO and other baselines on three PDE problems:\n1. Incompressible Navier-Stokes equation\n2. Darcy flow equation\n3. Shallow water equation", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Maybe some changes (again not required all of them but it would be great if most of them are incorporated)\n1 - Show some visualizations of how learned kernels vary with different input functions\n2 - Include failure case analysis - when might fixed kernels actually be preferable?\n3 - Add detailed complexity analysis including memory requirements and training time comparisons across methods/\n4 - Analyze stability of dynamic kernel generation\n5 - Better explain the connection between Schwartz operators and frequency interactions\n6 - Clarify how mode truncation affects dynamic kernel learning, again theres lots of works out there understanding the spectral bias of FNO in particular. How would that change this? There is a recent work called Incremental FNO https://openreview.net/forum?id=xI6cPQObp0 which exploits this.\n7 - Compare to the amortized version as mentioned in the weakness.\n8 - Does not clearly explain the architecture of their specific kernel generator? Like what has been used? Showing ablation studies with different generator architectures would be nice.\n9 - Ablation studies in the weakness.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the reviews and revision, all reviewers are happy about the paper being a meaningful contribution on top of FNOs.
Decision: Accept (poster)