Pseudo-Differential Neural Operator: Generalize Fourier Neural operator for Learning Solution Operators of Partial Differential Equations
Authors: Jin Young Shin
Jae Yong Lee
Hyung Ju Hwang
Venue: Accepted by TMLR ()

Abstract: Learning mapping between two function spaces has attracted considerable research attention. However, learning the solution operator of partial differential equations (PDEs) remains a challenge in scientific computing. Fourier neural operator (FNO) is recently proposed to learn the solution operators with an excellent performance. In this study, we propose a novel pseudo-differential integral operator (PDIO) to analyze and generalize the Fourier integral operator in FNO. PDIO is inspired by a pseudo-differential operator, which is a generalization of a differential operator and characterized by a certain symbol. We parameterize the symbol by using a neural network and show that the neural-network-based symbol is contained in a smooth symbol class. Subsequently, we prove that the PDIO is a bounded linear operator, and thus is continuous in the Sobolev space. We combine the PDIO with the neural operator to develop a pseudo-differential neural operator (PDNO) to learn the nonlinear solution operator of PDEs. We experimentally validate the effectiveness of the proposed model by using Darcy flow and the Navier-Stokes equation. The results reveal that the proposed PDNO outperforms the existing neural operator approaches in most experiments.

Published in Transactions on Machine Learning Research (03/2024)
Pseudo-Differential Neural Operator: Generalized Fourier Neural Operator for Learning Solution Operators of Partial Differential Equations
Jin Young Shin' Department of Mathematics Pohang University of Science and Technology (POSTECH), Pohang 37673, Republic of Korea.
sjy60068pastecchcommo uc.kr
Jae Yong Lee'
jaeyong@oau.ock oc.kr
Artificial Intelligence Graduate School Chung-Ang Universily, Scoul 06974, Republic of Korea,
Hyung Ju Hwangt
Department of Mathematics
Pohang University of Science and Technology (POSTECH), Pohang 87673, Republic of Korea.
Reviewed on OpenReview: https: // openrevee.......... net/ forum? id= 905gk20Gq
Abstract
Learning the mapping between two function spaces has garnerod considerable rescarch attention. However, learning the solution operator of partial differential equations (PDEs) remains challenge in scientific computing. Fourier neural operator (FNO) was recently proposed to learn solution operators, and it achieved an excellent performance. In this study, we propose a novel pseudo-differentio integral operator (PDIO) to analyze and generalize the Fourier integral operator in FNO. PDIO is inspired by a preudo-differettall operator, which generalized differential operator characterized by certain symbol. We parameterize this symbol using neural network and demonstrate that the neural network-based symbol is contained in smooth symbol class. Subsequently,
verify that the PDIO is bounded linear operator, and thus is continuous in the Sobolev space. We combine the PDTO with the neural operator to develop a pseudo-differentin newral operator (PDNO) and learn the nonlinear solution operator of PDEs. We experimentally validate the effectiveness the proposed model by utilizing Darcy flow and the Havier-Stokes equation. The obtained results indicate that the proposod PDNO outperforms the existing neural operator approaches in most experiments.
1 Introduction
In science and engineering, several physical systems and natural phenomena are described by partial differential equations (PDEs) (Courant & Hilbert 1953). Approximating the solution of the underlying PDEs crucial to understanding and predicting a system. Conventional numerical methods, such as finite difference methods (FDMs) and finite element methods, involve a trade-off between accuracy and the time required. Tn several complex systems, it may be excessively time-consuning employ numerical methods to obtain accurate solutions. Furthermore, in some cases, the underlying PDE may be unknown.
With remarkable advancements in deep learning, several studies have focused on utilizing deep learning to solve PDEs (Nabian & Meidani 2018 & Yu 2018 Sirignano & Spiliopoulos 2018 Raissi et al. 2019
"Equal contribution.
1 "Correpponding author.

---

Published in Transactions on Machine Learning Research (03/2024)
Hwang et al. 2020: Lee et al. 2021 J. An example is an operator learning (Guo et al. 2016 Bhatnagar et al. 2019 Khoo et al. 2021 which utilizes neural networks to parameterize mapping from the parameters (external force, imitial, and boundary condition) of the given PDE to the solutions of that PDE. In addition,
several studies have employed different convolutional neural networks as surrogate models to solve various
problems, such as the uncertainly quantification tasks for PDEs (Zhu & Zabaras 2018 Zhu et al.
2019)
and PDE-constrained control problems (Holl et al. 2020:
Hwang et al. (2021). Based on the universal
approximation theorem of the operator (Chen & Chen 1995; DeepONet was introduced by Lu et al.
(2019)
In follow-up up studies, extension models of the DeepONet were proposed [Wang et al. 2021 Kissas et al.
2022
Lee et al. 2023).
Another approach to operator learning is via neural operalor, proposed by Li et al. 2020c b a). Li et al. (220cc) proposed an iterative architecture inspired by Green's function of elliptic PDEs. The iterative architecture comprises a linear transformation, an integral operator, and a nonlinear activation function, allowing the architecture to approximate complex nonlinear mapping. As an extension of this rescarch, et al. (2020b) adopted a multi-pole method to develop multi-scale graph structure. Gupta et al. (2021) approximated the kernel of the integral operator using the multiwavelet transform. Recently, various operator
learning models based on neural operators have emerged, such as those presented by Tripura & Chakraborty
(023 and Rahman et al. (0022) The review papers (Lu et al. 2022 Goswami et al. 2023), along with their references, encompass a diverse range of operator learning models.
As one of such models, Li et al. (2020a) proposed a Pourier integral operator using fast Fourier transform (FFT) to minimize the cost of approximating the integral operator. They directly parameterized the kernel in the Fourier integral operator by its Fourier space coefficients, which only depend frequency mode. Here, we analyze the Fourier integral operator from the perspective of pseudo-differential operators (PDOs). PDOs are generalized linear partial differential operators and have been extensively studied mathematically (Boutet de
Monvel 1971 Hörmander 2007 Ruzhansky & Turunen, 2009 Taylor 2017). A deeuudddiffeeential integral operator (PDIO) is proposed to generalize the Fourier integral operator in the FNO based on the PDO. A neural network called symbol network utilized to approximate the PDO symbols] J The proposed symbol network is contained in a toroidal class of symbols; hence, a PDIO continuous operator in the Sobolev space. Furthermore, the PDIO can be applicd to the solution operator of time-dopendent PDEs using time-dependent symbol network.
The main contributions of this study are as follows.
The Fourier integral operator proposed in Li et al. 2020a is interpreted from PDO perspective. The symbol of the Fourier integral operator only depends on frequency domain &, rather than position x. Furthermore, the symbol may not be contained in a toroidal symbol class; hence, the Fourier integral operator cannot be guaranteed to be a contimuuu operator.
A novel PDIO is proposed based on the PDO to generalize the Fourier integral operator. PDIO approximates the PDO using symbol networks. We demonstrate that the proposed symbol network is contained in a toroidal symbol class of PDOs, thus implying that the PDIO with the symbol network a continuous operator in the Sobolev space.
Time-dependent PDIO, a PDIO with time-dependent symbol networks, can be utilized to approximate the solution operator of time-dependent PDEs. It approximates the solution operator, including the solution for time t, which is not in the training data. Furthermore, it is a continuous iattss--i-immeee time operator.
pseudo-diffen -peuddo-ifferenntaal neural operator (PDNO), which comprises a linear combination of our PDIOs and the neural operator proposed in Li et al. (020cc,, is developed. PDNO outperforms the existing operator learning models, such as the FNO by Li et al. (2020a) and the multivavelet-basen operator by Gupta et al. (2021), in hard problems (Darcy flow and Navier-Stokes equation). In particular, the PDNO reduces overfitting better than other models (Figure 1].
Guibas ct al. (202)) proposed adaptive FNO (AFNO), which adopts single layer neural network the Fourier domain. Our application neural network differs from AFNO. Although we utilized neural networks to parameterize the PDO symbol, A ANO adopts a neural network that directly takes frequency variable as imput. (Refer Figure 2
2

---

Published in Transactions on Machine Learning Research (03/2024)
PDNO (test)
0.25
FNO2d (test)
MWT (test)
PDNO (train)
0.20
FNO2d (train)
MWT (train)
Relative L2 Error
0.15
0.10
0.05
10
12
14
16
18
Time (t)
Figure 1: Comparison of the train and the test relative L error by time horizon t 10...., 19 on the Navier-Stokes equation with viscosity == 10-5, FNO and MWT are highly overfitted, while PDNO is not. See Section 5.2 for detailed experimental setups regarding the Navier-Stokes equation.
2 Fourier integral operator and PDO
Here, we attempt to approximate an operator G A - 24 between two function spaces A and U. The operator G can be considered as the solution operators of various parametric PDEs (kindly refer to Section 5 for examples). To determine a map from the function f(x) E A to the solution u(x) U4 we introduce a neural operator architecture to effectively learn infiite-dimmensional operators.
2.1 Neural operator
Inspired by Green's functions of clliptic PDEs, Li ct al. (2020c) proposed an itcrative neural operator to approximate the solution operators of parametric PDEs. First, the input f(xx was lifted to a higher
representation fo(x) P(f(z).. Next, the iterations fo fi... + fr were applied using the update ft + fi+1 formulated by D:
f(+(2)) = -V(l))))
(1)
for t 0,..., T T 1, where W is a local linear transformation and a is a nonlinear activation function. K. denoles an integral operator IC parameterized by & and expressed as
(olfe(<))= - Ko (2. v))(
(2)
where D represents a bounded domain of the input function. The output u(x) = ((T(x)) is the projection of (x) by the local transformation Q. Several studies have considered the best approach to choosing the kernel function Kg and computing the corresponding integral operator. The intogral operator Krg can be
parameterized using message passing on graph networks (Li al. 2020c). Here, we focused on the Fourier integral operator proposed by Li et al. (2020a).
3

---

Published in Transactions on Machine Learning Research (03/2024)
2.2
Fourier integral operator
2020aa proposed neural operator structure with the integral operator KB called Fourier integral operator. By letting k(z,y) = k(œ - y) and utilizing the convolution theorem, they define the Fourier integral operator as
Li et al.
[Rlf(()) = F [[ F(fil(E)
(z) (z)= (R(E) (2).
(3)
where F denotes the Fourier transform and FF1 its inverse. Note that the parameter R({)) is directly paramcterized the discrete space z". The Fourier integral operator can be extended as the generalization concept of a differential operator, PDO.
2.3 PDO
PDOs have been studied since the 1960's. We consider PDE [a(u(x)] f(x) with linear differential operator La La Da. To determine map T from f to u, apply the Fourier transform to obtain the following:
a del Σ
(4)
where & RR represents variables in the Fourier space and f(S) denotes Fourier transform of function f(x). II a(E) never attains zero, we obtain the solution operator of the PDE as
u(x) = T(f)()) der S._ 1 f(
(5)
a(E)
A PDO can be defined as generalization of differential operators by replacing a[E) with a(x, E), called a symbol (Hörmander 2003 2007 First,, we define symbol a(x,E) and class of symbols.
Definition 1. Let p S 1 and S 1. A function a(x, {) is called Euclidean symbol on T R" in class Sg (T" R"), where T is n-dimensional torus if a(2, {) is smooth T x R and satisfies the Sollowing inequality::
10 (xx, E)1 S cap (E)
(6)
for all 3 E Ni, and for all x E T and & E R", where a constant Cag may depend on a and B but not on x and E. Here, (E) dul (1 1/ with the Euclidean norm ||E||-
Note that m, Ps and class 5.g (T" x P²) can be defined as follows.
related to the regularity of the symbol. The PDO corresponding the symbol
(T t Definition 2. The Euclidean PDO T. A u with the Fuclidean symbol a(x, E) E 5%
IR²) is defined
Ta(f(x)= = -f alx, (E) 2
(7)
where f(E) denotes the Fourier transform of function [(x)..
The Euclidean PDO can be re-written using the Fourier transform as
(z)f)x)) = = (dc.
(8)
2.4 Difference between Fourier integral operator and PDO
Comparing the Fourier integral operator CR in (3) and Euclidean PDO T. in (8, there are two main differences. There a question about whether space & E 2 without treating & as a continuous variable. Furthermore, parameters R(E) only consider the dependency on &, while the symbol a(z, {) has a dependency on æ. To generalize the Fourier integral operator based PDO, the key idea of our method to parameterize the Fuclidean symbol using neural networks to
appropriate to parameterize R(E) directly on the discrete
4

---

Published in Transactions on Machine Learning Research (03/2024)
(x)
7(5)
T (x)
FFT (F)
IFFT (F )
X
x
20 (E)
4g (x)
5
X
4g
Figure 2: An architecture of a PDIO with symbol networks 0g, (z) and 0b
og (E). Considering that FFT and
inverse FFT are used, both the input and output are in the form of uniform mesh. Each value og. (x) and ag)(E) (E) is obtained from separate neural networks.
render the symbol smooth. This makes the model smooth, thereby mitigating overfitting by smoothening the symbols (Figure and Proposition
The following section introduces the PDO theory and proposed model based on the PDO. We also derive the proposed model's smoothness from the neural network's smoothness.
3 Proposed integral operator: PDIO
In this section, we cover three parts. First, we discuss the relationship between the symbols and smoothness of PDOs that have been studied in the past. Next, turn to the relationship between FNOs and PDOs. It is important to note that, although FNOs can be considered discrete versions of PDOs, they do not possess sufficiently smooth symbols. In the final part, we propose a PDIO model that has a sufficiently smooth symbol. We present concrete evidence that a neural network using a GELU-like activation function can generate such a smooth symbol (Appendix
3.1 Symbol network and PDIO
The primary idea in our study is to parameterize the Euclidean symbol a(x, {) using neural networks ag" (x, E)- This network is called a symbol network. The symbol network ag az"(s, E) is assumed to be factorized into ag ag"(z, (z,, <) ag (z))azn (E) (refer to B.I]. Both smooth functions og (x) and ap. (E) are parameterized by fully connected neural networks. We propose a PDIO to approximate the Euclidean PDO using the symbol network and Fourier transform follows:
<.]f(<):== : (ag"(x, ()F(((( db. (x) [a (C)FL(EE)
where J denotes the Fourier transform and J= is its inverse. The diagram of the PDIO is presented in Figure 2
Practically, F and F-1 in 9 are approximated by the FFT, which is an effective algorithm that computes the discrete Fourier transform (DFT). Although the symbol network (E) is defined on IR,, the inverse DFT is evaluated only on the discrete space zd. Therefore, the symbol network ( {) should be considered on the restricted domain T x nd (kindly refer to B.2 for details). The following section details the definitions and propertics of the symbol and PDO on T Z" to elucidate the PDIO on the domain T x Z". Morcover, introduce a theorem that bridges the gap between the Euclidean symbol and the restricted Euclidean symbol.
3.2 PDOs on T nn
The discretization of the Euclidean symbol and Fuclidean PDO are defined in this section.
5

---

Published in Transactions on Machine Learning Research (03/2024)
Definition 3. A toroidal symbol class is set (T.(TT" x 2n) comprising toroidal symbols c(x, which are smoolh in for all & E 7", and salisfy the following inequality:
1 02 E)I S cag (c)
(10)
for all a E No, and for all (z.E) E T x Z". Here, Ag represents the difference operators.
The PDO corresponding to the symbol class 576 (T x Z") can be defined as follows:
Definition 4. The torvidal PDO Ta : A + with the toroidal symbol a(x,5) E 5% x 2n) is defined by the following equation:
Ta()(z)= = E
a(x,E) (
(11)
EE
It is well-known that the toroidal PDO Ta(f) with f E (T well defined and Ta(f) E (T")
Ruzhansky Turunen 2009).
Here, it is necessary to prove that the restricted symbol network ag"lr" x2= belongs a certain toroidal symbol class. To connect the symbol network ag and restricted symbol network ag (T= x2", we introduce appropriate theorem that connects the symbols between the Euclidean and toroidal symbols.
Theorem 1. [Ruzhansky & Turunen, 2009) (Connection between two symbols) Let 0 < < 5 and < 8 < 1. A symbol à E Sp (T" x )") is toroidal symbol if and only if there erists Euclidean symbol a E 5Th (T x IR*) such that â = a T xz"
Therefore, is sufficient to consider whether the symbol network (z, belongs to a certain Euclidean symbol class.
3.3 Propositions on the symbol network and PDIO
We demonstrate that the symbol network (x, with the Gaussian error linear unit (GELU) activation function Hendrycks & Gimpel (2016) is contained in a certain Euclidean symbol class using the following proposition:
Proposition 1. Suppose the symbol networks (z)(()) (x) and (g)(E) (E) are fully connected neural networks with nonlinear activation GELU. Then, the symbol network ag (I, az, (z)a²"($) (E) is in (T, xR"). Therefore, the restricted symbol network ag " aa"" (T=z= is in toroidal symbol class (1,0(T" x 2").
Remark 1. Here, we focus on the most important case where 1= and = O, because STh 5 STo for 0 << SSPI and 0 I 5 < (Hörmander, (200)). Although the proposition only considers the symbol network with GELU, il can verified for warions aclivalion functions (refer to
Proof. The fully connected neural network for the symbol network ag, (x) is denoted as follows:
z = A|- b/41 1,2 TT), = (2 (1 = 1,2, In 1),
where w is weight matrix, bl' denotes bias vector in the 1-th layer of the network, represents element-wise activation function, A = x is an input feature vector, and z|!^| = ag azx(()) denotes an output
of the network with 0, = [W["), 60) n= Similarly, we define w:]0, b!'l - and Al (1 = 1,,...,, for the neural network (E).
The neural network (x)) and its derivative are continnous on a compact set T Therefore, 10 (x) ) S cæ for some constant eg > 0 and for all Ng. For the case |a| = 0,
10 00 (E) = lop, (E)I = |W! oz (2 + ||اا|| < c. (E):
(12)
for some constant co 0 because the absolute value of GELU (z)) is bounded by linear function |z]- Notably,
0 08g (E) = whial diag (o' (zf(z-11) x ... x diag diag (z(2=") Wlel...
nn
6

---

Published in Transactions on Machine Learning Research (03/2024)
This result implics that the multi-derivatives of symbol dz (5) with |a| 2 comprises the product of the weight matrix and the first higher derivatives of the activation functions. Furthermore, the derivative of GELU is bounded, and the second or higher derivatives of the function asymptotically become zero rapidly, i.c., ((*) E S(R) when & 2 2 (refer to Definition 5). Accordingly, we have the following inoquality:
og (5) S co 15)
(13)
for all E No with |a| 2 1 for some positive constants Ca. We bound the derivative of the symbol network 0gg (z, E) as follows:
of E)I 10_ag," (z) (z)1l00a 2z (()())(())))))) S Caco(E)
(14)
=Ca
Therefore, the symbol network az" (s, E) is in (T x R2)) as defined Delinition we deduce that azm = ann T x is in (1,((TT x 2*).
Finally, using Theorem
We introduce the theorem on the boundedness of a toroidal PDO as follows:
Theorem& 2. (Ruzhansky & Turunen 2009) (Boundedness of a toroidal PDO in the Sobolev space) Let m E T and E N, which is the smallest integer greater than 2 and let a Th Zn - such that
10 Ag a(x,E) S O(g)m-lal for all (x,E) E T x x
(15)
and all mulli-indices a such that la) S k and all mulli-indices 3. Then, the corresponding toroidal PDO Ta defined in Definition 4 extends to bounded linear operator from the Sobolev space WP*)(T") to the Sobolev space (TT"-m(TT") for all < < do and any s R.
Tmm is in a toroidal symbol class (1,0(TT x Zd) from Proposition Hence, it The restricted symbol network 0g
satisfies the condition in Theorem 2 Therefore, the PDIO K. (9) with the restricted symbol network ag is bounded linear operator from WP (TT) to WP.+-1(TT)) for all < < and R. This implies that the PDIO is continuous operator between the Sobolev spaces. Therefore, expect that the PDIO be applied to neural operator to obtain smooth and general solution operator. A description of its application to the neural operator is presented in Section 4.1
4
Neural operator with PDIOs
4.1 PDNO
Using the proposed integral operator (9) with the neural operator Dp the combined model is called PDNO. Consider the general case in which the input function f((z) and output function (1+1(x) are vector functions. Let f((z)= = [ff,(x)) R" + 18] E Re with the number of input channels Gi and x E R". Then, the PDIO Ka is expressed as:
Cin
Cout
ka(f)(z) Σ
(z) (f (f)FF((c)(
an
(x)
(16)
j=1
where 01, O2 denote the parameters of cach symbol network and Cout represents the number of output channels
with fe41(z) (x) E Reoue Certainly, the symbol network has Cin X Cout outputs for each channel as illustrated in Figure 6. In the experiments, we use three separate symbol networks ag (x), Re (E)), and Tm (azz(E)
Each symbol network takes z E R and & E _2"" as input, and generate Cin X Cout values.
4.2 Time-dependent PDIO
Consider the timo-dependent PDE
du
Lu, u(x,0) = uo(x)) (x, E T x oo).
(17)
Ot
7

---

Published in Transactions on Machine Learning Research (03/2024)
This is well-posed and has unique solution, provided that the operator L is semi-bounded (Hesthaven et al. 2007]. The solution is given by u(x, () (o)). Regarding the ID heat equation, L is cd.n with diffusivity constant c. Then, the solution of the heat equation is given by
u(x,t)= = Σe EEZ
1n e
(18)
This indicates that the mapping from uz( to u(tr, 1) is the PDO with the symbol ex Consequently, propose the time-dependent PDIOs given as follows:
K.. [, == ab, (x, (o0"" (E,
(19)
where (x, t) and 0g, (g, represent time-dependent symbol networks. For the heat equation, we verify that the timo-dependent PDIOs approximate the time-dependent symbol accurately even in a finer time grid than a time grid employed for training. Furthermore, the time-dependent PDIO is applied to obtain the continuous-in-time solution operator of the PDFs (refer to the experiment on the Navier-Stokes equation).
5 Experiments
5.1 Toy example: 1D heat equation
In this experiment, verify whether the proposed time-dependent PDIO learns the symbol of the analytic PDO. We consider the 1D heat equation provided in (17p. The solution operator of the 1D heat equation is a PDO, which is provided in 18). We attempt to learn the mapping from the initial state and time (1o( ()), to the solution u(x,t) t). The initial state ux(x) is generated from the Gaussian random field (0 7- - with the periodic boundary conditions. resolution are set 0.05 and 2nD 1024, respectively. We utilize 1000 pairs of training data, with inputs 0 and outputs at 10 time grids of t 0.05 0.1n (n 9) for cach of the 100 initial states. Wo conduct lesting with 20 initial states, using inputs at = 0 and outpuls at finer time grids of ( 0.05 + 0.05n (n = 0, 1,... 19). The time-dependent PDIO [19 is employed and achieves relative I,² error lower than 0.01 on both the training and test sets. Figure 3 illustrates the symbol network and the analytic symbol provided in (18) on tE.t) E 12, 12) x [0.05, 1]. Although the PDIO learns from a sparse time grid, it obtains an accurate symbol for all E [0.05, 1].
denotes the Laplacian. The diffusivity constant. e and spatial
5.2 Nonlinear solution operators of PDEs
In this section, we verify the PDNO on nonlinear PDE dataset. For all the experiments, employ the PDNO that comprises four layers of the network described in Figure G and (16) with nonlinear activation GELU. Fully connected neural networks are utilized for symbol networks up to layer 3 and hidden dimension 64. The relative I error is adopted for the loss function. Detailed hyperparameters are contained in Wo not truncate the Fourier coefficient in any of the layers, thus indicating that we utilize all of the available frequencies from -[ to [ 1. This because PDNO does not require additional learning parameters, even if all frequencies are utilized. However, because evaluations are required at numerous grid points, considerable memory is required in the learning process. In practical settings, it is recommended to truncate the frequency space into appropriate maximum modes kmar- We observed negligible degradation in performance even when truncation was utilized (refer to All experiments were conducted using up to five NVIDIA A5000 GPUs with 24 GB memory.
Benchmark models We compare the proposed model with the multivavelet-basee model (MWT) and the FNO, which are the advanced approaches based on the neural operator architect For the difference between PDNO (a(x, {)) and PDNO (a(E)), refer to Section 5.3 We conducted the experiments on Darcy flow and the Navicr-Stokes equation. Regarding the Davier-Stokes equation, we utilized the same data presented by Li et al. (2020a). For Darcy flow, we regenerated the data according to the same data generation scheme.
8

---

Published in Transactions on Machine Learning Research (03/2024)
Symbol neural network
1.00
0.81
Time (t)
0.62
0.43
1.0
0.24
0.8
0.05
0.6
-12
-6
12
Analytic symbol
1.00
0.4
0.81
0.2
Time (t)
0.62
0.0
0.43
0.24
0.05
-12
-6
Frequency (E)
Figure 3: Visualization of the learned symbol from the time-dependent PDIO dgl " (I, (S, (top) and analytic symbol a(x, t, = e 4x0.05057 (bottom) of the solution operator of the 1D heat equation. Note that learned og (z, t) constant function according x. i.c. 0z (x, c(f) (Sce Figure 7 Therefore, it does not require æ-coordinate to plot the learned symbol.
Darcy flow The Darcy flow problem is a diffusion equation with an external force, which describes the flow of a fluid through a porous medium. The steady state of the Darcy flow on the unit box is expressed
- V(()))((( = f(x), x [0, 1|a ((()) =0
(20)
E 0(0, 1)
where u, a(z), and fxx denote the density the fluid, diffusion coefficient, and external force, respectively. We attempt learn the nonlinear mapping from a(x) to the steady state u(x), fixing the external force f(x) 1. The diffusion coefficient a(x) generated from (0, (-A + 97)-2), where A the Laplacian with zero Neumann boundary conditions, and d' is the pointwise push forward, defined by (bxx)) 12 if the differential operator V (c) We
0, 3 elsewhere. The coefficient imposes the ellipticity generate a(r) and u(x) using the socond-order FDM on 512 512 grid. The lower-resolution datasct is obtained by subsampling. We utilized 1000 training pairs and 100 test pairs and fixed the hyperparameters for all resolutions.
The results on the Darcy flow are presented in Table 1 for various resolutions s. The proposod model achieves the lowest relative error for all resolutions. Regarding s 32, particularly, MWT and FNO exhibited the highest errors. Furthermore, the proposed model maintains its performance at low resolutions.

---

Published in Transactions on Machine Learning Research (03/2024)
Table 1: Benchmark (relative L error) on Darcy flow on different resolution 8.
Resolution
Data
PDNO (a(æ, E))
PDNO (a(E))
MWT Leg
FNO
train
3.52 X 10-3
4.08 10-3
1.17 10-3
2.65 10-3
= =22
test
3.34 X 10-4
3.82 10 10
1.62 x 10-2
1.78 10
train
2.59 x 10-3
2.98 x 10-3
1.81 x 10-3
2.93 x 10-3
64
test
2.52 x 10 10-3
2.85 x 10
1.08 x 10-2
1.12 x 10
train
1.58 x 10-3
2.57 x 10-3
1.49 x 10-3
2.77 X 10-3
= 128
test
1.62 x 10 10-3
2.45 x 10-3
9.27 10-3
1.04 10-2
train
1.54 x 10-3
2.62 x 10-3
1.34 x 10-3
2.78 10-3
= 256
test train
1.41 x 10-3
2.54 X 10-3
8.83 x 10-3
1.01 10
1.98 x 10-3
2.25 x 10-3
1.32 X 10-3
2.80 X 10-3
512
test
1.93 x 10-3
2.17 x 10-3
9.27 x 10-3
1.02 x 10-2
Table 2: Benchmark (relative I error) on the Navier-Stokes equation on the various viscosity the time horizon T,, and the number of data N.
V = le - 3
V == le
V = le
V = le
Networks
= 50
=
T = 30
T 20
N 1000
N 1000
N 10000
1000
PDNO (a(æ, E))
0.00903
0.1320
0.0679
0.1093
PDNO (a(x, t))
0.0299
0.2296
0.1605
0.1852
MWT Leg
0.00625
0.1518
0.0667
0.1541
MWT Chb
0.00720
0.1574
0.0720
0.1667
FNO-2D
0.0128
0.1559
0.0973
0.1556
FNO-3D
0.0086
0.1918
0.0820
0.1893
Navier-Stokes equation Navior-Stokes equation describes the dynamics viscous fluid. In the vorticity formulation, the incompressible Navier-Stokes equation the unit torus can be expressed
Ow
Vw vdw =,
(x, E (0, (0,7), (x,t) E (0, 1)2 x [0,T),
V P-u=0 0,
(21)
w(r, 0) = wo
x # 1)²,
where w, u, V, and f denote the vorticity, velocity field, viscosity, and external force, respoctively. We utilize the same Navier-Stokes data used in Li et al. (2020a) to learn the nonlinear mapping from w(x, O). w(2 9) to w(x, 10). wr 7), fixing the force f(x) 1 (2:r (2m(zz + +22) vos(2m(2) + x2))). The initial condition wo(x) is sampled from (0, 71.5 (-5 721)-22))) with periodic boundary conditions. We experiment with four Navier-Stokes datasets:(p, (p, .N = (10-3, 3.-3. 50, 1000). (10-4, 4,30 1000), 30, 10000), and (10-5, 20, 1000), where v, T, and N denote the viscosity, final time to predict, and number of training samples, respectively. Notably, the lower the viscosity, the more difficult the prediction. All datasets comprise 64 64 resolutions.
We employ recurrent architecture to propagate along the time domain. From w(x, ..... predicts the vorticity at t 10, w(x, 10). Then, from w(x, 1), w(z, 9), w(x, 10), the model predicts the next vorticity á(x 11). We repeat this process until f T.
..... w(z, 9), the model
For each experiment, we utilize 200 test. samples. For (, (10-3 50, 1000), we utilize batch size of or otherwise. Furthermore, we empoly fixed hyperparameters for the four experiments.
The results of the Navier-Stokes equation are presented in Table In all four datasets, the proposed model exhibits comparable superior performances. Notably, the relative error improves considerably for
10

---

Published in Transactions on Machine Learning Research (03/2024)
True
PDNO 0.2552
FNO 0.3459
MWT 0.3125
3
2
1
-1
-2
-3
Figure 4: Example of prediction on the Navier-Stokes data with w=le-5 showing the prediction w(x; 19) from inputs [w0(x, 0), w(x, 9)] Each value the top of the figure the relative I,2 error between the true w(x, 19) and each prediction. A prediction of FNO is more granular than PDNO, We suspect that this is due to non-smooth symbols of FNO.
Table 3: Benchmark (relative L error) on Burgers' equation on diffcrent resolution 8.
Equation Resolution
PDNO (a(x,E))
PDNO (a(E))
MWT Leg
MWT Chb
FNO
8 256
6.85 x 10 10-4
9.03 x 10-4
1.99 x 10
4.02 x 10
6.49 x 10
8 = 512
8.49 x 10-4
1.22 x 10-3
1.85 x 10-3
3.81 x 10
6.47 x 10
Burgers
8 1024
1.10 x 10-3
1.25 x 10-3
1.84 x 10-3
3.36 x 10-3
6.40 x 10 10-4
8 2048
1.18 x 10-3
1.29 X 10-3
1.86 x 10-3
3.95 x 10
6.47 x 10 10
8 4096
1.78 x 10-3
1.91 x 10-3
1.85 x 10-3
2.99 x 10-3
6.53 x 10 10
(u, T. N) 11005,200 20, 1000), thereby exhibiting the lowest viscosity. Figure displays sample prediction al. = 19, which is highly unpredictable.
5.3 Additional experiments
Darcy flow. On Darcy flow, we perform an additional experiment, which does not utilize symbol network ag (x), but (E). Tn this case, the PDNO has the same structure FNO except for symbol networks. Refer to PDNO (a(E)) in Table [] Although less than the original PDNO, the results of the PDNO without the dependency of the æ-symbol perform better than the other models, including FNO. This explains why the smoothness of the symbol of PDNO is important.
Navier-Stokes equation. On the Navier-Stokes equation, we also present the results with time-dependent PDIO (Section 4.2 in Table 2 This exhibits relatively high error but has the advantage of not. using recursive structure. In addition, for the Navier-Stokes equation with v = le 5, compare the training and test relative 2 errors along time t in Figure 1 All models demonstrate that the test errors grow exponentially according to time l. Among them, PDNO consistently demonstrates the least. lest errors for all time More notable is the difference between the solid and dashed lines, which indicates that MWT and FNO suffer from overfitting, whereas PDNO does not. The same trend is observed for Darcy flow (refer Table D. This might be related to the smoothness of the models' symbols. Furthermore, the symbols of PDNO and FNO are visualized in Figure
Non-smooth solution operator for Burgers' equation. We expect that the PDNO based on the PDO theory will provide a more accurate operator approximation compared to the conventional FNO when handling a smooth solution operator. To obtain a more precise understanding, we conducted additional experiments approximate the solution operator of the Burgers' equation. The 1D Burgers' equation is a nonlinear PDE, which descríbes the interaction between the effects of nonlinear convection and diffusion
11

---

Published in Transactions on Machine Learning Research (03/2024)
PDNO
FNO
-30
0.3
0.5
-20
0.2
Frequency 1 (51)
-10
0.0
0.1
0.0
-0.5
-0.1
10
-1.0
-0.2
20
-1.5
-0.3
30
5 10 15 20 25 30 Frequency (E2)
0 10 15 20 25 30
Frequency 2 (E2)
Figure 5: Examples of the real part of learned symbol o.y" (E) from the Navier-Stokes data with = le -. 5. x-axis and y-axis represent frequency domains. As we used real valued functions, the second coordinate is half the first.
follows:
Ou
8u 11
8Pu Oa
(x,t) (0, (0, xr E (0, 1),
88
8x
(22)
u(x, 0) u4 (x),
where un is the initial state and denotes the viscosity. We attempt to learn the nonlinear mapping from the initial state (x(x) to the solution u(x, 1) at f = 1. The Burgers' equation with small viscosity parameter v generates sharp shocks over time. In such cases, the solution operator becomes less smooth, and it is conceivable that in such scenarios, PDNO may provide a less accurate operator approximation than the FNO. The results from the Burgers data are presented Table 3 along with different resolutions s. As expected, the case the Burgers' equation with non-smooth solutions, the PDNO demonstrated errors that were either comparable
even higher than those of the FNO. In addition theoretical analysis in Section 3 this additional experiment confirms that PDNO is more beneficial when approximating smoother and more continuous solution operators. Refer to Appendix F.2 for more details.
6 Conclusion
Based the PDO theory, we developed novel PDIO and PDNO framework that efficiently learns mappings between function spaces. The proposed symbol networks are in toroidal symbol class that renders the corresponding PDIOs continuous between Sobolev spaces on the torus, which can considerably improve the learning of the solution operators in most experiments. This study revealed an excellent ability for learning operators based on the theory of PDO. However, there is room for improvement in highly complex PDEs such as the Navier-Stokes equation, and the time-dependent PDIOs are difficult. apply to noulincar architecture. We expect solve these problems by employing advanced operator theories (Duistermaat, 1996 Hörmander 1971 Duistermaat & Hörmander 1972) which will ultimately address engineering and physical problems.
12

---

Published in Transactions on Machine Learning Research (03/2024)
Acknowledgments
Jin Young Shin and Hyung Ju Hwang were supported by the National Research Foundation of (N))a(NRI) grant funded by the Korea government(MStT) (No. RS-2023-00219988 and RES-002-0001652666 and by Institute for Information Communications Technology Promotion (ITP) grant funded by the Korea goverament(MMII) (No. No.2009-0-01900, Artificial Intelligence Graduate School Program (POSTECH)). Jac Yong Lee was supported by KIAS Individual Grant (AP086901) via the Center for Al and Natural Sciences at Korea Institute for Advanced Study and by the Center for Advanced Computation at Korea Institute for Advanced Study.
References
Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Prediction of aerodynamic flow fields using convolutional neural networks. Compul. Mech. 64(2):525 545, 2019. ISSN
0178-7675. doi: 10.1007/4400466--11--0744444444 URL htt0///pp////ooooo////0/.///0000
Louis Boutet de Monvel. Boundary problems for pseudo-difeerentaal operatoss. Acta Math., 126(1-2):11-5A 1971. ISSN 0001-5962. doi: D0.107/BBF02390022 URL
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Newral Networks, 6(4):911-917 1995.
R. Courant and D. Hilbert. Methods of mathematical physics. Vol. I. Interscience Publishers, Inc., New York, N.Y., 1953.
J. J. Duistermaat. Fourier inlegral operalors, volume 130 of Progress in Mathemalics. Birkhäuser Boston, Inc., Boston, MA, 1996. ISBN 0-8176-3821-0.
J. J. Duistermaat and L. Hörmander. Fourier integral operators. IT. Acla Math. 128(3-4):883 269, 1972. ISSN 0001-5962. doi: 10.1007/BF02392110 URL http://ppp///oo...o/////////////
Weinan E and Bing Yu. The deep Ritz method: a deep learning- based numerical algorithm for solving varia- tional problems. Commun. Math. Stat., 6(1):1-12, 2018. ISSN 2194-6701. doi: URL
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323 JMLR Workshop and Conference Proceedings, 2011.
Somdatta Goswami, Aniruddha Bora, Yue Yu, and George Em Karniadakis. Physics-informed deep neural operator networks. In Machine Learning in Modeling and Simulation: Methods and Applications, pp. 219 254. Springer, 2023.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXio:2111.13559 2021.
Xiaoxiao Guo, Wei Li, and Francesco Torio. Convolutional neural net works for steady flow approximation. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 481 490, 2016.
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwaaveet--aased operator learning for differential equations. Advances in Neural Information Processing Systems, 34, 2021.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint rrXiv:1606.08415 2016.
Jan Hesthaven, Sigal Gottlieb, and David Gottlieb. Spectral methods for time- dependennddent problems, volume 21. Cambridge University Press, 2007.
13

---

Published in Transactions on Machine Learning Research (03/2024)
Philipp Holl, Vladlen Koltun, and Nils Thuerey. Learning to control pdes with differentiable physics. arXiv preprint arXi:22001.074 077447777777 2020.
Lars Hörmander. Fourier integral operators. I. Acla Math., 127(1-2):79 183, 1971. ISSN 0001-5962. doi:
D0.107/BFF039920 URL
Lars Hörmander. The analysis of linear portial differential operators. I. Classics in Mathematics. Springer- Verlag, Berlin, 2003. ISBN 3-540-00662-11. doi: 10. 001100779978-3662---1499-- URL https://doi.
1007/978-3--42--1197-2 Distribution theory and Fourier analysis, Reprint of the second (1990) edition Springer, Berlin; MR1065993 (91m:3001aa).
Lars Hörmander. The analysis of linear partial differential operators. III. Classics in Mathematics. Springe URL
Berlin, 2007. ISBN 378-3-540-49937-4 doi: 10.1007/978 -978----500-9988-----
978-3-540-49938 Pseudo-diiferretiall operators, Reprint of the 1994 edition.
Hyung Ju Hwang, Jin Woo Jang, Hyeontae Jo, and Jae Yong Lee. Trend to equilibrium for the kinetic Fokker-Planck equation via the neural network approach. J. Comput. Phys., 419:109665, 25, 2020. ISSN 0021-9991. doi: 11.01///..p.22200000660
URL https:///ol: org/10. 10w./101111111 202p://...p............. 2020. 109665
Rakhoon Hwang, Jae Yong Lee, Jin Young Shin, and Hyung Ju Hwang. Solving pde-constrained control problems using operator learning. arXiv preprint arXiv:2111.04911 2021.
Yuehaw Khoo, Jianfeng Lu, and Lexing Ying, Solving parametric PDE problems with artificial neural networks. European J. Aypl. Math., 32(3):421 435, 2021. ISSN 0956-7925. doi: 10. 10.1017/50056779552000012 URL
https://ddi.... org/ 10. 1017 1077/80056677722000011
Georgios Kissas, Jacob Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George Pappas, and Paris Perdikaris. Learning operators with coupled attention. arXiv preprinl. arXiv:2201.01032, 2022.
Jae Yong Lee, Jin Woo Jang, and Hyung Ju Hwang. The model reduction of the Vlasov-Poisson- Fokker- Planck system to the Poisson-Nerns--Plance system via the deep neural network approach. ESAIM Math. Model. Numer. Anal., 55(5):1803- 2021. ISSN 0764-583X. doi: 10.1051/m2an/222111 /2021088. URL https://dbi.orgg/ 10. httg:///....1111/////////2233110
Jae Yong Lee, Woong CHO, and Hyung Ju Hwang. HyperdeepONet: learning operator with complex larget function space using the limited resources via hypernelwork. In The Eleventh Internalional Conference on Learning Representations, 2023. URL http::///ppeuurreeeeeeeee/r/rooa
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint ar Xiv:2001 08895, 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. arXiv preprint arXi::2006.0955
2020b.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigode Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint (xXi:2003.034883 2020c.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identi- fying differential equations based the universal approximation theorem of operators. arxiv preprint 1xXf::1100.03133 2019.
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, 2022.
Mohammad Amin Nabian and Hadi Mcidani. A deep neural network surrogate for high-dimensional random partial differential equations. or Xie preprint or Xrxi 029507 2018
14

---

Published in Transactions on Machine Learning Research (03/2024)
Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadeneshelii U-no: U-shaped neural operators. arXiv preprinl or Kiv:22204. 11127, 2022.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: a deep learning
framework for solving forward and inverse problems involving nonlinear partial differential equations.
J. Comput. Phys., 707, 2019. ISSN 0021-9991. doi: 10.1016/1..cp.2018.10000 URL https:
//doi.org/10.. www./o.//.1.... 101./ wtt...oo//...11/...11 2018. 10.045
Prajit Ramachandran, Barret Zoph, and Quoc V Lc. Searching for activation functions.
arXiv preprint
arXix:17100005 2017.
Michael Reed and Barry Simon. Melhods of modern mathemalical physics. I. Funclional analysis. Academic Press, New York-London 1972.
Michael Ruzhansky and Ville Turunen. Pseudd--iiffrreniall operators and symmetries: background analysis and advanced topics, volume 2. Springer Science & Business Media, 2009.
Justin Sirignano and Konstantinos Spiliopoulos. DGM: deep learning algorithm for solving partial differential 2018. ISSN 0021-9991. doi: 10.1016//..cp.2018..00 icp.208..0.00029 URL
equations. J. Comput. Phys.., 375:1339-1111
https://doi.. aaroi.oorg//1 wttp///.////////........ jcp 2018............................
2018. 08.08..0222 029
Michael Eugene Taylor. Pseudodifferential Operators (PMS-34). Princeton University Press, 2017.
Tapas Tripura and Souvik Chakraborty. Wavelet neural operator for solving paramctric partial differential
equations in computational mechanics problems. Compuler Methods in Applied Mechanics and Engineering, 404:115783, 2023. ISSN 0045-7825, doi: https://doi.orr/11.10011 htt..////.........2220.1.1 cma.2022. 111778. URL httph: /
sciencedirect. cwm http://iieeeeeattccceea////////0
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. arXiv preprint arXiv:2103.10997, 2021.
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder-dccoder networks for surrogate
modeling and uncertainty quantification. J. Comput. Phys., 366:415-441" 2018. ISSN 0021-9991. doi: 10. 10.1016/1.ep..208..0.01 URL htt.httts:///a/ http::://ooo..og/1111001.53
2018.04 14.18.04..11
Yinhao Zhu, Nicholas Zabaras, Siaaednn-Stelion Koutsourelakis, and Paris Perdikaris. Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data. J. Comput. Phys., 394:56-81 81, 2019.
ISSN 0021-9991.
doi: 10.1016/1.cp.2119.000.0
URL
htt:///ii.oog/1..1111110100 1016/ jcp. 2019.05..12
15

---

Published in Transactions on Machine Learning Research (03/2024)
A
Notations
The main notations employed throughout this paper are presented in Table
Table 4: Notations
Notations
Descriptions
A
an input function space an output function space an operator from A to U variable in the spatial domain variable in the Fourier space
G:A-1
R or T
E Rn or 2n f(E)
Fourier transform of function f(z)
Spo
Euclidean (or toroidal) symbol class
a(x,g) E Smp pd
Euclidean (or toroidal) symbol symbol network parameterized by 0 PDO with the symbol a(x,E) PDTO with symbol network ao(x, {) a Fourier integral operator (Li et al. 2020a
(x)
T A A-U
Ko A +ll
Kn A +u
F: A + ;
Fourier transform and its inverse Euclidean norm
II
(E)
(1 |
Ag
a difference operator order a on & the maximum number of Fourier modes
kmaa
B Analysis symbol a(x, E)
B.1 Decomposable assumption a ()) nm (x)aa"(E)
The symbol was assumed to be decomposable because of computational costs. With a decomposable symbol network am" (z), we only needed one IFFT computation
(x(f)x)) ax"(x) Σ mm (S) aa (x)F [ (((T(())(() E
IFFT
With the symbol network (x, E), an input of IFFT depends on the spatial domain Hence, need IFFT computations as many times as the number of grids in the spatial domain œ.
(a())x)) Σ nn (2,)))()))
f-1 (x) {){{
IPFT
Therefore, the non-decomposable symbols require the number of æ grid points multiplied by the cost of decomposable symbols.
B.2 Reason for considering the symbol network on domain T
nn
In this section, we explain why the proposed model should be addressed T" x Tn instead of T x R,"" For
convenience, we assume that n = 1. Let f T and its N points discretization f(+) = yo. ) = y/,
( y(1) = 9N-1- Then, the DFT the sequence fyn JogusN-1 1 is expressed as
N -1
E
ene 2
(23)
N
n=0
16

---

Published in Transactions on Machine Learning Research (03/2024)
and the inverse DFT of (E*JoSAgN-1 is expressed
Un
2mik
(24)
As N Lends oo, it can be deduced that
lim &
lim
N-1
Σ
n
2riik N + //² [(x)e )
20rke dx j(k)
N Noxoc
N
and
lim V/ lim f( N00 -00
1
lim +00
Exe 2rik
f(k)e f(x),
where r R Hence, DFT is an approximation of integral on T and IDFT is an approximation of the infinity sum on Therefore, the PDO thoory on T g is more suitable for our model.
N.
C Hyperparameters
Table 5: Hyperparameters for learning PDNOs on each dataset. # layers, # hidden and activation are for
symbol networks.
Data
Batch size
LR
Weight docay
Epochs
Step size
# Channel
# Layers
# Hidden
Activation
Heat
20
1
16-6
10000
2000
40
GELU
Darcy
20
le-2
le-6
1000
200
20
32
GELU
N-S
20
5e-3
1e-6
1000
200
20
32
GELU
Burgers
20
5e
1000
100
64
TANH
Resource requirements
Table 6: Resource comparison of PDNO and FNO on NS data. PDNO uses two-layer symbol network with hidden dimension 32. The memory requirements are obtained from nvidia-smi command. We used a single NVIDIA A5000 GPU.
Model
kmar
# Channel
Memory (train)
# Parameter
Time (sec/epoch)
PDNO
12
20
9939 MB
1.90 x 10a
20.77
PDNO
12
30
17819 MB
3.91 X 10"
40.73
PDNO
32
20
10143 MB
1.90 x 105
22.05
FNO
12
20
3625 MB
4.66 x 105
4.58
FNO
12
30
3941 MB
1.05 x 106
5.49
FNO
32
20
3957 MB
3.28 x 106
4.80
In Table 6 we compared the memory requirement, the number of parameters, and the training time of PDNO and FNO. PDNO requires more memory Ihan FNO in training because it needs to compule the symbol networks ann (w) and a (E). However, PDNO has fewer parameters than FNO; hence, it requires lower storage save the trained model. If faster inference required, the evaluated values of symbol network may be stored. To reduce memory resourees and time consumption during training, one possible future work is to seoothen the symbol network via regularization the parametric symbol Rig in
17

---

Published in Transactions on Machine Learning Research (03/2024)
E Activation functions for symbol network
In this section, we discuss the activation function for the symbol network. We proved the Proposition | when the GELU activation function was utilized for the symbol network. In addition to GELU, other aclivalion functions can be used for the symbol network. To explain this, we first define the Schwartz space (Reed &
Simon 1972) as follows:
R" - C such that Definition 5. The Schwartz space S(R") is the topological vector space of functions f:R" / E (R") and
0 f(z) - 0. as ||| + co.
(25)
for every pair of malti-indices a, a,3 No.
In other words, the Schwartz space comprises smooth functions whose derivatives decay at infinity faster than any power. As mentioned in the proof of Proposition T it can be easily demonstrated that the second or higher derivatives of GELU are in the Schwartz space S(R). Because GELU is defined as a(2) = =0(=2) with ((z)= = V2n - exp(-u²/2)
the second or higher derivatives of GELU is the sum of exponential decay functions exp(- = =² /2). Hence, the second or higher derivatives of the function are in the Schwartz space, ie., ((k) S(R) when k 2.
Next, we prove that another activation function 6(2) is in symbol class (T x R") if the difference between the function 6(z) and GELU a(2) is in the Schwartz space. We refer to function like Φ(z) the GELU-like activation function. It can be casily demonstrated that the function Φ(2) is bounded by the linear function |z| because GELU is bounded by the linear function. Because the Schwartz space is closed under differentiation, ((z) - (2) E S(R) implies (z) - o (2) E S(R) for & E N. Because GELU satisfies 02) S Ca and (*) E S(R) when k: 2 2, the activation function Φ(2) also satisfics d'(2) S Ca and ( ((RR when k 2 Therefore, the proof of Proposition D can be obtained by altering another activation function ((=) instead of GELU (=). GFLU-like activation functions, such as the Softplus (Glorot al. 2011), and Swish (Ramachandran et al. 2017) etc., satisfy the aforementioned assumption, such that it can be used for the symbol network in our PDIO.
We can easily demonstrate that the symbol network az "z"((r, E) with tanh(z) --: is in (T R"). In the proof of Proposition [ we adopted the characteristic of GELU and its high derivatives. The tanh function is bounded and the first or higher derivatives of the tanh function are in the Schwartz space. Therefore, neural network (E) satisfies the following boundedness:
|0 o0 (E)l S Ca. if |al [80 (5) S ca (g) -ta,, if |a| 21
(26)
(27)
Note that the boundedness of the neural network (x)()) is same in the case of GELU. Hence, can bound the derivative of the symbol network az"
10 of a(x, E)1 I cap (E) lal.
(28)
Therefore, the symbol network with the tanh activation function is in S%,0 (T x RR). Similarly, it is easy to prove that the sigmoid function 1 = is also in symbol class (TM x R"). Therefore, the PDIOs with these two activation functions are bounded linear operators from the Sobolev space W**((T") to the Sobolev space WP (TT) for all 1 < p < oc and any E R.
(T"
F
Additional figuees and experiments
F.1 1D heat equation Symbol network až," (x,
In 1D heat equation experiments, we assume that the symbol is decomposed by a(x, E,t) ze og (x,t) ()x03 20p0015 (E.t). t). Figure illustrates the learned symbol network ag ag (x. 1) on (æ, T [0.05, 1 Evidently ag,
ann (., is almost.
18

---

Published in Transactions on Machine Learning Research (03/2024)
fx()
f41 (x)
WiA0O)
(x)
K (F)
lus2(00
(x)
Wsh00
Linear map
Figure 6: Structure of 1 using the integral operator K.a in 16 with Gi = 3 and Cout 2. Each black solid
line represents a PDIO with symbol network a,y"
1.00
0.750
0.725
Time (0)
0.62
0.675
0.650
0.74
0.05
0.5
1.0
0. 0.600
Location (x)
Figure 7: Learned symbol (x, t) from 1D heat equation.
constant function for each E [0.05, 1]. Accordingly, az, () is considered a function of / by taking the average along the r-dimension to visualize az (z, t)ag (E, in Figure 3
In addition, Figure visualizes a sample prediction on the 1D heat equation.
F.2 Details of experiments on 1D Burgers' equation.
Here, we employ the same Burgers' data utilized in Li et al. (2020a). The initial stale un(x) is generated from the Gaussian random field AO (-0 25/) -2) with the periodic boundary conditions. The viscosity and finest spatial resolution are set to 0.1 and 213 8192, respectively. The lower-resolution datasct is obtained via subsampling. We experiment with the same hyperparameters for all resolutions. In addition, we utilize 1000 train pairs and 100 test. pairs.
19

---

Published in Transactions on Machine Learning Research (03/2024)
Prediction u(x,
True u(x, t)
015
015
010
010
0.05
0.05
0.00
0.00
-0.05
0.10
-0 0.10
0
0.20
0.20
10
10
0.0
0.2
0.4
Time (t)
0.2
0.4
0.6
Time (t)
0.4
0.4
0.6
0.2
0.6
0.2
Position (x)
10
Position (x)
10
Figure 8: A sample of prediction 1D heat equation from PDIO. The model is trained on 1024 x 10 dataset and evaluated on 1024 x 20. Dashed lines on the surface are contour lines.
Darcy (s 256)
Relative La Error
Maximum Modes (kmax)
Figure 9: Test relative 2 error that depends on maximum modes kmaw of PDNO on Darcy flow (resolution 256).
F.3
Changes in errors according to kmazz
As mentioned in Section 5.2 we utilize all possible modes. Although PDNO does not require additional parameters to employ all modes, it demands more memory in the learning process. Hence, we conduct additional experiments on Darcy flow by limiting the number of modes kmaa- In Figure g changes test relative 2 error along kmax are illustrated. Even with small kmazs still outperforms MWT and FNO (Table 1 In addition, for kmaz 20, PDNO obtains comparably relative 122 error the Darcy flow problem.
20

---

Published in Transactions on Machine Learning Research (03/2024)
PONO train
FNO train
PONO test
Relative L Emor
0.08
0.06
0.04
10
40
Number of channels
Figure 10: Train and test error of the proposed model and FNO on Navier Stokes data with = -c - 5 according to the number of channels.
0,200
Train tw/ GeLU)
Test (w/ GeLU)
851
Train (w/ ReLU)
RelU)
80150
Relative L2 error
BC125
8.100
0.087
8,029
8.0000
256
Resolution
Figure 11: Training and test errors of PDNO on Darcy flow according to the resolution of data using GELU and ReLU activation functions for the symbol network.
F.4 Changes in activation of the symbol network.
We attempt to investigate how experimental results differ when using Rectified Linear Unit (ReLU) activation function, opposed to the smooth activation functions discussed in Appendix E for the symbol network. Figure 11 illustrates the training and lest. relative 1 errors as data resolution varies when employing GELU and ReLU activation functions in the symbol network for the Darcy problem. As demonstrated in Proposition L using the GELU activation function helps reduce the overfitting of the solution operator, while employing the ReLU activation function results notable disparity between the training and test relative I errors.
21

---

Published in Transactions on Machine Learning Research (03/2024)
True
PDNO 0.0262
FNO 0.0392
MWT 0.0794
OO
True
PDNO 0.0331
FNO 0.0406
MWT 0.0645
-1
-2
True
PDNO 0.0323
0.0449
MWT 0.0602
1
0
-1
-2
True
PDNO
0.0813
0
Comparison of the prediction Navier-Stokes equation with 1e 5. This test sample shows the lowest relative L error on average of three models.
Figure 12:
F.5
Navier-Stokes equation with le
Samples with the lowest and highest error Figures 12 and Figures 13 present the samples with the highest and the lowest errors, respectively. PDNOs consistently obtain the lowest error at all time steps of both samples.
PDNO and FNO with different number of channels We compare the performance of PDNO and FNO, which varies depending on the number of channels. For a fair comparison, the truncation is not utilized in the Fourier space for both FNO and PDNO. Furthermore, PDNO utilizes only a single symbol network og (E), not (z).. In Figure 10 as the number of channels increases, the test error decreases both models. PDNO achieves lower test errors than FNO and also exhibits negligible gap between the training and test. errors.
22

---

Published in Transactions on Machine Learning Research (03/2024)
True
PDNO 0.0610
0.1050
0.2086
T= 10
True
PDNO 0.1444
FNO 0.2319
MWT 0.3089
T= 13
-2
True
PDNO : 0.3223
FNO 0.4195
MWT : 0.5063
2
16
-2
True
PDNO 0.3757
0.4402
0.5528
2
T= 19
0
Figure 13: Comparison of the prediction Navier-Stokes equation with 1e 5. This test sample shows the greatest relative 2 error average of three models.
23


Reviews:
[{"review_id": "XHPbAuwYmp", "reviewer": "Reviewer_bQQP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a neat application of an existing mathematical concept that could be useful in solving PDEs that depend upon the state in a non-trivial fashion.\n\nThis is a very incremental paper that directly uses an existing idea. It is not clear for what kinds of problems this approach would perform better (the authors do show some results, but are these results conclusive...e.g., do we have enough evidence that this is indeed a replacement for a Fourier neural operator even for these equations in all cases?). In order for these results to be meaningful, there should be some argument, either theoretical or experimental, that characterizes the kinds of problems that will/will not be benefit from this approach, and why.\n\nFor problems that involve learning maps between function spaces, one should characterize the kinds of maps that can/cannot be learned via experiments a bit more precisely. It is not sufficient to simply say that a method obtains the solution for these few equations and so and so resolutions. Ideally, we would like to see detailed trends of how computational complexity of a different neural operator trades-off as the solution demands higher and higher resolutions. A simple example of this is the user trying out different Runge-Kutta methods, (i) one can always use a higher order method but it would be infeasible in any real problem with computational constraints, and (ii) the point of proposing a numerical method to solve a problem is to be able to crisply characterize the kinds of problems that the said method is well-suited for. Indeed, if a real-problem does not require a high-resolution solution of a PDE, then one would be perfectly happy with the Fourier neural operator.\n\nThe paper has a lot of grammatical errors. The authors are advised to proof read the narrative.\n\nIn the reviewers’s opinion, the discussion on the toroidal symbol is not directly relevant to the problem. Solutions to PDEs predicted by the learned neural operator will necessarily have a gap with respect to the true solution (due to sample sizes errors, incorrect modeling assumptions such as truncation of frequencies) and the discretization error of the symbol is just one among these. So in principle, while the argument of the authors that using the pseudo-differential integral operator is better than the Fourier operator is well-taken, it does not seem important to worry about the discretization of the PDIO.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper develops a procedure to use the so-called pseudo differential integral operator (PDIO) to fit the solutions of certain partial differential equations. The idea is as follows. The Fourier integral operator is defined to be\n\nK_R[f](x) = F^{-1}[R(xi) F[f](xi)](x)\n\nwhere F is the Fourier transform and xi in Z are the discrete-valued frequencies. When used to obtain the solution, of say PDE L[u] = f for a linear operator L = sum_i c_i D^i, the solution u can be calculated using\n\na(xi) F[u] = F[f]\n\nwhere a(xi) is the so-called “symbol”. The pseudo-differential operator generalizes the symbol to be a(x, xi), i.e., one can think of it as solution of the PDE using a local Fourier transform calculated at each point in the domain x. This is very well-studied and classical mathematics. There are two key ideas discussed in this paper:\n\n* The general symbol a(x, xi) could be decomposed as a product over the problem domain and the Fourier frequencies as a_1(x) a_2(xi) to fit each term using a multi-layer perception (MLP);\n* since the MLP-based approach will have to use the discrete Fourier transform when it performs its computations, the authors discuss a discretization of the Euclidean symbol (where xi is real-valued) into the “toroidal symbol” (where xi takes values in integers). There are existing results that suggest how to do this discretization effectively. The authors then discuss some minor variations of \n* One can use pseudo-differential integral operators to build what is called a “neural operator” which approximates the solution of a PDE (or a map between two function spaces in general) layer-by-layer.\n\nThe experimental part of the paper demonstrates how to use this operator to calculate the solution of a heat equation, the porous medium equation with a constant boundary condition (Darcy flow), and the Navier Stokes equation for incompressible fluids. The authors argue that modeling the solution using the pseudo-differential neural operator enables the solver (i.e., the neural network) to predict the values of the solution accurately on a grid that is finer than the one in the training set that was used to fit the operator. The experimental results show that this approach has a smaller error than, say the Fourier neural operator, which does not depend upon x.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above. it would be useful to show detailed characterization of which problems this new neural operator would be well-suited for.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3ZcpVgviv5", "reviewer": "Reviewer_LJcT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Architecture modifications appear to be novel\n2. Authors presented reasonable comparison with other architectures and performed ablation study\n\n**Weaknesses** (details are given in the Requested Changes below)\n1. Presentation\n\n    Some parts of the article are not clearly explained, and the overall organization of the material can be improved.\n\n2. Motivation\n\n    It is not immediately clear why one needs to consider novel architecture. Which problem does it resolve and why? Similarly, the necessity of the introduced theory is questionable. Why should the readers care about pseudo-differential operators and toroidal symbols?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a generalization of an integral kernel of the Fourier Neural Operator (FNO) called Pseudo-Differential Neural Operator (PDNO).\n\nThe original FNO kernel has a form\n$$\n\\text{FNO}[f^{\\beta}] = \\text{FFT}^{-1}\\left[\\sum\\_{\\beta}A_{j}^{\\alpha\\beta}\\left(\\text{FFT}\\left[f^{\\beta}(x\\_{i})\\right] (\\xi\\_{j})\\_{j=1:k\\_\\max}\\right) \\right] (x\\_i),\n$$\n$$\nf^{\\beta}(x_i) \\in \\mathbb{R}^{N_\\text{features in}\\times N_x}, ~ A_j^{\\alpha\\beta}\\in \\mathbb{C}^{k_\\max\\times N_\\text{features out}\\times N_\\text{features in}}.\n$$\n\nIn the present article, the authors used\n\n$$\n\\text{PDNO}[f^{\\beta}] =\\sum_{\\beta} b_{i}^{\\alpha\\beta}\\left(\\text{FFT}^{-1}\\left[a_{j}^{\\alpha\\beta}\\left(\\text{FFT}\\left[f^{\\beta}(x_{i})\\right] (\\xi_{j})\\right) \\right] (x_i)\\right),\n$$\n$$\nf^{\\beta}(x_i) \\in \\mathbb{R}^{N_\\text{features in}\\times N_x}; ~ b_i^{\\alpha\\beta}, a_j^{\\alpha\\beta}\\in \\mathbb{C}^{N_x\\times N_\\text{features out}\\times N_\\text{features in}},\n$$\nwhere both $a$ and $b$ are generated by feedforward neural networks evaluated at the grid points in frequency and coordinate domains respectively.\n\nIn words, in comparison to FNO, there are several novelties:\n1. No truncation in frequency space (in some experiments truncation is present).\n2. Weights for linear layers are generated by neural networks with a separate set of trainable parameters.\n3. There is an additional linear transformation that explicitly depends on $x$.\n\nThe authors provide several theoretical results on the relations between the properties of neural networks used to generate weights and the smoothness of the resulting neural operator.\n\nAccording to the author's experiments\n1. PDNO performs better than the multiwavelet neural operator and origin FNO on elliptic partial differential equation with Dirichlet boundary conditions\n2. ditto in most settings for the Navier-Stokes equation on the torus\n3. on the same two problems proposed operator has a better generalization gap", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I consider all the issues mentioned in this section to be critical. In my opinion current paper warrants publication in TMLR after a suitable revision.\n\nI encourage authors to address the questions below, clarify/correct highlighted pieces, or answer with a rebuttal.\n\n**Motivation**\n1. What is the problem? Why do we need to consider new architecture? Reasons that I found scattered in the article:\n\n    i. FNO overfits. Why? What is the evidence for that?\n\n    ii. FNO is not general enough to represent simple time-dependent operators. Is that true? It does not seem to be the case when FNO is used to infer spatiotemporal physical fields, i.e., it applies to the function $u(x, t)$.\n\n    iii. FNO does not constitute a smooth operator in the Banach space. Why is this important? Should we have smooth operators?\n\n    iiii. Why PDNO helps? How and why is it going to solve the problems above?\n\n2. Why do we need the theoretical statements about the symbol of the operator? Suppose we have a continuous operator between Sobolev spaces. Please, explain why this is relevant. Perhaps, it is possible to provide statements on the solution operators for some PDEs in wide use that are continuous operators between Sobolev spaces.\n\n**The list of problematic statements, omissions, unclear parts, etc:**\n1. [page 1] *\"An example is an operator learning (Guo et al., 2016; Bhatnagar et al., 2019; Khooet al., 2021), which utilizes neural networks to parameterize the mapping from the parameters (external force, initial, and boundary condition) of the given PDE to the solutions of that PDE.\"* This description is misguided. The setting authors described is generic and not specific for neural operators.\n2. [page 1] *\"Another approach to operator learning is a neural operator, proposed in Li et al. (2020c;b;a). Li et al. (2020c) proposed an iterative architecture inspired by Green’s function of elliptic PDEs. The iterative architecture consists of a linear transformation, an integral operator, and a nonlinear activation function, allowing the architecture to approximate complex nonlinear mapping. An extension of this work, Li et al. (2020b) used a multi-pole method to develop a multi-scale graph structure. Gupta et al. (2021) approximated the kernel of the integral operator using the multiwavelet transform.\"* The description is significantly incomplete. There are dozens of neural operators, including DeepONet, wavelet neural operators, hierarchical attention, kernel-based methods, SVD-based neural operators, discretization-invariant convolutions, etc. Please, provide more examples or reference some review papers. In particular, I recommend the authors look at \"HyperFNO: Improving the Generalization Behavior of Fourier Neural Operators.\"\n3. [page 2] *\"Figure 1: Comparison of the train and the test relative $L_2$ error by time horizon $t = 10, \\dots, 19$ on the Navier-Stokes equation with viscosity $\\nu = 10^{-5}$. FNO and MWT are highly overfitted, while PDNO is not.\"* Please, provide more details right here. What is a time interval used for training? What is a training strategy?  Is it an autoregression setting or a spatiotemporal \"one-shot\" prediction? What is the number of data points in train and test sets? etc\n4. [page 2] *\"Furthermore, the symbol may not be contained in a toroidal symbol class so that the Fourier integral operator cannot be guaranteed to be a continuous operator\".* Please, clarify. It is easy to show that FNO is bounded in $L_2$ space by construction. It implies that it is a continuous operator.\n5. [page 3] *\"Analysis of Fourier integral operator based on Pseudo-differential operator\".* This title is not accurate. The section does not contain an analysis of the Fourier integral operator. There are several definitions there, but the discussion of properties is absent.\n6. [page 4, Definition 1] What is $\\mathbb{T}^{n}$? Is it a torus? This notation is not defined.\n7. [page 4] Section *\"Proposed integral operator : Pseudo-differential integral operator\".* The content of the section is simple but the presentation is poor. Please, explain the results to the reader in plain English. As I understand authors state that:\n\n    i. In a continuous setting, there is a known relation between the smoothness of the symbol and the properties of pseudo-differential operator (references are given).\n\n    ii. The known results are not directly applicable to the setting authors consider since they work with discrete Fourier transform, so the frequencies live on the lattice. Authors formulate and prove the result valid for the lattice (differential operator is replaced by difference operator).\n\n    iii. Now, it is enough to show that chosen parametrization of the symbol (given by neural networks) fulfills the inequality used in the two statements above. The authors demonstrate that for GELU directly and provide more details in the appendix.\n\n8. [page 4] *\"The parameters $R(\\xi)$ may not satisfy the condition of the toroidal symbol equation 10 in Definition 3 so that the Fourier integral operator cannot be guaranteed to be a continuous operator\".* Forward reference to equation 10 unknown to the reader at a time. The claim about the discontinuity of FNO is questionable.\n9. [page 7] *\"Consider the general case which the input function $f_t(x)$ and the output function $f_{t+1}(x)$ are multi-valued function.\"* Probably, the authors mean a vector function, not an implicit (multi-valued) function.\n10. [page 7] *\"Toy example : 1D heat equation\".* What is the purpose of this section? Perhaps, the authors adopt the approach from https://arxiv.org/abs/2003.03485, which contains a similar example but for an elliptic equation. What is the point of learning the symbol of the heat equation if we know in advance that PDNO has the same parametric form? It barely comes as a surprise that learning in this setting is possible.\n11. [page 8] *\"time grids $t = 0.05 + 0.1n$ ($n = 0, 1, \\dots, 9$)\"* Why time does not start from zero?\n12. [page 8] *\"dataset. For all the experiments, we use the PDNO that consists of four iterations of the network described\".* Why iterations? \"Layer\" sounds more appropriate in this context.\n\n**Additional experiments:**\n1. The role of the smoothness of a symbol is not clear. In all scenarios, the authors consider infinitely smooth symbols. What if they are only continuous (e.g., neural networks with ReLU activations)? Is it going to lead to a wider generalization gap?\n2. The fact that PDNO performs better than FNO is not enough to claim that the reason is the smoothness of PDNO in Banach space. One also needs to consider the problem with the non-smooth operator in Banach space. For example, a ReLU-like operator $$\\\\mathcal{A}[f] =  \\\\begin{cases} f,~ \\\\left \\\\| f \\\\right \\\\| \\\\geq a; \\\\\\\\ 0,~ \\\\left \\\\| f \\\\right \\\\| < a\\end{cases}$$ for some positive real $a$. Is that true that for this operator FNO performs better than PDNO because PDNO constitutes unsuitable prior?\n3. All experiments are performed on smooth inputs and outputs. What if both input and output are discontinuous? For example, simple advection equation with discontinuous data from https://arxiv.org/abs/2203.13181 will do.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1tMXVrjCNp", "reviewer": "Reviewer_fuUu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n- The approach in this paper is theoretically and historically motivated. \n- The empirical results are appreciated. \n\nWeaknesses:\n- The proposed method suggests multiplying the output of the inverse Fourier transform in FNO layers with a function. Much more complex pointwise operations are already considered in prior works and prior codebases. For example, \nAdaptive Fourier Neural Operators: Efficient Token Mixers for Transformers\nand \nFourcastnet: A global data-driven high-resolution weather model using adaptive Fourier neural operators\nconsider putting the pointwise MLP layer after the inverse Fourier transfer layer, which is regarded as more complex than the proposed multiplication. \nFurthermore, such MLP options are heavily adopted in neural operator codebases, I guess as a default mode.\nhttps://github.com/neuraloperator/neuraloperator/blob/main/neuralop/models/fno_block.py#L10\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Pseudo-differential method for operator learning\n\n\nThis paper studies the problem of operator learning between function spaces. Inspired by pseudo-differential operation, the paper proposes a variant of Fourier neural operators that applies a pointwise multiplication after the inverse Fourier transform. \n\nThe paper provides a solid inspiration for the proposed method and backs the study with relevant empirical studies. \n \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Acknowledge the prior work considered the MLP setting. Moreover, a comparison with MLP based model is helpful. ", "broader_impact_concerns": "The work has broader impact to the operator learning studies. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers are either accept or leaning toward accept, although notice incremental nature and misprints.
Decision: Accept (poster)