GIT-Net: Generalized Integral Transform for Operator Learning
Authors: Chao Wang
Alexandre H. Thiery
Venue: Accepted by TMLR ()

Abstract: This article introduces GIT-Net, a deep neural network architecture for approximating Partial Differential Equation (PDE) operators, inspired by integral transform operators. GIT-NET harnesses the fact that common differential operators commonly used for defining PDEs can often be represented parsimoniously when expressed in specialized functional bases (e.g., Fourier basis). Unlike rigid integral transforms, GIT-Net parametrizes adaptive generalized integral transforms with deep neural networks. When compared to several recently proposed alternatives, GIT-Net's computational and memory requirements scale gracefully with mesh discretizations, facilitating its application to PDE problems on complex geometries. Numerical experiments demonstrate that GIT-Net is a competitive neural network operator, exhibiting small test errors and low evaluations across a range of PDE problems. This stands in contrast to existing neural network operators, which typically excel in just one of these areas.

GIT-Net: Generalized Integral Transform for Operator Learning
Chao Wang
chaow@nuss www.ruus......... sg
Department of Statistics and Dala Science National University of Singapore
Alexandre Hoang Thiery
a. t..ttheryynun edu.
Department Statistics and Data Science National University of Singapore
Reviewed on OpenReview: https: // openreview. net/ forum? id= OWKTmr Vkd2
Abstract
This article introduces GIT-Net, deep neural network architecture for approximating Par- tial Differential Equation (PDE) operators, inspired by integral transform operators. GIT- NET harnesses the fact that common differential operators commonly used for defining PDEs can often be represented parsimoniously when expressed in specialized functional bases (e-g. Fourier basis). Unlike rigid integral transforms, GIT-Net parametrizes adap- tive generalized integral transforms with deep neural networks. When compared to several recently proposed altcrnatives, GIT-Not's computational and memory requirements scale gracefully with mesh discrelizations, facilitating its application PDE problems com- plex geometries. Numerical experiments demonstrate that GIT-Net is competitive neural network operator, exhibiting small test errors and low evaluations across range of PDE problems. This stands in contrast to existing neural network operators, which typically excel in just one of these areas.
1 Introduction
Partial differential equations (PDEs) are vital mathematical tool in the study of physical systems, providing means to model, control, and predict the behavior of these systems. The resolution of PDEs is tial component of many computational tasks, such optimization of complex systems (Tröltzsch 2010
Bayesian inference (Stuart. 2010 El Moselhy & Marzouk 2012), and uncerlainty quantification (mithh 2013). Conventional methods for solving PDEs, such as finite element methods (Bathe 2007). finite volume methods (Eymard et al.) 2000). and finite difference mothods (Liszka & Orkisz 1980 are based on the fiittediinnnsional approximation of infinite-dimensionaal function spaces by discretizing the PDE on prede- fined sampling points. However, the increasing demands for PDE solvers with lower computational cost, greater flexibility on complex domains and conditions, and adaptability different PDEs have led to the development of modern scientific computing and enginecring techniques.
The recent advances in neural network methodologies, particularly their ability to process large datasets and approximate complex functions, have made them promising candidates for solving PDEs [Blechschmidt & Emnst (2021). The two primary strategies to utilize neural networks approximating PDE solutions are lunction learning and operator learning, both of which have proven effective addressing PDE solver-related challenges.

---

Neural networks for function learning Neural networks have been widely used as a powerful tool for approximating complex functions. In this framework, solutions to PDEs are parametrized by deep neural networks whose weights are obtained by minimizing an appropriate loss function
E & Yu, 2018 Raissi
et a
2019
Bar & Sochen, 2019; Smith el al. 2020 Pan & Duraisamy 2020 Sirignano & Spiliopoulos 2018
Zang et al. 2020 Karniadakis et al. 2021). For instance, mhysics-informee neural networks (PINNs) (Raissi
et al.] 2019:
approximate solutions to PDEs by minimizing loss functions consisting of residual terms
approximating the physical model and errors for initial/boundary conditions. The theoretical understanding
of this emerging class of methods is an active area of research (He et al. 2020 E et al. 2019 Shin et al.
2020 Daubechies et al. 2022). Compared to traditional methods (eg. finite volume, finite elements), PINNs are mesh-free and can therefore be applicd flexibly to a broad spectrum of PDEs without spocifying a fixed space-discretizaaion.. Additionally, by exploiting high-level aulomatic differentiation frameworks (Bradbury ett al.) 2018 Paszke et al.) 2017), derivative calculations and optimizations can be easily performed without considering the specific regularity properties of the PDEs. However, PINNs need to be retrained for new instances, making them inefficient in scenarios where PDEs need to be solved repeatedly, as is common for instance in (Bayesian) inverse problems Stuart 2010).
Neural networks for operator learning Given PDE, the operator that maps the set of input functions
(eg. boundary conditions, source terms, diffusion coefficients) to the PDE solution can be approximated using
deep neural networks. In prior work (Guo et al. 2016;] Zhu & Zabaras 2018 Adler & Öktem 2017 Bhatnagar
ct al.]
2019 Khoo et al. 2021), surrogate operators parametrized by deep convolutional neural networks
(CNNs) have been employed as a mapping between finite-dimensional function spaces defined on predefined coordinate points. These methods address the need for repeated evaluations of PDEs in applications such as inverse problems, Bayesian inference, and Uncertainty Quantification (UQ). However, the training of these networks is not straightforward for meshes of varying resolutions, and for complex geometries. Furthermore, interpolation techniques are required for evaluating the solutions at coordinates outside the predefined set of coordinate points the approximate operator has been trained on. More recently, and as followed in the present article, the problem of approximating a PDE operator has been formulated as a regression problem between
dimenseadimensionall function spaces
Long et al., 2018
2019
Hesthaven & Ubbiali
2018
Bhattacharya Kissas ct al.
ct al. 2021 Li et al. 2021 Lu et al. 2021 Ong et al., 2022
Gupta ct al., 2021 Cao
2021
2022
Brandstetter al. 2022 Huang el al., 2022). The PDE-Net methodology of Long et al.
2018
2019)
uncovers the underlying hidden PDE models by training feed-forward deep neural networks with observed
dynamic data. Discretization invariant learning is studied in Ong et al. (2022); Gupta et al.
[2021] h. Modern
neural architecture designs (e.g. transformers, attention mechanisms) are applied to operator learning CCo
2021 Kissas et al. 2022 Li et al. 2023). The recently proposed method MAD (Huang et al.
2022 also
achieves mesb-independent evaluation through the use of a two-stage process involving training the model and finding latent space for new instances. In the Fourier Neural Operator (FNO) of Li et al.
(2021) ) the
Fourier transform is leveraged to efficiently parametrized integral transforms. The universal approximation
property of this approach is discussed in Kovachki et al. (2021). The DeepONet approach of Lu et al. (2021)
encodes the input function space and the domain of the output function by using two separate networks;
this method was later improved upon in Lu et al. 2022) by using Principal Component Analysis (PCA), a
technique also referred to as the Proper Orthogonal Decomposition (POD) or Karhunen-Loóve expansions in other communtics. More recently, Bhattacharya et al. (202)) doveloped a general model-reduction framework consisting of defining approximate mappings between PCA-based approximations defined on the input and output function spaces. The PCA-Net neural architecture Bhattacharya et al. (2021); [esthaven & Ubbiali (2018) belongs to this class of methods. An important advantage of this class of methods is that the resulting approximate operators are mesh-independent,, enabling their easy application to varied settings when functions are approximated at disparate resolutions. Some of these approaches have been compared and discussed in terms of their accuracy, memory cost, and evaluation in previous work such as Lu et al. (2022; de Hoop et al. (2022). These studies indicate that the FNO approach is competitive in terms of accuracy for most PDE problems defined on structured grids, although modifications may be necessary for more complex geometries. These comparative studies also hint that DeepONet- methods are more
2

---

flexible and accurate when used to tackle complex geometries. Finally, the CCA-NeeBhattacharye et al. [2021]; Hesthaven & Ubbiali (2018) architecture is more advantageous in terms of its lower memory and computational requirements. Nevertheless, as our numorical experiments presented in Section 3 indicate, the simplistic PCA-Net approach can struggle to reach high predictive accuracies when approximating complex PDE operators; we postulate that it is because the PCA-Net architecture very general and does not incorporate any model-structure contrarily to, for instance, the FNO approach.
Our contribution We propose novel neural network operator, the Generalized Integral Transform Neural Network (GIT-Net), for operator learning between for partial differential equations (PDEs).
function spaces serving as a solver
The GiT-Net architecture is derived from the remark that a large class of operators commonly used to define PDEs (eg. differential operators such as the gradient or the divergence operator) can be diagonalized in suitable functional bases (eg. Fourier basis). Compared to other widely used integral transforms such as the Fourier transform or the Laplace transform, GIT-Net takes advantage of the ability of neural networks to efficiently learn appropriate bases in data-driven manner. This allows GIT-Net to provide improved accuracy and flexibility.
Inspired by the model reduction techniques presented in Bhattacharya et al. 2021 GIT-Net cru- cially depends on Principal Component Analysis (PCA) bases of functions. As both input and output functions are represented using their PCA coefficients, the proposed method is robust to mosh discrotization and enjoys favorable computational costs. When using O(N,) sampling points to approximale functions, the GIT-Net approach results in an evaluation cost that scales as (N,)) floating point operations, markedly more efficient than the ON, log(NN,) evaluation costs of the FNO architecture
(Li et al. 2021
The GTT-Net architecture compared to modern methods for data-driven operator learning, includ- ing PCA-Net, POD--DepOONet, and FNO. For this purpose, the different methods are evaluated on five PDE problems with complex domains and input-output functions of varying dimensions. The mu- merical experiments suggest that when compared to existing methods, GTT-Net consistently achieves high-accuracy predictions with low evaluation costs when used for approximating the solutions to PDEs defined on rectangular grids or more complex geometries. This advantage is particularly pronounced for large-scale problems defined on complex geometries.
The rest of the paper is structured as follows. Section 2 describes the GIT-Net architecture. Section 3 provides an overview of the PDE problems used for evaluating the different methods. The hyperparameters of the GIT-Net
A comparison of the GIT-Net with various other neural network operators is presented in Section 5 Finally, present our conclusions in
studied numerically and discussed in Section
4
Section i Codes and datasets are publicly available !
2 Method
Consider two Hilbert spaces of functions 2 = {f Qu + Rd } and V= (g: Ω2 + respectively defined on the input domain Q2 and output domain 22, For f E l we have that f(x) (x) fdhn (x)) E Rd for x E Qn, and similarly for functions g E V. Consider a finite training set of Nirain 2 pairs (fi, g.) Musain with gi =(f,) where F :
denotes an unkown operator. We seek to approximate the operator F with the member N N₂ of parametric family of operators indexed by the (fnite-dimensional)) parameter o O For probability distribution p(df) of interest the input space of functions U, the ideal parameter
0. E θ is chosen so that il minimizes the risk defined as
0 ト E [(f) No [f)
(1)
IGithub: httpes//gituub.co///aawwaa Genncrr Intogral Transform Neural Network
3

---

For Bayesian Inverse Problems, the distribution p(df) is typically chosen as the Bayesian prior distribution, or as an approximation of the Bayesian posterior distribution obtained from computationally cheap hods (og. Gaussian-like approximations). In practice, the risk is approximated by its empirical version so that the following loss function is to be minimized,
Murin
1 Ntrain
[())
(F()) - (J:)l
(2)
The loss function (2) describes standard regression problem, although expressed in an infinite dimennsionll space of functions, and can be minimized with standard (variant of) the stochastic gradient descent algorithm. In the numerical experiments presented in Section 5 used the ADAM optimizer {Kingma & Ba 2015).
A standard method for approaching this infinite dimensional regression problem consists of expressing all the quantities on functional bases and considering inite-dimensional approximations. Let {eu, and (ev, |22 be two bases of functions of the Hilbert spaces U and V respectively so that functions and g V can be expressed as
(f) Bu,k A21
and
g -
Σ (g b 121
(3)
for coefficients a E R and (g E E. Truncating these expansions at finite order provides finite- dimensional representations of cach element of U and V,
f [a ................................ N.. (f) E R.N
g + a (g): ..., Nv, N. (g) E R-N.,,
and transforms the infiniteddimenniooaal regression problem into a standard finte-dimennsioall regression setting. For example, the PCA-Net approach (Bhattacharya et al. 2021 de Hoop et al. 2022) approximates the resulting finite dimensional mapping F RN - R.V with a standard fully-connected multilayer per- ceptron (MIP) and use Karhunen-Lodes expansions (i.e., PCA) of the probability distributions μ and its push-forward F# (1) as functional bases. Since not all Principal Components (PCs) are necessary, computing full SVD is typically computationally wasteful. Instead, it suffices to compute the first few dominant PCs. While low-rank SVD algorithms based the Lanczos algorithm have been available for some time, modern techniques HHalko et al. 2011) primarily rely on random projections and randomized numerical lincar alge- bra (Martinsson & Troppp 2020). In this lext, when PCA decompositions are needed and a full SVD is too computationally costly to be implemented, we advocate utilizing these randomized techniques.
2.1 Generalized Integral Transform mapping
This section describes GIT-Net, a neural architecture that generalizes the Fourier Neural Operator (FNO) approach (Li et al. 2021). ). It also takes inspiration from the Depthwise Separable Convolutions architoc- ture [Chollet 2017 that has proven useful for tackling computer vision problems. Depthwise Separable Convolutions factorize convolutional filters into shared channel-wise operations that are combined in a fi- nal processing step. This factorization allows one significantly reduce the number of learnable parameters.
For clarity, we first describe the Generalized Integral Transform (GIT) mapping in function space since the finite discretization is straightforward and amounts to considering finitely truncated basis expansions. The GIT mapping relies on the remark that mappings F : + V that represent solutions to partial differential equations (PDF) are often parsimoniously represented when expressed using appropriate bases. For example, large class of linear PDEs can be diagonalized on the Fourier basis. Naturally, spectral methods do rely

---

on similar principles. The GIT mapping described in Section 2.2 is defined as composition of mappings that transform functions f f2 12 + RC where C' 2 denotes the number of channels the computer-vision terminology, to functions g : > R We first describe a related lincar mapping IC that is the main component for defining the GIT mapping.
1. Assume that each component of the imput function f f + RC is decomposed on common basis of functions {bx 1k21 with by : S + R. In other words, each component f for 1 S c S C of the function f = (f f.)., is expressed as
(())) Σ 221
ackk (x()
(4)
for coefficients Oc.kk E R. Define matrix a a= [ac.]] E RRC.K Note that it is different from expanding the function f on a basis of eel-vlludd functions. In contrast, the GIT-Net approach proceeds by expanding each component f : + I on common basis of real-valued functions.
2. A linear change of basis is implemented. The expansion on the basis of functions f is trans- formed into expansion onto another basis of functions {ba with by 9 + R. Heuristically, this can be thought of as the equivalent of expressing everything on a Fourier basis, i.c., a Fourier transform. We have
((() = Eãe, 121
de,k (x(
(5)
for coefficients åc,k E R. Define matrix ā [dc,K] E R...K. Continuing the analogy with Fourier expansions, the coefficients (āe, hk21 represent the Fourier-spectrum of the function fo Ω - R. The mapping + & is linear: it represents a standard (linear) change of basis.
3. The main assumption of the GIT-Net transform is that the different frequencies do not interact with each other. In the degenerate case C = 1, this corresponds to operator that diagonalized in the Fourier- like basis {bx]221. The function g Kf with g (g' (g'....,g') is defined
gz(x)) ba)
where
B
Σ Dd,e, ād,t d=1
(6)
for coefficients Dd,c, R. The coefficients Dd,e,K for S c, S C and k > 1 represent the linear mapping à + B. Crucially, for each frequency index 2 1, the frequencies (Be,x are linear combinations of the frequencies (āa*) only.
As a final step, a last change of basis is performed. Heuristically, this can be thought of as the oquivalent of implementing an inverse Fourier transform in order to come back to the original basis. More generally, the proposed approach only assumes a linear change of basis and expresses the quantities in a final basis of functions (bx (6x]22 with by Ω + R. The basis (ba is not assumed to be the same the original basis (ba)221- We have
(x)) = Σ
Bo Dx(r)
(7)
R21
for coofficients Bo, E R. The mapping + B is lincar.
The operations described above, when expressed in finite bascs of size K instead of infinite expansions,
can be summariizd as follows. The input function f : Ω + RC is represented a [Mc,,] E RCCK when the coordinate functions are expanded on the finite basis {bx K After the change of basis { + (
1K

---

the function is represented as à = [åc E R This linear change of basis can be implemented by the right-mulliilicationn by a matrix E RR, so that a. To succinetly describe the next operation, we use the notation & to denote the froquency-wise operation defined Equation (6) For E RC, and a tensor D Da,c,al E R.CCK, the operation B = à @ D with B E RCCK is defined as
Be Σ d=1
Da.c, ād,k
for any S c C and 1 k K. Finaly, the last change basis (be) + fo can be implemented by the right-multiplicatinn by a matrix Q RK..K The composition of these three (linear) operations, namely 3 Ka with
Ka ((aP))D)
is generalization of the non-local operator introduced in Li et al. 2021 For clarity, have used the notation K. to denote the operator mapping K and the correspouding discretization B Ko when expressed on finite bases. Note that the set of such generalized non-local operators is parametrized by a set of (2 K2 + K C2) parameters while the vector space of all linear transformations from RC,K onto itself has dimension (KC)?. The (nonlincar) GIT mapping defined as
G(a)) o + Kx)
for a matrix T E RC,C and a non-linear activation function a IR applied component-wise. The numer- ical experiments presented in Section use the GELU nonlinearity (Hendrycks & Gimpel 2016) although other choices are ccrtainly possible. more intuitive diagram of the GIT mapping and corresponding lensor's dimensions are shown in Figure !1 Introducing the term T does not make the transformation
Left-multiplication T
Right-muuliplication
Cxe
K
frequency-wise operation
D
CxOxK
Right-mmlliplication
K K
Activation
Learnable parameters
Figure 1: GIT mapping G and corresponding tensor's dimensions.
more expressive when compared to only using the transformation a a(Ca:) since it is possible to define
Ka = ((a Q for another set of parameters (P, Q) so that =o))= Nevertheless, we have empirically observed that the over-parametrization defined in Equation eases the optimization process

---

and helps the model reach higher predictive accuracies. It is worth emphasizing that the GIT-Net mapping, as described in Section 2.2 does not attempt to learn the basis functions by and by and b2- Another similar transformation Ta + (Ka) is also explored and compared with ( numerically for all PDE problems. It is shown that they achieve similar prediction errors Appendix
2.2 GIT-Net: Generalized Integral Transform Neural Network
This section describes the full GIT-Net architecture whose main component the GIT mapping described
in Equation (9. Recall that we are given a training set of pairs of functions {(f:, 9: ]r) Neaan where gi = F(f))
for some unknown operator F 2 - with f: Q - Rd and gi S2 -> Rdon. The purpose of the GIT-Net architecture is approximale the unknown operator F.
In order to transform this infinite dimensional regression problem into finite one, assumee set of functions (Cu,1, P.. with Gu,i f2 + R, as well as set of functions (ev.1,.... Cr,P.)) with Co,i Qr + R. This allows one to define the approximation operator Au U + Rd Rdd such that, for function f = (f) ..... we have
2
Oc, Ca.k
for 1 15 dm
k=1
and coefficents a Rdm.P. given by Au(f); the approximation operator A, V - Ro is defined similarly. For completeness, we define the operator Pu Rddi.P + U that sends a Rdu.Ps to the function (f'...,,J" diy = f (u((x) defined as f = Ek=1 Oc Eu,k- The operator P. Rdout,P + V is defined similarly so that
n Pu (f) and 9 g o.
for any functions E and E V. The mappings P, o Au and P. o A, can be thought of as simple linear auto-encoders [Rumelhart et al., 1985): naturally, more sophisticated (eg. non-linear) representations can certainly be used although we have not explored that direction further. In all the applications presented in this text, we have used PCA orthonormal bases obtained from the finite training set of functions; the approximation operators Au and A, are the orthogonal projections of these PCA bases. Numerically, given discretized input functions {F.) ] Nerain with F Nxd,,, PCA bases computed from matrix F1, F2,..., F N. Similarly, PCA bases of discretized output functions can be computed. Other standard finite dimensional approximations (eg. fiite--lement, dictionary learning, spectral methods) could have been used instead. Through these approximation operators, the infinite-dimessional regression problem transformed into a finite-dimensional problem consisting of predicting (FFF( Rdown, from the input Ao(f) E R...P..
we consider a lifting operator Φt RRdm.. - RRC that maps E Rdd.. to (a) E RC,K with number of channel C > 1 possibly significantly higher than d Similarly, we consider a projection operator at RRC,K - Ps In the numerical experiments presented in Sections and 5 the lifting and projection operators were chosen as simple linear operators defined as
To leverage the GIT mapping 9
t (a) = - R and T+(aa) = -LaR
for (learnable) matrices L' E C.di and R E R and L Rdoak K and R RK..: we chose P. = P. K, but that is not requirement. The unknown operator F U - V is approximated by the GrT-Net neural network N : u + V depicted in Figure 2 and defined as
N = P. o ot Gl . A..
(10)
L transtormations

---

Input (f.)
Au
(din. P..)
Teff-muleiplioacionn
Lifting d
(C, K)
Riihht-nnliiplication
Ix
GIT mapping
(C, K)
Projection d
(dout P.)
Left-molriplioationn
Pg
(dout, Po)
niight-rultipliratiin x
Output (g:}
(4)
(b)
Figure 2: (a) The architecture of an L-layer GIT-Net. The dimension of data after each operation is shown.
(b) Top: lifting layer; bottom: projection layer. The blocks in orange are learnable parameters.
The GIT-Net operator N u + V is defined by composing 2 1 different GIT-Net layers activation functions a are used for ()) = 1.... I- last layer G(L), The operator first approximates each component of the input function f = (f',.... fdu) as a lincar combination of Pa 2 1 functions (fu,1,.... P., thanks to the approximation operator AA.. The finito representation (f) E Rdd.PP. then lifted higher dimensional space of dimension R... thanks to the lifting operator t", before going through I, 2 nonlinear GIT mappings (9) Finally, the representation is projected down to P.. in order to reconstruct the output g = N(f) E V. Each coordinate of the output linear combination of the functions (ea,1. Ce.,P.) through the operator P..
9 nonlinear and identity activation function is used instead in the
function 9 is expressed As explained in Section 2 the learnable parameters are obtained by minimizing the empirical risk defined
in Equation
3 PDE problems
In this section, we introduce the PDE problems employed evaluate the performance of GIT-Net compar- ison with other established methods. These PDEs are defined on domains ranging from simple rectangular grids to more complex geometries. The mumerical experiments include the Navier-Stokes equation, the
Helmholtz equation, the advection equation, along with the Poisson equation. The datasets for the Navier- Stokes, Helmholtz, and advection equations are identical to those used in the work of de Hoop et al. 2022 The dataset. for the Poisson equation follows the setup described in the work of Lu el al. 2022).
Navier-Stokes equation Following the setup of de Hoop et al. (2022), the incompressible Navier-Strokes equation is expressed in the vorticity-stream form on the bwo-dimensional periodic domain [0, 2m72. It

---

reads as
Ow
+ (v. , V)w vAw)
(a,t) f(x), (æ, E D x [0,T].
0t
w A S v/- = 0,
(x,t) 6 D [0,T].
(11)
aい 0bb
(a, E D x (0,T),
0xz²
dr1
w (z, 0) wa(ar))
a: E D
where the quantitics f()) and w(x; t) and u(x, t) represent the forcing, the vorticity field, and the velocity field, respectively. The viscosity v is fixed at 0.025 and the final time is set to T 10. In this study, we investigate the mapping f(x:) r 7) for a D. The input function is a sample from a Gaussian random field (V I)); where the covariance operator is given by I (-A + 9)-² and A is the Laplacian operator the domain D = [0, 2T)² with periodic boundary conditions. The output function w(,,T) is obtained by solving Equation 11 with the initial condition wo The initial condition wh is generated from the same distribution as f. For solving this equation, the domain is discretized on uniform 64 x 64. Further details can be found in de Hoop et al. (2022). The same training and lesting sets as de Hoop et al. 2022) were used in the numerical experiments. Figure illustrates pair of input and output functions.
w(,
Figure 3: Navier-Stokes Equation
Left: input force f(x). Right: final vorticity field wxx T).
Helmholtz equation Following the setup de Hoop et al. (2022) Equation
12
describes the Helmholtz
equation on the two-dimensional domain = 0,1
w2
== 0,
x E D,
(xx)
du (z) uN (x))
a E B
(12)
On
Ou (x) O,
x AD A
where [(x, 1) x E [0, 1]) C AD. The quantity c(x) is the wavespeed field and the frequency set 103. The boundary condition is given by uN is (0.335 We are interested in approximating the mapping c(x) u(x). The wavespeed is defined c(x:) = 20 tanh(f) where the quantity & D - R is sampled from the Gaussian random field & (O,, where the covariance operator is given by r = (-A+9)-2 and A is the Laplacian operator on the domain on D with the homogeneous Neumann boundary conditions. The solution u D + IR is obtained by solving (12) using the finite element method on a uniform grid of size 101 x 101. Numerical experiments were implemented using a dataset identical to the one presented in de Hoop et al. (2022) for training and testing. Figure shows an example of wavespeed c: + E and associated solution u R.

---

55
6.5
Figure 4: Helmholtz Equation
12
Left: (input) wavespeed field c(x).. Right: (output) solution u(x:).
Structural mechanics equation We consider two-dimensional plane-stress elasticity problem (Slaugh-
ter 2012) on the rectangular domain D [0, 1|2 H with round hole H. removed in the center, as depicted
in sigure
It. is defined as
V 0 = O, 1
x D,
g Vuu (Vu)
D,
2
(13)
o C:E,
a D,
n b(x)
x
u(x)) 0,
0D B
for displacement vector u, Cauchy stress tensor o, infinitesimal strain tensor E, fourth-order stiffness tensor [0, 1]} OD. The notation 31' denotes the inner product between two second-
C and B = {zx11) order tensors (summation over repeated indices is implicd). The Young's modulus is set to 2 x 10 and the Poisson's ratio to 0.25. We are interested in approximating the mapping from the boundary condition to the von Mises stress field. The input boundary condition 6(x) is sample from the Gaussian random field ((100, T) with covariance operator 400 (-A + 9)-1) where is the Laplacian on the domain with Neumann boundary conditions. Equation 13 is solved with the finite element method using a triangular mesh with 4072 nodes and 1944 elements. When implementing the FNO method that requires a uniform rectangular grid, the functions are interpolated to [0, 1|2 using uniform grid of size 101 x 101. Figure 5 shows an example of input and output.
Surface traction
Stress field
biX
-30
Figure 5: Structural mechanics Equation
13 Left to Right: discretization of the domain, surface traction
Ωi corresponding von Mises stress field.
10

---

Advection equation Following the setup described in de Hoop et al. (0022) we consider the advection equation defined in the one-dimensional torus = [0, 1) = R/2,
Ou du =0
0t c
(a, D (0, T)
dr
(14)
w(0,t) = u u(,,t), w(x, 0) uo(s),
E [0,T] x x
with constant advection speed c 1 and periodic boundary conditions. The initial condition is "o = sign(E) where sign R - [-1,1] C is the sign function and ç D -> R is sampled from the Gaussian random field ç ~ (0, with covariance operator I = (-A+9) where A is the Laplacian the domain D with periodic boundaries. We are interested in approximating the mapping from imitial condition (z( to solution u(x; 7) at time T= 0.5. We use a dataset identical to the one used de Hoop et al. (2022) for training and testing. Figure shows an example of input condition un D + and output solution u(z,T)
U
ut, T)
05-
-05
55
0
0.3
Figure 6: Advection Equation 14 Left: initial condition uo (a) Right: solution u(x, T)
Poisson equation Consider the twoo-dimensional Poisson equation defined on triangular domain D [0, 13 with a notch, as described in Lu et al. (2022) and depicted in Figure Tt reads
(a) f, D D,
(15)
with pressure field h: D - R and constant source term f-1 The region of the notch is denoted by Ω and the condition h(æ)ladnea = is imposed. We are interested in approximating the mapping from the boundary condition aaD to the internal pressure field h(x) p. The input boundary conditions ((æ))aDY need to specified on the triangle 0D aΩ. As Lu et al. (022),, on each side of this triangle, the conditions are sampled from a ono-dimensional centered Gaussian Process (GP) with RBF covariance kernel R(x,, v) = expl-(x y)² ()2/(((2) with lengthscale é 0.2. The internal pressure field h(x)) for E D is obtained by solving Equation 115 using triangular mesh with 2295 nodes and 1082 elements, represented in Figure 7 Since the FNO method requires uniform rectangular grid, functions extrapolated to [0, 1]a using a 101 101 grid and linear interpolation when outside of D.The reader is referred to Lu et al. (2022) for details. Figure 7 shows an example of a boundary condition and associated internal pressure field.
4 Influence of the hyperparameters
This section investigates the influence of the number of channels C' 1 as well as the dimension K 2 1 when using the GIT-Net approach for solving PDE problems introduced Section 3 The number 2 1 of GIT layers is fixed at = and we used C (2, 4,8, 8, 16, 16,32) and K (16, 64, 128, 256, 512]. The number of PCA basis functions set by ensuring that 99.999% of the energy preserved. If this number
11

---

Boundary
Pressure field
Figure 7: Poisson equation 15 in a triangular domain with a notch. Left to right are discrotization of the domain D; boundary condition h(x) aayans corresponding internal pressure field h(x)).
of PCA basis functions is larger than P 200, it truncated at P 200 in order prevent the model from becoming excessively large. For the Navier-Stokes equation, Helmholtz equation, structural mechanics equation, advection equation, and Poisson equation, the number of PCA basis functions used as input-oulput in the network are 200-200, 200-200, 28-200, 198-198, and 16-16, respectively. To evaluate the performance of the different methods, four training datasets of respective size Ntrain E {2500, 5000, 10000, 20000} were generated for training; the methods were evaluated on independent test sets. The GELU nonlinear activation
function (Hendrycks Gimpel 2016) was used to implement the GIT-Net. Let Nurain N Figure
reports the relative test error defined as
Neaet [[N(f.) 9ill2
1 Ntest
Evest
(16)
119.112
a function of the size of the training set, well the number of channels C 2 and the dimension K 2 1. In the small dataset regime Nuain 2500, there is overfitting for all PDE problems, with particularly severe overfitting in the Helmholtz equation and Poisson equation. In general, smaller value of C leads to more overfitting when increasing K, as seen in the cases of the advection and Helmholtz equation with Nerain = 2500 training data. As expected, overfitting as a function of the dimensional parameter K 2 1 is decreased as the size of the training size is increased. For the Navier-Stokes problem, the dimension appears to play a more important role than the number of channels C. For K 16 the relative test error is insensitive the number of channels. It only when K 2 64 that increasing the number of channels C helps to decrease the relative test errors. However, for the Poisson equation, the number of channels C appears to play a more important role. The largest value of C, i.e., C' = 32, corresponds to the smallest relative test error even for the smallest value of K, i.e., K 16.
5 Comparison with existing neural network operators
This section compares the GIT-Net architecture three recently proposed baseline approaches: the PCA- Net architecture [Li et al. 2021 Bhattacharya et al.] 2021 de Hoop et al., 2022). the POD-DecpONet architecture (Lu et al. 2022 and the Fourier neural operator (FNO) approach (Li al. 2021 These neural architectures are briefly summarized in Section 5.1 for completeness. Various hyperparameters were tested for each one of these methods and Section 5.2 reports the predictions corresponding to the hyperparameters with the smallest relative test errors. Additionally, in order to demonstrate the computational efficiency of
the GTT-Net architecture, Section 5.2 compares the evaluation cost of these operators in terms of floating point operations.
12

---

C=2 4
C
1
C
2500
5000
10000
20000
Nawier-ttokke
10
10
10
10
Helmholtz
20
10
Strucural mechanics
Test T Error
10-3
Advection
0.1
Poisson
10
16
64
128
256 512
16
64 128 256 512 16
64 128
256 512 16
64 128
256 512
Network width K
Network width K
Network width
Network width K
Figure 8: Relative test eπror as defined in Equation 16
5.1 Neural architectures for operator learning
This section briefly summarizes the PCA-Net architecture, the POD-DeepONet architecture as well as the Fourier neural operator (FNO) approach. These neural architectures have recently been proposed for ap- proximating infnitteddimensional operators. As described in Section we consider two Hilbert spaces of
13

---

functions 4 - f: Sz - Rdu and V g Ω. + R defined on the input domain Gy and output domain P2 respectively. We also consider a finite training dataset (f Neran of pairs of functions 9: = F(f)) The unknown operator F F:ll - V is to be approximated by a neural network N U - V trained by empirical risk minimization, as described in Equation
In order to construct the PCA-Net architecture, consider the eigen-decomposition of the empir- ical covariance operator of the training set of functions (ff) Maran and (g.) Mrrin. This allows one to consider truncated orthonormal PCA basis ( CuN.) of Uf well truncated PCA basis (ee,1.---s Cu,N.) of V.
PCA-Net:
The projection operator APCA u + RN onto the PCA basis is defined as APCA (f) ((f, eu.1).... . (f, eu,N,))- Similarly, define the linear operator PrCa : TR + V as Prca(3) = B +
... + ev.. With these definitions, the PCA-Net architecture defined as
NPCA = PPCA o MLP a APCA
(17)
where MLP R.. RN. is a standard multilayer perceptron, i.e., fully L-layer connected feedforward neural network, with hidden layers of width C. Because the PCA bases are fixed, the computational cost of the PCA-Net architecture is independent of the discretization of the PDE. Furthermore, the PCA-Net method can readily be applied to PDE problems defined in irregular domains.
The DeepONet architecture (Lu et al. 2021 encodes the input function space in the branch network and the domain of output functions in the trunk network, respectively. Specifically, for an input function, f E U, the branch network B 4 + Rp takes the discretized input function f as input and outputs a vector {ff) E RR. For y E S2p the trunk network T S2 -> R.P.dout takes the coordinates of the output function as input and outputs a vector T(y) g.ddou, The vanilla unstacked DeepONet then defined as
Net-DeepONet:
P Σ +6 + 60. ==1
NDeepONer (DeepONet(f)()))
(18)
for bias vector by E Rdoa, The architecture of the branch network depends on the structure of the problem and is typically chosen as CNN or an RNN or an MLP. The trunk network is typically a standard MLP.
The POD-DeepONel architecture, an exlension of the vanilla DeepONet, was proposed Lu et al. (2022) The FOD-DeepOON use precomputed proper orthogonal decomposition (POD) basis functions instead of trunk net. It is defined as
P NPODD Σ B.(f) +
(19)
where B : 4 + R.. denotes standard branch net as used in the vanilla DeepOnet architecture and (eI,...,er) are P 2 precomputed basis functions of V that are generated from training data and ep is the mean function. The experiments presented in Lu et al. (2022) suggest that the POD-DeepONet archi- tecture outperforms the vanilla DeepONet architecture for wide class of PDE problems. In our simulations, following Lu et al. (2022). the combination of CNN with C channels and MLP of width K used as the branch network.
Fourier neural operator (FNO) For input function U, the FNO architecture Li al. 202 is defined as
NENO (f) @ o Lu .... L1 P(f).
(0
The lift operator P and projection operator @ are defined as
P()) PS + bp, and
@(ft) Qfr + bb,,
14

---

with vectors bp E RC and be E RR and matrices P E RCdd and Q E R.u... The Fourier layers L,,..., Ly in Equation 20 are defined as
fi(z)
=o - + ((--)))))))) (fi-i)))
(21)
for matrix Wi E RCC and (s(s) E CCxC for each frequency 8. The notations & and I denote the Fourier transform and its inverse. In Equation 21 denotes nonlinear activation function except in the last layer F where a is the identity function.
5.2 Performance comparisons
This section evaluates the performance of GIT-Net on the PDE problems introduced in Scction when compared to the baseline methods described in Section 5.1
Configurations
To study the impact of the hyperparameters, the following settings were considered: FNO: C {2, 4, 8, 16,32):
PCA-Net: K E {16, 64, 128, 256, 512, 1024, 2048, 8096)
(H) POD-DeepOnet: (C, K) E {(2, 16), (4, 64), (8, 128), (16, 256), (32, 512)];
GIT-Net: CE E (2, 8,,, 8, 16, 32), K E (16, 64, 128, 256, 512].
Following the setup of the Fourier Neural Operator (FNO) in de Hoop al (0022). we used twelve Fourier modes and three Fourier Neural Layers (L = 3 in 20 For the PCA-Net, as in de Hoop et al. (2022)
we use four internal layers (4-layer MLP in 17.). For the POD-DeepONet, following the recommendations
of Lu et al. (2022), we use two-dimensional convolutional neural network and fully connected layers as branch nets for the two-dimensional problems, and a fully connected network as the branch net for one- dimensional problems. If the input of two-dimensional PDE is one-dimensional boundary condition, it dimensions before being fed into the convolutional neural network. We test various
is first expanded pairs of the number of channels C. of the convolutional layers and the width of the fully connected layers K. The activation function used is the GELU function in FNO and the RELU function in PCA-Net and VOD-DeepONet.
We employed the aforementioned configurations to assess and compare the neural network operators based on their test error, error profile, and evaluation cost. These metrics collectively evaluate various aspects of the operators' performance.
Test error compares the relative test errors defined in Equation 16 For this comparison, we only show results corresponding the hyperparameters (C, K) {(2, 16), (4, 64), (8, 128), (16, 256), (32, 512)} for GIT-Net as representatives. In the large data regime Ntrain 20, 000, GIT-Net consistently achieves the smallest test error for all tested PDE problems. Furthermore, the FNO and GIT-Net architecture outperforms other methods for PDE problems defined on rectangular grids. For these simple geometries and in the small data regimes, FNO performs slightly better than GIT-Net, with a lower test error and less overfitting. However, FNO does not behave as well for more complex geometrics. mechanics equation, FNO obtains similar lest. errors to POD-DeepONel and GIT-Net. For the Poisson equation, the test error of FNO significantly larger than that of the other methods. The PCA-Net and POD-DeepONet architectures achieve similar test errors for problems defined on reetangular grids. In these cases, PCA-Net can be seen a POD-DeepONet with fully connected neural network as branch net. However, for problems defined on more complex geometries, the PCA-Net performs better for the structural mechanics equation, but worse for the Poisson equation when compared to the POD-DeepONet. Note that when C=11, 1, the PCA-Net and GIT-Net architectures are equivalent. This remark is confirmed by Figure g which indicates that when = 2 and K small (e.g. 16), the performance of the GTT-Net and PCA-Net are similar.
the structural
15

---

-O-
N 2500
-a- 5
N 10000
20000
PCA
PODD
FNO
GIT
Navier Navier-Stokees Strokes
10
20
10
10
Helmholzz
2
Structural mechanics
10
Test Error
2
Advection
0.2
0.1
Poisson
10
16
128
512
2048
B192
(2.16)
(4,64)
(8.128)
(16.266)
(325122)
8
16
32
(2.18)
(4.64)
(8.128)
(16,250
(3,512)
Network channel
Network width
Network channel-wior witth (C.K)
Network channel- width (C,
Figure 9: Test error of neural network operators for all PDE problems.
Error profiles Figure illustrates the average prediction error on the test data. To gain further insights, the statistical distribution of the test errors and profiles of median-crror cases and worst-crror cases are provided. To demonstrate the best performance that. these network operators can achieve, we selected the predictions and error profiles corresponding to the hyperparameters C. and K that achieve the minimum
16

---

test еттоr on Nurain 20, 000 data samples. The procedure for selecting hyperparameters and cases which is denoted by (S) is as follows.
Train network: For each PDE problem, train the networks of architecture PCA-Net, POD- DeepOnet, FNO, and GIT-Net with 20, 000 training data, where the used hyperparameter sets in (H) are denoted by (h Ik-1, {he [hPODD]S Ik=1, {he (
( Ih-i. and {hg Ik= respectively.
Choose hyperparameters with minimum test errors: Test these architectures with another 20, 000 test data get the test error {el {epoAja j==1 fer {ePODD)E 1k=1, {cr - and fe for all lested hyperparameters. For each archilecture, we choose the hyperparameter sets that minimize the test. errors which denoted by ACCA, MPODD, ANO and h Here, the test error means the average over all test data.
3. Choose input-output image pairs and their error image: For these architectures with selected ANO, and herr, show the inputs, outputs, and predictions with hyperparameters APCA, APODD, Wis
largest test error and median test error among 20000 test data.
We choose hyperparameters for all neural network operators according to Step 2 above and show their test error distribution oigton on 20000 lest data in Figure 10 13
19 and 22 Then according to Step 3, 16
the images of median-error and worst-error cases, including their input, output, prediction, and error images, are shown in Figure
12
15
18
21
and 24 The comparison of crror profiles for the same sample are shown
in Figure
14
I7
20
and
Figure 10 shows histogram of test errors of selected models 20000 test data for Navier-Stokes equation where used hyperparameters of cach neural network operator are
APCA [K= 512], Ni PODD {(C, K) (32, 512)]; hkk {C 32), M4
GIT = ((C,K)=((12)) (512)).
Figure 11 shows sample of test error, including input, output, and predictions of all neural network operators and corresponding error images. Figure 12 shows the median-error cases and worst-error cases. It can be seen from the histogram 10 that GIT-Net and FNO are significantly better than the other two methods. At the same time, the upper limit of the test error of GIT-Net is smaller than that of FNO, but the lower limit is similar. The test error interval of GIT-Net is more concentratod than that of FNO. From the error profiles of the median-error cases, it can be seen that FNO and GIT-Net achieve much smaller test errors than the other methods. Moreover, the error profiles of FNO and GIT-Net exhibit analogous structures for the worst error cases, indicating the presence of similar function approximators in both scenarios. This observation is further supported by the example shown in Figure
Figure 13 shows a histogram of test errors of selected models on 20000 test data for Helmholtz equation 12 where used hyperparamotors of each neural network operator are
PCA = {K- = 512), MOODD {(C, K) = (16, 256)). , ANOO = {C = 32}, n { K) = (32,512)) 512)]. hk
shows an example and Figure 15 shows the median-error cases and worst-crror cases. From the histogram, we can see that the lower limit of the test error of GIT-Net is slightly smaller than that of FNO, and the upper limit is almost the same, but the test error interval of FNO is more concentrated. Not only in Figure 13 but also in Figure 15 FNO and GIT-Net showed significantly better predictions. Meanwhile, their error images of median-erron cases and worst-crror cases had similar textures and intensity.
Figure 14
Figure 16 shows a histogram of test errors of the selected model on 20000 test data for structural mechanics problem where used hyperparameters of each neural network operator are
GIT (C, (32,512)) PODD_ (CC (32, 512)) ENO {C 32), his PCA [K 256), like hk1
Figure shows an example and Figure 18 shows the median-error cases and worst-error cases. From the histogram, the test error ranges of POD-DeepOnet and FNO are similar, but FNO more concentrated.
17

---

1400
POOD
FNO
1200
GFT
1000
800
600
400
200
Test
Figure 10: Histogram of test errors of Navier-Stokes equation
PCA
PODD
FNO
GIT
0.02
Prediction
0.1
Input
-0.1
-0.02
0.1
Output
Error
10
10 4
10
-0.1
10
Figure 11: An example of Navier-Stokes, including input, output, predictions, and error images of neural network operators.
GIT-Net has the smallest upper and lower bounds, and PCA-Net's is the largest In the median-error case, the error profiles of the GIT-Net and POD-DeepONet have similar intensitics, while the error profile of the FNO exhibits more severe oscillation structures, particularly near the hole, which is likely due to the interpolation triangular meshes onto rectangular meshes. Tn the worst-error case, the PCA-Net obtains the largest error, while the intensity of the error image of GIT-Net is slightly smaller than FNO and POD- DeepONet. The error profiles of all methods show the most violent oscillations near the top boundary, where the traction force applied, leading to more complex changes in stress in the output space.
Figure 19 shows histogram of test errors of the selected models on 20000 test data for advection equation where used hyperparameters of each neural network operator are
14
{K- 256)), 1 PODD {(C, (16, 256)) PNO {C 16), hi
GIT (CC,K) (16, 256)).
18

---

PCA
PODD
FNO
GIT
PCA
PODD
FNO
GIT
0.01
Input
Input
0..11
Output
Output
Prediction
Error Prediction
0
4.1
3013
10
Error
10.9
10
20
10
20
Figure 12: Input, output, and prediction of Navier-Stokes equation using the hyperparameters selected by (S) on 20000 testing data for each neural network. Left: the case with median test error. Right: the case with the largest test error.
600
POOD
FNO
500
GIT
300
200
Test
Figure 13: Histogram of test errors Helmholtz equation
Figure 20 shows example and Figure 21 shows the median-crror cases and worst-crror cases. As can be seen from the histogram,, the mean values and ranges of the lest errors of all neural network operators are similar, but the distribution of FNO is more concentrated. From the median-error case, it can be seen that all operators produce visually good predictions, including sharp jump points. In the worst-error cases, GIT-Net is obscrved to predict jump points much more accurately than the other operators. This consistent with the observation from the histogram that the test. error upper limit of GIT-Net is smaller than that of FNO. This may be due to the fact that FNO is more inclined fit smooth, low-frequency periodic functions, and
19

---

PCA
PODD
FNO
GIT
0.1
Prediction
Input
20
0
19.5
-0.1
0.1
Output
10 10-3
Erro
10
10
-0.1
10
Figure 14: An example of Helmholtz equation, including input, output, predictions, and error images of neural network operators.
PCA
PODD
FNO
GIT
PCA
DeepOnet
FNO
GIT
Input
20.2
Input
20.1
19.9
0.04
Output
0.02
Output
0
-0.02
0.05
Prediction
0.04
Error Prediction
0.05
0.02
-0.02
4.05
10
10a
Error
20
10
20
10
10
Figure 15: Input, output, and prediction of Helmholtz equation using the hyperparameters selected by (S) 20000 testing data for each neural network. Left: the case with median test error. Right: the case with the largest test error.
thus its ability to fit the jump points of piecewise linear functions is limited by the number of Fourier modes used.
Figure shows histogram of test errors of selected models on 20000 data for Poisson equation 15 where used hyperparameters of each neural network operator are
h {K 64), h
PODD {(C, K) (16, 256)), FNO (C = 32). GIT ((C, (32, 512)).
Figure 23 shows an example and Figure 24 shows the median-error cases and worst-error cases. Since the test error ranges of PCA-Net and GIT-Net almost coincide, in order to more clearly represent the difference
20

---

400
POOD
350
FNO
GIT
300
250
200
150
100
50
Test error
Figure 16: Histogram of test errors of structural mechanics equation
PCA
PODD
FNO
GIT
200
200
Prediction
Input
100
200
200
10
Output
Error
100
10
10
Figure 17: An example of structural mechanics equation, including input, output, predictions, and error images of neural network operators.
between the two, the vertical axis in the histogram is set to logarithmic scale. It be observed that although the mean of test error of GIT-Net is very close to that of PCA-Net, GIT-Net is more concentrated, and its upper bound much smaller than that of PCA-Net. In the median-error case, the PCA-Net and GIT-Net achieve better predictions than the other methods. The areas with large errors tend to be where the output values are large. In the worst-error case, GIT-Net performs better than the other methods. Although the PCA-Net method obtains smaller average of test errors, it performs slightly worse than the POD-DeepOnel method in the worst-error case, which indicates that the test error variance of PCA-Net is larger, which is consistent with the histogram. The error profile obtained by the FNO method shows severe oscillations, which is likely due to interpolation error.
Evaluation cost The approximate PDE operators discussed in this text are especially useful in situations such as Bayesian Inverse Problems when necessary to repeatedly compute solutions to PDFs. As result, their evaluation costs are an important consideration. In de Hoop al. (0022), a cost analysis based on
21

---

PCA
PODD
FNO
GIT
PCA
DeepOnet
FNO
GIT
200
MW mw M
Input
Input
-200
600
100
Output
Output
60
200
20
600
Prediction
Prediction
80
200
40
101
Error
Error
1017
10
10%
Figure 18: Input, output, and prediction of structural mechanics equation using the hyperparameters selected by (S) on 20000 testing data for each neural network. Left: the case with median test error. Right: the case with the largest test error.
2500
P00D
FNO
6IT
2000
1500
1000
Testerror
Figure 19: Histogram of test errors of advection equation
foating-point operations was provided. This analysis showed that, assuming the number of sampling points for the input and output functions is O(N)), the evaluation costs for PCA-Net and FNO scale as O(N, + K2) and OCNN, log(N, + N2CC),, respectively. The cost scaling of POD-DeepONet is O(N,C2 + CKN, N₂) for two-dimensional problems and O(N, + k2) for one-dimensional problems. In contrast, the cost scaling of GIT-Net is O(N, CK(C + K)). Figure 25 reports the evaluation costs as a function of the test errors for of the neural network operators considered this study and for four different. amounts of training data.
22

---

PODD
FNO
GIT
Prediction
Input
10
Output
Error
10
10
0.5
0
0.5
0.5
1
Figure 20: An example of advection equation, including input, output, predictions, and error images of neural network operators.
PODD
FNO
GIT
PCA
PODD
FNO
Input
Input
Output
Output
Predictin
Prediction
10"
109
Error
10:2
Eraor
20
0.5
0.5
0.5
0.5
0.5
0.5
2.5
0.5
Figure 21: Imput, output, and prediction advection equation using the hyperparameters selected by (S) on 20000 testing data for each neural network. Left: the case with modian test error. Right: the case with the largest test error.
For the similar evaluation costs when sufficient amount of training data is used (c.g. Nerain 20, 000). However, when the amount of training data is small (e.g. Nuraia 2500), GIT-Net exhibits more pronounced overfitting when compared to FNO. PCA-Net and POD-DeepONet show even more severe overfitting in this regime. For the Helmholtz equation and structural mechanics problems, GIT-Net achieves much smaller test errors than the other methods the same evaluation cost. For the advection equation, FNO can achieve smaller test errors than the other methods low evaluation costs, but GIT-Net performs better the cost. increases. For the Poisson equation, FNO performs the worst, with the largest evaluation cost and largest error. Overall, FNO and GIT-Net outperform PCA-Net and POD-DeepONet problems defined rectangular grids and can achieve similar test errors. However, FNO exhibits less overfitting on the Navier-Stokes equation with a small amount training data. GIT-Net achieves similar or better test errors than FNO at similar or lower evaluation cost. For problems defined on complex geometries, the FNO architecture exhibits much larger Lest errors than the GIT-Net approach.
equation, GIT-Net and FNO achieve the best performance and similar test errors with
6 Conclusion
We present GIT-Net, neural network operator designed for the approximation of partial differential equation (PDE) operators. Our data-driven approach learns an integral transform from paired input-output functions.
23

---

P0O0
6IT
102
10
15a
Test error
Figure 22: Histogram of test errors of Poisson equation
PCA
PODD
FNO
GIT
Prediction
Input
-1
-1
10
Output
Error
10
10.5
10-6
Figure 23: An example of Poisson equation, including input, output, predictions, and error images of neural network operators.
GIT-Net exhibits robustness to mesh discretizations and can be readily implemented for PDE problems on complex geometries or when the input and output functions are defined on disparate domains.
We demonstrate the efficacy of GIT-Net across variety of PDE problems of varying dimensions and ge- ometries. When compared with recently proposed operator learning methods, including PCA-Net, POD- DeepONet, and FNO, GIT-Nel consistently delivers the lowest (est error in the large data regime. For PDEs defined on rectangular grids, GIT-Net and FNO typically reach comparable accuracies, although GrT-Net typically operates at lower computational costs. Our experiments suggest that on more complex geometries, FNO not competitive when compared to GIt-Net. In terms of scalability with respect to the number of sampling points Np, the complexity of FNO and GIT-Net are OCNN, log(N,) + N₂C²) and O(N,+ + + CK3), respectively. These results suggest that GIT-Net has favorable properties for learning
24

---

PCA
PODD
FNO
GIT
PCA
PODD
FNO
GIT
Input
Input
Output
Output
Prediction
Prediction
2.10
10
Error
Error
10
10
Figure 24: Input, output, and prediction of Poisson equation using the hyperparametens selected by (S) 20000 testing data for each neural network. Left: the case with median test error. Right: the case with the largest test error.
PDE operators in wide range of settings. As part of ongoing work, establishing a thcoretical framework to elucidate these empirical observations.
References
Jonas Adler and Ozan Öktem. Solving ill-posed inverse problems using iterative deep neural networks. Inserse Problems, 33(12))124007, 2017.
Leah Bar and Nir Sochen. Unsupervised deep learning algorithm for pde-based forward and inverse problems. arXiv preprint arXit:1904.00. 05411, 2019.
Klaus-Jirgen Bathe. Finite element method. Wiley encyclopedia of computer science and engineering, pp.
1-12, 2007.
Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Prediction of aerodynamic flow fields using convolutional neural networks. Computational Mechanics, 64(2):525 545, 2019.
Kaushik Bhattacharya, Bamdad Hosscini, Nikola B. Kovachki, and Andrew M. Stuart. Model reduction and neural networks for parametric PDEs. SMAI Journal of Computational Mathematics, 7:121 157, 2021. doi: 10.5802//maaijem.774.
Jan Blochschmidt and Oliver G Ernst. Three ways solve partial differential equations with neural net- 200210000)) 2021.
works review. CAAMM-Milledeee
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miine and Qiao Zhang. JAX: compos- able transformations of Python-NumPPy programs, 2018. URL http:://gtthhi www. comm/oogge//ja.
Johannes Brandstetter, Daniel Worrall, and Max Welling, Message passing neural pde solvers. arXiv preprinl arXiv:2202.0332 2022.
25

---

PCA c-
PODD
FNO
* GIT
N 2500
5500
N 100000
=00000
Navier-S Stokks
10a
10
10
10a
Helmholtz
10
Structural mechanics
10
Test E Error
10
Advection
0.2
10
Poisson
10
10
10
10
10
10
10
10
10a
Evaluation cost
Evaluation
Evaluation cost
Evaluation
Figure 25: Evaluation cost vs test error of all methods for all problems.
Shuhao Cao. Choose a transformer: Fourier or galerkin. Advances in neural information processing systems, 34:24924-24940, 2021.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pp. 1251 1258, 2017.
26

---

Ingrid Daubechies, Ronald DeVore, Simon Foucart, Boris Hanin, and Guergana Petrova. Nonlinear approx- imation and (deep) relu networks. Constructive Approzimation, 55(1):127-172, 2022.
Maarten de Hoop, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew M. Stuart. The cost-accuracy trade- off in operator learning with neural networks. Journal of Machine Learning, 1(3):299 341, 2022. ISSN 2790-2048. doi: https:/ http://ooooor///// 10.4206/jmm.222255555 URL http:://global-sci..rrr/// detail/jm1/2130 html
Weinan E and Bing Yu. The deep ritz method: a deep learning-basee numerical algorithm for solving
variational problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
Weinan E, Chao Ma, and Lei Wu. Barron spaces and the compositional function spaces for neural network
models. Xiviv preprint arXi::1906.0880 08039, 2019.
Tarek A El Moselhy and Youssef M. Marzouk. Bayesian inference with optimal maps. Journal of Compula- tional Physics, (31(23)77815-77) 7850, 2012.
Robert Eymard, Thierry Gallouët, and Raphaèle Herbin. Finite volume methods. Handbook of numerical analysis, 7:713-1018, 2000.
Xiaoxiao Guo, Wei Li, and Francesco Torio. Convolutional neural networks for steady flow approximation. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 481-490, 2016.
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet- based operator learning for differential equa- tions. Advances in neural information processing systems, 34:24048- 2409422, 2021.
Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomnoss: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217 288, 2011.
Juncai He, Lin Li, Jinchao Xu, and Chunyue Zheng. Relu deep neural networks and linear finite elements.
Journal of Computational Mathematic, (8(3)5502-521 2020.
Dan Hendryeks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error 08415
linear units. CoRR, abs/ abs/1606.084415, 2016. URL
Jan S Hesthaven and Stefano Ubbiali. Non-intrusive reduced order modeling of nonlinear problems using neural networks. Journal of Computational Physics, 363:55-78, 2018.
Xiang Huang, Zhanhong Ye, Hongsheng Liu, Shi Ji, Zidong Wang, Kang Yang, Yang Li, Min Wang, Haotian Chu, Fan Yu, al. Meta-auto-decoder for solving parametric partial differential equations. Advances in Newral Information Processing Systems, 35:23426 23438, 2022.
George Em Karniadakis, loannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics- informed machine learning. Nature Reviews Physics, 3(6):422-44)) 2021.
Yuchaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric pde problems with artificial neural net- works. Furoopan Journal of Applied Mathemalics, 32(3):421 435, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Tn Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL TRRLhttp:// http:/t/t////..ortttttt aww//ass.....//
Georgios Kissas, Jacob Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George J Pappas, and Paris Perdikaris. Learning operators with coupled attention. Journal of Machine Learning Research, 23 (215):1 63, 2022.
27

---

Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art-No, 2021.
Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openrevies: ner/forummIdd<EPPPQS3E.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Avizzadenesheli, Burigede liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL httpps//opperreview.eee wttp:/iiwwweeeffrruoeeee formm? id CBP9NQVVmm
Tadeusz Liszka and Janusz Orkisz. The finite difference method at arbitrary irregular grids and its application in applied mechanics. Computers & Structures, 11(1-2):83-91 95, 1980.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data, In International Conference on Machine Learning, pp. 9208-3216 PMLR, 2018.
Zichao Long, Yiping Lu, and Bin Dong. Pde-net 2.0: hybrid deep network. Journal of Computational Physics, 399:108925, 2019.
Learning pdes from data with mumeric-symbolic
Lu Lu, Pongzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlincar operators via DeepONet based on the universal approximation theorem of operators. Noture Machine Intelligence, 3(3):218- 229, 2021. ISSN 25225839. doi: 11088//42255--001000000--
Tu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Fm Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Compuler Melhods in Applied Mechanics and Engineering, 393:114778, 2022.
Per-Gunnar Martinsson and Joel A Tropp. Randomized numerical linear algebra: Foundations and algo- rithms. Acla Numerica, 29:403-57 572, 2020.
Yong Zheng Ong, Zuowei Shen, and Haizhao Yang. Integral autoencoder network for discretiiatiooniinaiian learning. Journal of Machine Learning Research, 23(286):1 45, 2022.
Shaowu Pan and Karthik Duraisamy. Physics-informed probabilistic learning of linear embeddings of non- linear dynamics with guaranteed stability. SIAM Journal on Applied Dynamical Systems, 19(1):480 509, 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. infriee-nnformed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-7001 2019.
David E Rumelhart, GeofTrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Yeonjong Shin, Jerome Darbon, and George Em Karniadakis. On the convergence of physics informed neural networks for linear socond-order clliptic and parabolic type pdes. Communications in Computational OA-20000000-00001..... URL
Physics, 2845):0042 2074, 2020. ISSN 1991-7120. doi: html http://goobalssc...oo///ttoooooo
Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differ- ential equations. Journal of computational physics, 375:1339 1364, 2018
28

---

William S Slaughter. The linearized theory of elasticity. Springer Science & Business Media, 2012.
Jonathan Smith, Kamyar Arizzadenesheli, and Zachary E Ross. Eikonet: Solving the cikonal equation with deep neural networks. TEEE Transactions on Geoscience and Remole Sensing, 59(12):10685-100 10696, 2020.
Ralph C Smith. Uncertainty quantification: theory, implementation, and applications, volume 12. Siam, 2013.
Andrew M Stuart. Inverse problems: a bayesian perspective. Acta numerica, 19:451 559, 2010.
Fredi Tröltzsch. Optimal control of partial differential equations: theory, methods, and opplications, volume 112. American Mathematical Soc., 2010.
Yaohua Zang, Gang Bao, Xiaojing Ye, and Haomin Zhou. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics, 411:109409, 2020.
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder decoder networks for surrogate modeling and uncertainty quantification. Journal of Compulational Physics, 366:415-444 447, 2018.
A The effect of coordinates of grid as input in FNO method
Figure 26 compares the test error for FNO with and without grid coordinates as input. It found that the two results are similar except for advection. For the advection equation, the with-grid case obtains smaller test error in most hyperparanctors but except for C with 2500 training data, 2, for 5000 training data, and = 2 for 20000 training data.
B The comparison of (T + Ka) and T + (Kaa)
Figure 27 shows the comparison between GIT-Net with GIT mapping oTT Ka) and its variance in which the GIT mapping is replaced by Ta o(Ka). The tested hyper-parameters (C, K) {22, 16), (4, 64), (8, 128), (16, 256), (32, 512)}.
29

---

Without grid
With grid
N
2500
N 5000
N = 10000
N = 20000
10
Navier Strokes
10:2
0
10
10a
Helmholtz
2
Structural mechanics
10
Test Error
10
0.2
Advection
0.15
10
Poisson
10
32
32
2
32
32
Network
Network
Network
Network
Figure 26: Test error of FNO method for the with-grid case and the without without-grid case for all problems.
30

---

-0- a(Ta + Ko) -o- To + a(Ka)
N 2500
5000
N 10000
20000
Navier- Dovier-Strokes
10
10
10
10a
Helmholtz
2
10
Structural mechanics
Test Error
10
Advection
0.2
0.1
Poisson
10:3
10
(2.16)
(4,84)
(8.128)
(16.256)
(32,512)
(2.18)
(4.64)
(8, (8.28)
(16,256)
(32512)
(2.18)
(4.64)
(8.128)
(16,256)
(32512)
(2.16)
(4.64)
(8.128)
(16.2 (16,266)
(32512)
Channel- Wtth (C.K)
Channel- Width (C
Channel- Width (C
Channel-W With (C.K)
Figure 27: The comparison test errors between GIT-Net and its variance.
31


Reviews:
[{"review_id": "h9RrXJO3yc", "reviewer": "Reviewer_CRkN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- This paper studies an important and timely problem, on how to approximate solutions to PDE in an efficient and data-driven manner.\n- The detailed discussion of related works and approaches is highly appreciated, as it provides a comprehensive background on alternative methods.\n- The proposed approach compares favorably, alas mildly, with the alternatives.\n- The paper is quite clearly written, and the presentation is generally balanced\n\nWeaknesses\n\n- The modification presented in this manuscript, and that leads to the proposed GIT architecture, seems pretty minor with respect to existing alternatives. In particular, it is not completely clear to this reviewer what the specific changes are (see below).\n- While the authors have some heuristic motivations for the construction of their operator \\Kappa, little is mentioned about why the function \\sigma( T\\alpha + \\Kappa \\alpha), or their multiple compositions, might be doing to the obtain a solution as a function of \\alpha.\n- In my view, the presentation of the numerical results does not clearly present clear comparisons between the reported methods (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript proposes a new neural network architecture to learn operators between function spaces in the context of solutions to PDE. The methodology draws inspiration from both PCA-based recent alternatives (which allows them to be robust to the choice of the discretization mesh) as well as the Fourier Neural Operator approach (which seeks to find efficient representations for classes of functions). The authors demonstrate and benchmark their approach, dubbed GIT-Net, in 5 different problems and compare with relevant recent alternatives for data-driven Operator Learning techniques, studying the resulting approximation error and analyzing computational complexities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Main observations\n\n1. As far as I understand, the main novelty of this method is in the introduction of the GIT mapping (eq 8 and 9). While the authors have a descriptive motivation (but only high-level and heuristic) on why the chosen parametrization of \\Kappa might make sense, it is unclear why the authors then choose to do away with this by choosing the mapping (T+\\Kappa), for a linear operator T. The authors mention that this seems to help numerically from an optimization perspective, but to me this is a bit unsatisfying: If the operator T is so important for the good performance of the obtained trained model, is \\Kappa needed at all? Why couldn’t one simple define an operator \\tilde{T} = T+\\Kappa, and simply train \\tilde{T} from data? In other words, what is the added value to the heuristic motivations in page 4 and 5? I would like to see the authors discuss this in detail, and demonstrate that employing T+\\Kappa, instead of simply learning a linear operator \\tilde{T}, is beneficial.\n\n2. As a continuation of the above, it is a bit unclear how GIT-net differs from other approaches. In particular, the PCA-net approach composes two PCA-based transformations with an MLP (multi layer perceptron). But from the discussion above, there exist an operator \\tilde{T} = T + \\Kappa, and therefore their GIT-net also composes PCA transformations with an MLP. Therefore, it is unclear whether their improvement over PCA-net is simply because of a different parametrization that allows for -somehow- better optimization? The authors do mention that when C=1, then PCAnet and their method are equivalent, but in light of this -and respectfully- the novelty of the proposed approach seems quite limited - GIT-net simply allows for more channels?\n\n3. When comparing the numerical performance of the different methods, the authors first present the test error of the different approaches (for each of the different problems) as a function of some hyperparamter than, in each case, controls the size of the model. The authors seem want to make an argument for over-fitting (or lack thereof), but without incorporating statistics of the training error, it is impossible to make claims about overfitting. Can the authors add or comment on this? Moreover, the comparison among models is a bit strange, since each x-axis (for each column) is different and it is unclear how to relate each access to the other. Some natural way might be, say, the minimum number of parameters (as a function K, C, or combinations thereof) that make the training error zero.\n\n4. In comparing the error profiles achieved by different methods, the authors chose to present two cases per approach: the one with minimum error and the one with maximum error. I don’t find these comparisons useful, for two reasons: a) this leads to different samples for different methods, and b) it is sometimes unclear how relevant this distinction is, given that often the difference in error between the maximum and minimum cases is very very small/subtle. \nThis reviewer thinks it would be drastically more useful to i) provide error profiles for different methods for the same sample(s), thus allowing the reader to more easily compare across methods, and ii) provide full statistics for the distributions of those errors; i.e. Fig. 7 only provides the mean, and Figs 8-11 only provide illustration of max and min errors. It would be better to provide a histogram, or at least some illustration with confidence intervals, of the errors provided by each method. This would allow important conclusions, on whether e.g. these distributions are more or less concentrated.\n\n5. Fig. 13, which aims to reflect computational costs, is unclear to me: how is it that test error increases as more computations are used?\n\nMinor comments:\n\ni) authors might want to use more standard notation for the inner product in Eq. 13, such as < . , .>\n\nii) page 7: “In the numerical presented in..” → numerical experiments presented in?\n\niii) page 2: “furthermore” → Furthermore.", "broader_impact_concerns": "There's no Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QWPwQW9sOk", "reviewer": "Reviewer_gw94", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper conducts a comprehensive comparison in several forward PDE operator learning problems.\n- The architectural connection of the depthwise separable convolution and the combination of basis is interesting.\n- The application of PCA enables the model to handle problems on non-tensor-product-type grids nicely.\n\n### Weaknesses\n- The GIT stands for \"generalized ***integral*** transform\", but where is the ***integral***? There is no component in the neural network computing an integral transform. The analogy with a learnable integral transform is not clear at all.\n- PCA is used for the whole bulk data, i.e., a full SVD is done for a matrix of size `(N_data, N_discretization)`. The full SVD is very memory hungry! This defeats the purpose of introducing deep learning (using mini-batch to avoid operations in bulk, see also the requested changes below).\n- The extension to PCA-Net is somewhat limited theoretically.\n- The details in writing could use some proofreading and the presentation needs polishing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a new neural architecture called GIT-net. GIT-net is an operator learner that learns the PDE solution map under the standard end2end pipeline. One key ingredient is to treat both the channel/latent dimension and the discretization dimension as the coefficients of a \"basis\" (from PCA). The other draws inspiration from the depthwise separable convolution to create a learnable combination of different bases. Evaluation results are presented along with some other baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Requested changes\n- It would be nice to add a diagram marking the tensors' dimensions in Step 1-4 in Section 2.1.\n\n- It would be nice to see a comparison of (9) with $T\\boldsymbol{\\alpha} + \\sigma(\\mathcal{K}\\boldsymbol{\\alpha})$.\n\n- It is worthwhile to mention explicitly that $\\{b_k\\}$ comes from PCA's $V$ matrix (assuming the data has shape `(N_data, N_discretization)`) on page 6 as well as in the beginning of Section 2.1 (instead of saying just \"PCA bases\"), since there are two \"bases\" in PCA. Another thing is that, as a reader, I initially found the basis analogy in Section 2.1 somewhat puzzling until reaching the PCA part. \n\n- In all examples for GIT, `np.linalg.svd` is performed, and the spectral information of the data bulk is furthermore used to determine the number of bases (a hyper-parameter of the model). I understand that FNO does not need SVD to change basis, but I think this is somewhat unfair to FNO.\n\n- The evaluation cost on page 18 is somewhat unfair to FNO as PCA is involved in model reduction.\n\n- In FNO's code, for example, `FNO2d` module in the `model_ol.py` file, the lines ` grid = self.get_grid(x.shape, x.device)` and `x = torch.cat((x, grid), dim=-1)` have been commented out. Is FNO's benchmarks tested under this set-up??? If so, I think this is unfair for FNO. I understand that it is nontrivial to adapt this step for the triangular domain case, but for the square domain case, these lines should not be commented out.\n\n### Minor things that need to be clarified or fixed\n- When presenting the core architecture on page 5, sometimes bold-faced checked ${\\boldsymbol{\\alpha}}$ is used, while non-boldfaced is used as well, for example, \"$\\check{\\alpha}\\in \\mathbb{R}^{C,K}$\".\n- Page 4: \"Assume that each coordinate of the input function\", the \"each coordinate\" part is confusing. It should be \"each coordinate of the image of the input function\" or something similar.\n- Page 5: Section 2.1, bullet 2, the boldfaced $\\boldsymbol{\\alpha}$ is used, but it needs to be formally defined in bullet 1.\n- Page 5: Section 2.1, the author uses the term \"change of coordinate\" a lot, but the coordinate is not changed (every basis uses $\\boldsymbol{x}$ as coordinates). I think here the author meant to say \"change of basis\".\n- Page 5: After PCA, the neural network has to scale the input functions in the basis dimension (`d_width` in the `lift_d` part). Is this approach still make this neural network achieve invariance with regard to the discretization size?\n- Page 5: $\\mathcal{K}$ here is used as a linear operator, yet in the example section, it is used as the notation for the RBF kernel.\n- Page 6: the term \"functional mapping\" is confusing as functionals map a function to a scalar; please use \"operator\".\n- Page 10: if the permeability is a constant, then this is not modeling a porous medium. The BCs are not modeling in/out-flows either, thus the equation is simply a Poisson equation, not Darcy.\n- Figure 8-12: Can another set of figures be added maybe in the appendix such that the inputs to all models be the same? Another nitpicking is that, in the caption of the figures, it says \"smallest test error\", but later it says \"median test error\", a more precise description should be used. \n- Page 18: \"FNO method shows severe oscillations, which is likely due to interpolation error\". What is the \"interpolation error\" referred to here?\n- Page 20: I think \"favorable properties for learning PDE operators\" should be \"learning forward PDE operators\" as no inverse problem example is presented. The paper does mention the loss function for the Bayesian inverse problem on page 3, and I suggest deleting this presentation on page 3.\n- Missing references about designing neural networks for PDE operator learning:\n    - Gupta et al. Multiwavelet-based operator learning for differential equations. NeurIPS 2021.\n    - Cao. Choose a transformer: Fourier or galerkin. NeurIPS 2021.\n    - Kissas et al. Learning Operators with Coupled Attention. JMLR 2022.\n    - Ong et al. Integral autoencoder network for discretization-invariant learning. JMLR 2022.\n    - Brandstetter et al. Message passing neural PDE solvers. ICLR 2022.\n    - Huang et al. Meta-auto-decoder for solving parametric partial differential equations is in NeurIPS 2022.\n- On several occasions, \"i.e\" or \"i.e.\" should be \"i.e.,\".\n- Page 20: \"ongoing work\" -> \"an ongoing work\".", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rleQAKXd4R", "reviewer": "Reviewer_DXV5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In the following, I only conclude a several concerns I have. I hope the authors can address them.\n\nThe notation used in this paper is \"language-heavy.\" The overall takes are close to an image-to-image regression. The central proposal of the work is to use a block consisting of a skip connection and a factorized linear transform (pointwise linear transform and a depthwise linear transform, then pointwise linear transform again), followed by a GELU nonlinearity. The overall idea seems relatively straightforward. So I suggest the authors make an effort to simplify the notation. The language-heavy style makes such a simple idea look unnecessarily sophisticated.\n\nThe authors conclude that this method is better than FNO. However, insufficient understanding is provided. A MobileNet architecture would not necessarily perform better than a ResNet in terms of accuracy. The computation saving is expected since the architecture follows an efficient network design. The only surprising results are that some of the ablations seem to show that GIT-Net has less prediction error than FNO. But I think the authors need to discuss why. Otherwise, I would see this paper as parameter-tuning work.\n\nOn page 4, the first paragraph of Section 2.1, could the authors provide more evidence to support \"This factorization allows one to mitigate overfitting issues significantly\"?\n\nOn page 7, \n, is the cdot a typo?\n\nOn page 11, equation (16), why not stick to the previous notation introduced in the first paragraph of Section 2.2? \n is confusing.\n\nSection 5.2 - Error profiles are relatively uninformative; these only discuss what the error looks like. But why is the proposed architecture better? Why is this work not just parameter tunning? The overall setting of the work follows exactly FNO. So I think one can modify the neural architecture, tune the parameters to get better ablation results, and quickly write such a paper. Or many more papers like this. So please provide more insights.\n\nOn page 5, \"The main assumption of the GIT-Net transform is that the different frequencies do not interact with each other.\" This statement seems quite misleading. If we think depthwise linear transform follows this assumption, then the two pointwise linear transforms do not, right? Maybe I misunderstood. But now, this statement looks like a forced one, which is trying to provide an unsupported theoretical insight into an engineering design choice.\n\nSo overall, I don't think this paper is ready for publication. Anyone with FNO code could modify the neural architecture with new network architectures and sufficient hyperparameter tuning to write such a paper. So the authors should provide in-depth insights and understanding to ensure the work is deeper than what I just mentioned.\n\nFinally, I'm not an expert in this field. So if my understanding is significantly wrong, please ignore it, and I apologize in advance for the potentially naïve comments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "First, I'm not an expert in PDE operator learning, so my evaluation should be taken with a grain of salt. This paper proposes a neural architecture, GIT-Net, with residual blocks that use factorized linear transform inside. In some sense, GIT-Net compares to FNO, which is like MobileNet compares to ResNet. The ablation studies look promising. However, I also feel that the writing of this paper is language-heavy, and there is insufficient understanding provided to support the arguments.\n\nMy understanding of this work is that the authors propose a new neural network architecture, which is somewhat similar to MobileNet, and claim that the network achieve better empirical performance on PDE operator learning. No theoretical insights are provided and no understanding was provided at the moment. As I'm not an expert in PDE operator approximation, my review should be taken with a grain of salt. And I'm also not in a good position to evaluate whether the empirical advancement are convincing enough.\n\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition to my earlier comments, it is important to discuss what is to be learned from GIT-Net. My current take away is the following: GIT-Net is an efficient network architecture for doing PDE operator approximation. And it achieves better performance. This message is purely on the empirical side, so the authors should do a more thorough ablation study to show that GIT-Net can achieve better performance than all of the previous methods with a clear margin. And the code should be released to ensure that the results are reliable. And the paper should be significantly simplified. \n\nIf the authors wish to claim more on the theoretical insights, then much more works are needed to establish these arguments. I do not see any convincing insights in the current manuscript supported by convincing evidence.", "broader_impact_concerns": "I do not see any such concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work has been positively evaluated by three referees, who highlight the importance of the topic. That said, they all raise the fact that this work is particularly incremental with respect to the literature, so it is particularly important that the authors, in their final revision, make an extra effort to place this work in the literature and clarify in an objective manner the similarities and novelties.
Decision: Accept (poster)