Quantitative Approximation for Neural Operators in Nonlinear Parabolic Equations
Authors: Takashi Furuya
Koichi Taniguchi
Satoshi Okuda
Venue: ICLR 2025 Poster (2025)
Keywords: Neural operators
Partial differential equations
Nonlinear parabolic equations
Quantitative universal approximation
Primary area: learning theory

Abstract: Neural operators serve as universal approximators for general continuous operators. In this paper, we derive the approximation rate of solution operators for the nonlinear parabolic partial differential equations (PDEs), contributing to the quantitative approximation theorem for solution operators of nonlinear PDEs. Our results show that neural operators can efficiently approximate these solution operators without the exponential growth in model complexity, thus strengthening the theoretical foundation of neural operators. A key insight in our proof is to transfer PDEs into the corresponding integral equations via Duahamel's principle, and to leverage the similarity between neural operators and Picard’s iteration—a classical algorithm for solving PDEs. This approach is potentially generalizable beyond parabolic PDEs to a class of PDEs which can be solved by Picard's iteration.

Published as a conference paper at ICLR 2025
QUANTITATIVE APPROXIMATION FOR NEURAL OPER- ATORS IN NONLINEAR PARABOLIC EQUATIONS
Takashi Furuyal Koichi Taniguchi**
Satoshi Okuda
Doshisha University, takashi.. furuya0101 @emal..ma...............m......l com
Shizuooa University,
koichi 8shi zuoka ac.
3 RRikkyo University, okudase ri .kyo.as ac. jp These authors contributed equally to this work
ABSTRACT
Neural operators serve universal approximators for general continuous oper- ators. In this paper, we derive the approximation rate of solution operators for the nonlinear parabolic partial differential equations (PDEs), contributing the quantitative approximation theorem for solution operators of nonlinear PDEs. Our results show that neural operators can efficiently approximate these solution oper- ators without the exponential growth in model complexity, thus strengthening the theoretical foundation neural operators. A key insight in our proof is to transfer PDEs into the corresponding integral equations via Duahamel's principle, and to leverage the similarity between neural operators and Picard's iteration a classi- cal algorithm for solving PDEs. This approach is potentially generalizable beyond parabolic PDEs to a class of PDEs which can be solved by Picard's iteration.
INTRODUCTION
Neural operators have gained significant attention deep learning as an extension of traditional neural networks. While conventional neural networks are designed to learn mappings between liiee-dimensional spaces, neural operators extend this capability by learning mappings between dinensoadimeniionll function spaces. A key application of neural operators is in constructing sur- rogate models of solvers for partial differential equations (PDEs) by learning their solution opera- tors. Traditional PDE solvers often require substantial computational resources and time, especially when addressing problems with high dimensionality, nonlinearity, or complex boundary shape. In contrast, once trained, neural operators serve as surrogate models, providing significantly faster in- ference compared to traditional numerical solvers. Neural operators are recognized as universal approximators for general continuous operators. However, the theoretical understanding of their approximation capabilities, particularly as solvers for PDEs, is not yet fully developed.
This paper focuses on whether neural operators are suitable for approximating solution operators of PDEs, and which neural operator architectures might be effective for this purpose. The idea of this paper is to define neural operators by aligning them with Picard's iteration, a classical method for solving PDEs. Specifically, by associating each forward pass of the neural operator's layers with a Picard's iteration step, we hypothesize that increasing the number of layers would naturally lead to approximate solution of the PDE. We constructively prove a quantitative approximation theorem for solution operators of PDEs based on this idea. This theorem shows that, by appropriately selecting the basis functions within the neural operator, it is possible avoid the exponential growth in model complexity (also called "the curse of parametric complexity") that often arises in general operator approximation, thereby providing theoretical justification for the effectiveness of neural operators as PDE solvers
1.1 RELATED WORKS
Neural operators were introduced by Kovachki et al. (2023) as one of the operator learning methods, such DeepONet (Chen & Chen 1995 Lu al. 2019) and PCA-Net (Bhattacharya al. 202

---

Published as a conference paper at ICLR 2025
Various architectures have been proposed, including Graph Neural Operators (Li et al. 2020b),
Fourier Neural Operators (Li et al. 2020a). Wavelet Neural Operators (Tripura & Chakraborty
2023
Gupta et al., 202T). , Spherical Fourier Neural Operators (Bonev al. 2023), and Laplace
Neural Operators (Chen et al., 2024). These architectures have demonstrated empirical success as the surrogate models of simulators across wide range of PDEs, as benchmarked in Takamoto al.
(2022). In the case of parabolic PDEs, which are the focus of this paper, neural operators have also shown promising results, for example, the Burgers, Darcy, Navier-Stokes equations (Kovachki et al. )023). the KPP-Fisher equation (Takamoto et al., 2022). the Allen-Cahn equation (Tripura & Chakraborty 2023 Navaneeth et al. 2024), and the Nagumo equation (Navaneeth et al., 2024).
Universal approximation theorems of operator learning for general operators were established for neural operators (Kovachki et al. 2021
2023 Lanthaler et all]] 2023 Kratsios et al. (024), Deep-
Onet (Lu et al., 2019; Lanthaler et al. 2022). and PCA-net (Lanthaler) 2023). This indicates that
these learning methods possess the capabilities to approximate wide range of operators. However, operator learning for general operators suffers from "the curse of parametric complexity", where the number of learnable parameters exponentially grows as the desired approximation accuracy in- creases (Lanthaler & Stuart 2023).
A common approach to mitigating the curse of parametric complexity is to restrict general operators to the solution operators of PDEs. Recently, several quantitative approximation theorems have been established for the solution operators of specific PDEs without experiencing exponential growth in model complexity. For instance, Kovachki et al. 2 Lanthaler (2023) developed quantitative approximation theorems for the Darcy and Navier-Stokes equations using Fourier neural operators and PCA-Net, respectively. Additionally, Lanthaler & Stuart (2023) addressed the Hamilton-Jacobi equations with Hamilton-Jacobi neural operators. Further studies, such as CCen et al. (2023); Lan- thaler et al. (2022); Marcati & Schwab (2023), investigated quantitative approximation theorems using DeepONet for a range of PDEs, including elliptic, parabolic, and hyperbolic equations, while Deng et al. (2022) focused on dnvvectionddiffsiinn equations.
In line with these researches, the present paper also concentrates on restricting the learning opera- tor to the solution operators of specific PDEs, namely parabolic PDEs. However, unlike previous studies, this work leverages the similarity with the Picard's iteration in the framework of relatively general neural operator. This approach is potentially generalizable beyond parabolic PDEs to a range of other equations, including the Navier-Stoke equation, nonlinear dispersive equations, and nonlinear hyperbolic equations, which are solvable by Picard's 's iteration. See Kovachki et al. (2024) for other directions the quantitative approximation of operator learning.
1.2 OUR RESULTS AND CONTRIBUTIONS
In this paper, we present a quantitative approximation theorem for the solution operator of nonlinear parabolic PDEs using neural operators. Notably, in Theorem[]] we show that for any given accuracy, the depth and the number of neurons of the neural operators do not grow exponentially.
The proof relies on Banach's fixed point theorem. Under appropriate conditions, the solution to nonlinear PDEs can be expressed as the fixed point of contraction mapping, which can be im- plemented through Picard's iteration. This contraction mapping corresponds to an integral operator whose kernel is the Green function associated with the linear equation. By expanding the Green function in a certain basis and truncating the expansion, we approximate the contraction mapping using a layer of neural operator. In this framework, the forward propagation through the layers is interpreted as steps in Picard's iteration. Our approach does not heavily rely on the universality of neural networks. More specifically, we utilize their universality only approximate the nonlin- carity, which is represented as a one-dimensional nonlinear function, and hence, the approximation rates obtained in Theorem [ correspond to the one-dimensional approximation rates of universality. As result, our approach demonstrates that exponential growth in model complexity of our neural operators can be avoided.
1.3 NOTATION
We introduce the notation often used in this paper.
2

---

Published as a conference paper at ICLR 2025
For r E so], we write the Hölder conjugate of r as r': 1/r + 1/1/ = 1 i (1,00;;
r' = 0o if = =fr==1;r'
if = 20.
Let X be a set. For an operator A X + X and l N. we denote by A the times compositions of A (or the times products of A): A means the identity operator on X and A Aow.o.
times
Let X and Y be normed spaces with norm || IIx and || lly, respectively. For a linear operator A X + Y, we denote by ||Al|x-y the operator norm of A: ||||x--
sup][/(iI= =1 ||Aʃ[|r.
Let X be a normed space with norm || Ilx- We denote by (x(R) the closed ball in X with center 0 and radius R 0: Bx (R) = (f e X IIfllx s R}.
For q E [1, oo]. the Lebesgue space (D) defined by the set of all measurable functions
f = f() on D such that Le := (Sp (x))
< co if q * оo and ll ==
ess DED |/(x)] < co if q= cc. For r, E [1, oo). the space T,; (D)) is defined by the set of all measurable functions f = (1, (0. D such that
( == = {1, (
do) (L1((,)))
di ) < al
(with the usual modifications for 00 s oo).
For r,8 E oc), the notation ( means the dual pair of (2((D) and (D) or L (0, L (D)) and
(u, = / u(x))(x) dr or (u, S J"J, ult, u(t, dædt,
respectively.
2 LOCAL WELL-POSEDNESS FOR NONLINEAR PARABOLIC EQUATIONS
To begin with, we describe the problem setting of PDEs addressed in this paper. Let D be a bounded domain in Rdd with d E N. We consider the Cauchy problem for the following nonlinear parabolic PDEs:
[O.u Cu Lu Pw)) (u())= = uo
in (0, D, in D,
where T 0,0,0. 0 = 0/at denotes the time derivative, L is a certain operator (e.g. L. == - 82 /0x is the Laplacian). F R + I is a nonlinearity, (0,7) D - IR is solution un D - R is an initial data (prescribed function). It is important to note that boundary conditions are contained the operator C. this sense, the problem (P) can be viewed abstract initial boundary value problem on D. See Appendix A examples of L.
to (unknown function), and
As it a standard practice, we study the problem (P) via the integral formulation
u(t) = Sc(t)un + S. S2(t (((t)))F(u(T))dn
(P')
where (Sc(t))ezo is the semigroup generated by L (i.c. the solution operators of the linear equation 0,u + Lu = 0). The second term in the right hand side of (P') is commonly referred to as the Duhamel's integral. Under suitable conditions on and F, the problems (P) and (P') are equiva- lent if the function u is sufficiently smooth. For example, in the case where L A, it follows from the smoothing effect of Sc(t) that the solution to (P is a classical solution to (P) (see the argument in the proof of Brezis& & Cazenave (1996 Theorem 1)). Even in the case where L is more general operator satisfying Assumption I a similar argument can be also done by replacing the differentiation in with the operator L (together with of the techniques in the proofs of Iwabuchi et al. (202T) Lemmas 3.5 and 3. 3.10) for instance). The aim of this section is to state the results on local well-posedness (LWP) for (P" Here, the LWP means the existence of local in time
3

---

Published as a conference paper at ICLR 2025
solution, the uniqueness of the solution, and the continuous dependence on initial data. Its proof is based on the fixed point argument (or also called the contraction mapping argument). These results fundamental to study neural operator in Sections 3 and In particular, Propositions and below serve as guidelines for setting function spaces as the domain and range of neural operators in Definition (Section and for determining the norm measure the error in Theorem (Section
In this paper we impose the following assumptions on L and F.
Assumption 1. For any S 91 92 oo, there exists constant Cc > O such that
|| S Cct P - t t (0, 1],
for some O. Assumption 2. F E satisfies F(0) = 0 and
[F(z1) F(z2) CF max |z1| 1-11111121 #2),
(2)
for any 21, 22 E R and for some p > 1 and CF 0.
Remark 1. The range (0, 1] of in ① can be generalized (0, Tc), but it is assumed here to be (0, for simplicity. This generalization is not essential, as the existence time I of the solution is sufficiently small the fixed point argument later. Long time solutions are achieved by repeatedly using the solution operator of (P' constructed in the fixed point argument (see also Subsection 4.2
Remark 2. Typical examples of L are the Laplacian with the Dirichlet, Neumann, or Robin bound- ary condition, the Schrödinger operator, the elliptic operator, and the higher-order Laplacian. See Appendix for the details. On the other hand, typical examples of the nonlinearity F are F(u) | 10-2, --, I/u P (which can be regarded as the main term of the Taylor expansion of more general nonlinearity F if F is smooth in some extent). See Appendix for further remarks on Assumptions 7 and
Under the above assumptions, the problem P is local well-posed, where, as a solution space, we use the space L' LO, (D)) with the parameters r, satisfying
1
E [o] and
+
(3)
More precisely, we have the following result on LWP.
Proposition 1. Assume that r,8 satisfy (3. Then, for any uo E (D),
there exist time
T = T(w)) E (0, 1] and a unique solution u E (7 T; (D)) to (P Moreover, for any uo, to T (D), the solutions u. and v to (P') with u(0) uig and v(0)
vo satisfy the con-
tinuous dependence on initial data: There exists a constant C' > 0 such that
||u - ull & C'Iluo vo 1
where T' min (nin(T(uu)) (uo), (vv))-
The proof is based on the fixed point argument. Given an initial data uo E (D) and T, M > 0,
define the map & = 2ung by
Φ(ul(t) = Sc(t)uo + ' Se(t-r))(uu()) dr
(4)
for E [0, 7] and the complete metric space X = BL-10T:LLC (M) equipped with the metric
d(u, v) ; ||u u1|2-(0.7;2-))
Let M 0 be arbitrarily fixed, and let T > 0 (which is taken sufficiently small later). Then, under Assumptions [ and it can be proved that for any us E Br (lt), the map & X X is 5-contractive with contraction rate (0, 1),
d( u), Φ(v)) x 5d(u, v) for any u, u X
where T is taken small enough to depend R. M and 5 (not to depend uo itself). Therefore, Banach's fixed point theorem allows us to prove that there exists uniquely a function u E X such that ø/u] (a fixed point). This function is precisely the solution of (P') with the initial data w()) r4g Thus, Proposition is shown. See Appendix B for more details of the proof.
This proof by the fixed point argument guarantees the following result (see e.g. Zeidler (1986)).
4

---

Published as a conference paper at ICLR 2025
Proposition 2. Assume that r, satisfy (3) Then, for any R,M O and for any å E (0. 1), there exists a time T (0, depending on R,M and 6, such that the following statements hold:
There exists unique solution operator r BL= (R) + (Ir(o.T:to)(MM such that
r (uo) = u
for any uo E BL (R), where the solution with u(0) ug given in Proposition !
(ii) Given un B1 (R), define Picard's iteration by ( = Sc(t)uo,
ulet1) $[ul0]] =w(t) [ Scll (F) t= 1-1,,,.'
(5)
that is,
(0) = 0
00) 8= 1,1,,
times
where 4 : X X is 8-contraction mapping defined by (0, T; T. (D)) as l 0o and
Then w0)) in
8 d(u u) S 1-8 1- d() 0), = 1,2,
3 NEURAL OPERATOR FOR NONLINEAR PARABOLIC EQUATIONS
In this section, we aim to construct neural operators I' that serve accurate approximation models of the solution operators 1+ for nonlinear parabolic PDEs (P) We start by explaining our idea in rough form. Our idea is inspired by the fixed point argument and Picard's 's iteration. By Section [ for any un E B (R), the solution u to (P') on [0, T x can be obtained through the Picard's 's iteration (5) under appropriate settings. If the semigroup (t) has integral kernel G = G(t, z, y), which is the Green function of the linear equation 0,u u + Lu = 0, then we can write
Sc(()o(<)) / G(I, G(I,z,,)uolv) dy,
L Sc(t - FT.) dr 1/' G(t T, ((u dydr..
Suppose that G has expansion
G(t - T.
Σ M.REA
lim N
Σ
Cm. (m
m.nEAN
for 0 < T, < T and v, D. For convenience, always assume that G(t, v, 0 for t < 0. Here, A is an index set that is either finite or countably infinite, and Ay is subset of A with its cardinality |A N E N and the monotonicity AN C AN for any N < N'. We write the partial sum as
Gx(l - T, = Σ Cn (T. (l,
M.EAAN
for 0 S r,t E T and v. D. We always assume that Gx((,z,v) = 0 for t 0 as well. We
define NN by
(u(uu(t, = /
S. [.Jo
GN(I,
- F(u(T, dydr
Σ
Cm. (1 (0, (o),p
Σ
Cm. F F(u))p
m.nENN
M,NEAN
Σ
Cm.n (1 (0, -), uo) + (I'm F(u)))(())))) (t)),))),
m.nEAN
5

---

Published as a conference paper at ICLR 2025
and moreover, define approximate Picard's ' iteration by
i0)= = S 2))oo) dy
Σ M,NEAN
(1 un) (1,2))
((++1)= = (u L 1,
Then, for sufficiently large N, we expect:
1. n Re 4 and n is also contractive on XN == -Br10.TT:22 for some Tv > 0.
2. There exists fixed point á E Xw of &N such that a(() + ē 00.
3. á RO u on [0, min (T, NwHl x for any uo E BL (R).
4. Define : BL= (R) + N by ((wo) == (L)) for L E N. Then I' ~ I+t as N + co.
The above is the prototype of our neural operator. where N corresponds to the rank and L the layer depth. In other words, in our neural operator, the Picard's iteration step corresponds to the forward propagation in the layer direction of the neural operator, which converges to the fixed point (i.e. the solution to (P' as the layers get deeper for sufficiently large rank N.
The above is only a rough idea. The precise definition of our neural operator is the following:
Definition 1 (Neural operator). Let T > O and r, E [o). Let & = {puin and 1b == (vm hm be families of functions in L' (0, T; L (D)) and L'" (0,T;I1 LA (D)), respectively. We define neural operator - (D) + (0, L* by
T:
L" up + (L+1))
Here, the output function a(I++) is given by the following steps:
1. (Input layer) a(1) = (1( (1)
.22 ..... do is given by
(1)
i ( = (K (0)
N uo (t, x) 622 (t,
Here, K() (D) + (0, T; T (D)) and 1/ (0, T; I (D))& are defined by
(K ( = Σ
c(0) n.m (m. (0, :), Pn with n.m. E Rdix1
M,nEAN
6/ (t, == Σ
(0)) 2t)
with b/0 E ERd
nEAN
(Hidden layers) For 2 2 L, (0))
(ū) .ug
(P)
are iteratively given by
a (1, = (w(0)) (K (0) + bb()), -
(Output layer) (L+1) given by
(L+1) ((++)) w(L) (( + (t,)) +b(4)
Here, σ : - is a nonlinear activation operating element-wise, and W(t) E Rdee1xdee is a weight matrix of the l-th hidden layer, and b(t) E Rdde is bias vector, and K(C : (O T; (D)) I0 (O; L is defined by
(),,z)) = Σ
Oam (m (),) with C(t) E Rddi
M.nEAN
where we use the notation
(Wm, u) = ((mm: ui)...., (Wm, wd,)) ) E Rdd,,
for (u1, ......... Ud, (O, L. Note that di-1 = H.o No the class of neural operators defined as above, with the depth L, the number
We denote by L. of newrons H Ekn de, the rank N, the activation function a, and the families of functions up sp.

---

Published as a conference paper at ICLR 2025
The operators x play crucial role in capturing the non-local nature of PDEs. They defined by truncating the basis expansion, a definition for neural operators inspired by Lanthaler et al. (2023)
In this context, K are finite-rank operators with rank N. The model complexity is determined not only by the depth L and the number H of neurons, but also by the rank N. The families and e hyperparameters, which chosen that their expansions can approximate the Green function. Examples are the Fourier basis, wavelet basis, orthogonal polynomial, spherical harmonics, and eigenfunctions of When we select y and as the Fourier basis and wavelet basis, the resulting network architectures correspond to the Fourier neural operator (FNO) in Li et al. (2020a) and the wavelet neural operator (WNO) in Tripura & Chakraborty 2023 respectively. See also Appendix for FNOs and WNOs in some simple cases.
4 QUANTITATIVE APPROXIMATION THEOREM
In this section prove quantitative approximation theorem for our neural operator I (Defini- tion [. For this purpose, we assume that L and F satisfy Assumptions [ andd] respectively, and the parameters r, satisfy the condition (3). In addition, assume the following:
Assumption 3. Let A be index set that is either finite or countubly infinite and AN a subset of with its cardinality |Ax] = =N E and the monotonicity AN C AN for any I N'. For the Green function G of the linear equation d Lu O, there exist the families of functions = (nnT neE and db == (ymbmet in L' (0, (D)) and L' (0, (O,T; L' (D)), respectively, such that has the following expansions:
G(t (T,-),,,)==
Σ m.nEA
0< x
(6)
G(I,,,y)=
Σ M.REA
Co (0. 0< 2DDED
in the sense that
Eo(N) (1((--,,E,))) GN(l
- -aw-:ppppp--------slllarrarlrrr- 0
(N) || GN Gt, 1100r +0
as N + co, respectively, where
Gy (T,t)),
Σ M.
(n. 0 <
(8)
4.1 MAIN RESULT
Our main result is the following quantitative approximation theorem.
Theorem 1. Suppose that C and F satisfy Assumptions 7 and 2 respectively, and the parameters r, s satisfy the condition B Then for any R > there exists time T > such that the following statement holds under Assumption 3 For any (0, there exist a depth L, the number of
neurons H, rank N, and T E NO N.P.V
B. such that
[[ I'(w0) Se.
sup B (R)
Moreover, L = L(T') and H = H(') satisfy
L(T) < Cz 1)2 and H(T) I CHE "Heel((og(e11)
for some positive constants Cr, and Ch depending on v. M, F, F.D.p,r, R. and L, and N = N(T) satisfies
Ea(N(T)) Cae and (N()) < CGe for some positive constant Cg depending on u, M, F, D.p.r,C 8, and f.

---

Published as a conference paper at ICLR 2025
Let us give remark model complexity of our neural operator I in Theorem I The model com- plexity depends on the depth L(T), the number of neurons HT),, and the rank N. We observe that L(T) increases only squared logarithmically at most as - 0 and (I') increases on the order c-1 at most as 6 + 0. This is reasonable since each forward pass of the neural operator's layers corresponds to step in Picard's iteration, and the convergence rate of Picard's iteration decays expo- nentially (see Proposition 2 (ii). As to the rank N, Theorem doess not mention anything regarding the rates of the errors EG (N) and E% but these error rates are also necessary for quantitatively estimating the model complexity of T. These error rates strongly depend on the choice of şp and db. and the research results are enormous (see e.g. [Chen (200); ; DeVore & Popov (1988); De Vore al.
1993;; ; Düng (2) For example, in the case of Fourier bases on d-dimensional torus T it can be shown by Parseval's identity that the approximation rate in L is M[N-++4//) (a > d/2) if the target function belongs the L type Sobolev space Hº (T (see Dyachenko (9955 for
L (.--framework.. As another example, for many wavelet bases, the approximation rate O(N ([N) is known based on Jackson-typp type estimates (see Cohen (2003) Theorem 4.3.2)). Furthermore, there are cases where the rates differ between linear approximation and nonlinear approximation (wavelet approximation): in particular, when the target function is discountinuous (see Cohen (2003) Remark 4.2.5)). Therefore, if the Green function G has certain regularity, we may obtain quantitative estimate for the rank N such as the polynomial rate of (NV and (NV) and result, it is shown that the model complexity of I' does not grow exponentially. This prevents the exponential growth in model complexity, which often arises in general operator approximation.
Next, we remark that the constants CL. Cи and CG in Theorem I depend on D, in particular on the dimension d. According to the proof of Theorem (ecoorem[esspppppee (especially, Lemma 4 in Appendix D). the dependencies on D are C2 R 1, CH R |D|1/* and CG S 1 for sufficiently small > O. For simplicity, letting D [0, R]j, we see that the growth of C# with respect to d is Ch R Rd/s,
Sketch of proof. (See Appendix Dioor the exact proof). The proof is based on the following scheme:
Un Sc > (1)
+
(2)
4
+ + (... J u)
(Picard's iteration)
40
SC.N (1) NN. a No.
(1)
(Neural operator iteration)
where Sc, SC,N, L, and 4 Ninet are defined as
St/uol(t, = 1 G(t, x. v)voo(v)) dy, Se,(uol(t,x)= = GN dy,
+[u](t,x) = Sc/uollt. + 1/' G(t,z, v)) dydt,
2 ne W.neiiu](tta)) xt) SC.N (cov(uo((t, + S. v.)tt dydr.
First, by the truncation of Picard's ss iteration, the solution = (woo) to ( is approximated by u 22 (d)) -- c Sclun).
for sufficiently large J. Next, by approximating the Green function G and the nonlinearity F with a truncated expansion GN and a neural network Faet, respectively, the contraction mapping & is approximated by & Neet for sufficiently large N (Lemmas 2 and 3). leading to
(J) ()
o N.ne Sc,N(uo),
(Lemma Finally, we represent (()) as a neural operator I' (Lemma
The key idea is mimic the Picard's iteration. The convergence rate of Picard's iteration is given in Proposition 2 (ii), which reads J x log(c-1). By applying the one dimensional case of Gühring Fnet with the rates L(Fret) E (log(e")) and [[ net) X E (((o(((-)) Note that the curse of dimensionality that occurs in conventional neural -
al. 2020 Theorem 4.1), we approximate
network approximations does not appear here. Consequently, the rates of the depth L(T) and the
number of neurons HT)) are evaluated as
L(T) S L(Fnet) J Rd log 11 log((- (log(e -2)² H[') S ((Fee)) JJ SJSE" (log(E-!)) (log(E-l)) S -1 (log(e 2))²
8

---

Published as a conference paper at ICLR 2025
where the first inequality is given by the exact construction of the neural operator I discussed in the proof of Lemma
as
I(wo) W'
((0) (W + RN) Fnet o.... ( + RNN o Fnet (K + (uo),
(9)
J-1
where W and W are some weight matrices, Ky and K are some non-local operators, 6 is some bias function, and Fnee is some neural network so that L(Fnet) = L(Fret). Note that the operation (W + KN) Faet corresponds to the one operation NN,net, and it can be T.(Fnet) + layer neural operator. In this construction, when denoting by (W + -Kx) ) Free (W(r)(Paa))) (II(FF))+)) + 6( ((L((Fee)+1)
a oo o (W()) + K() + 60)) the local operators W(() and biases ((0) for 1, ..... (( correspond to those in the neural network Fnet. while the non- local operators (f)
for é 1...., ((Fnee) At the peF + 1-th layer. W[ W and ([L((net)+)) = 0, and the non-local operator K KN.
Remark 3. It is important to note that the hidden layers of the neural operator I' constructed in ( o o which significantly reduces
structured repeatedly apply the same operator, (W x memory consumption. Our constructed neural operator is closely related to the deep equilibrium models (Bai et al. 2019), which utilize weight-tied architectures to find fixed points. More recently, inspired by deep equilibrium models, weight-tied neural operators have been experimentally in- vestigated in the context of solving steady- state PDEs (Marwah et al. 2023). Based the above findings, our results also give weight-fied neural operators for cunsteady-state parabolic PDEs and theoretically guarantee their approximation ability.
Remark 4. One of the next issues in approximation theory is to verify in what sense the obtained approximate solution is good approximation. In particular, it is important to discuss that approximator I obtained in Theorem preserves desired properties in parabolic PDEs, such as the maximum principle and the comparison principle. In the special case of the nonlinearity, we can show that approximator I' preserves the positivity. For more details, see Appendix
4.2 APPROXIMATION OF LONG TIME SOLUTIONS
The problem of approximating global in time solutions using neural operators is more challenging and significant task. It is known that global solutions can be extended by repeatedly applying the solution operator I when the L - -orm of the solution remains uniformly bounded. Based on this idea, this section discusses the approximation of long time solutions using neural operators.
Extension of short time solution to a long time solution. By Proposition (i), for any initial data Mp E Br= (R), there exists a short time solution u = (*)))) to (P" 0. T] x D. Then, if (T) at T belongs to Br, the same solution operator + can be applied u(T) as an initial data, and hence, the solution can be extended to that on [0, 27] D. Thus, long as the supremum value of the solution is less than or equal to R, the solution can be extended with respect to by applying I+ many times possible. We denote by Umax the extended maximal solution with respect to t and by Kmax = Kmax( E {oo} the maximal number of extensions by I++ Then we write Umax as
Umax (t) = r [r]] (to)()
for (K - 1)T < KT and K = Kmax-
Approximation of the long time solution Umax by neural operator. Similarly, for neural operator I' given in Theorem[] [] the output function of I can be extended with respect to time / by applying many times as possible. We denote by úmax the extended maximal output function of I' with respect to t and by &max = Kmax((u) E NN U {oo} the maximal number of extensions by I. Then we can also write umax as
Amax (t) == (ug)
for (K 1)T < f < KT and k I. Riaax. Then the error between Umax and Úmax can be controlled by the error e Theorem I and another error ē given by
(n) (uo) (u))))

---

Published as a conference paper at ICLR 2025
More precisely, we have the following result on the error between umax and imax-
Corollary 1, Let E 10 and R > O. Suppose the same assumptions as in Theorem and n" := min Rmax is a finite number. Then there exists constant C' > 0 such that
|Imax (maxxIlrr & C(e+a) +
for any uo E BL (R), where T andd T are the same as in Theorem
The proof is given in Appendix H
DISCUSSION AND FUTURE WORK
Applicability and limitation. In this paper we focus on parabolic PDEs with power-type (or some- what general) nonlinear terms as a first step of study of neural operators based on Picard's iteration. However, we believe that our approach is restricted to parabolic PDEs and can be adapted to a broader class of nonlinear PDEs, because the Banach's fixed point theory is widely applicable to many nonlinear PDEs (see Appendix G for the details). On the other hand, for PDEs that in- volve nonlincarity in the principal part such as p-Laplacian equations dzu - Apu = F(u) with Ap = Vil and * 2, it may not be possible to represent them in the form integral equations. Such equations might not be effectively handled by our approach.
Further development of Theorem I. In this paper we discuss the LWP for (P) and the approx- imation error in Theorem [ in the framework of the Lebesgue spaces. However, we expect that these results can be generalized to those in the framework of Sobolev spaces. In fact, LWP can be established in the Sobolev space framework as shown in [Ribaud (1998). Further, it is also possible to discuss the approximation error in the same framework, since the basis expansion of the Green function and the approximation of nonlinearity by neural networks are studied in Sobolev spaces.
Next, in the results of this paper, only bounded domains D are considered due to technical reasons in the proofs. However, the question of whether Theorem Saarrem[[aaa Caan be extended to general domains, in- cluding unbounded ones, is an interesting problem (at least mathematically) and remains direction for future work.
Moreover, in this paper we discuss the problem in the setting of an abstract operator L and general bases sp, 1h. However, is also important to develop a more detailed analysis by restricting the setting to more specific operators and basis functions. For example, by focusing specific operators and basis functions, one can construct approximation theorems that include concrete rates for rank N, and it is erucial to investigate how these rates vary depending on the choice of operators and basis functions. This issue remains one of the directions for future work.
Implementation. This paper does not address the implementation of neural operators, comment here. Our proof based on Picard's iteration is constructive, and this constructive ap- proach may offer valuable insights for future experimental studies, particularly when incorporating constraints specific to the architecture of PDE tasks (for instance, weight-tied architecture discussed in Remark 5) Note that, due to the form of integral equation P our neural operators need to include the integral with respect to time, which may be computationally expensive. However, the previous work Kovachki et al. (2023) Section 7.3) employed the Fourier transforms with respect to time for computing FNOs, whose techniques might be useful for computing our neural operators. We leave the details of experimental studies of our theory the future works.
briefly
Neural operators with desirable properties. In numerical analysis, it is crucial that the solutions of discrete equations retain the properties of the original continuous solutions. Similarly, neural op- erators must also preserve these properties, especially when used surrogate models of simulators in practical applications. While we have shown positivity preserving a specific case of nonlinear- ity (Remark future work may extend this result to more general nonlinear settings. Additionally, other important properties, such as the comparison principle and the smoothing effect, which are essential in parabolic equations, will be the focus of future studies.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
T. Furuya is supported by JSPS KAKENHI Grant Number 1624K JST CREST JP- MJCR24QS, and JST ASPIRE JPMIAP2329, K. Taniguchi is supported by JSPS KAKENHI Grant Number 24K21316.
REFERENCES
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in neural information processing systems, 32, 2019.
Matania Ben-Artzi, Philippe Souplet, and Fred B Weissler. The local theory for viscous hamilton- jacobi equations in lebesgue spaces. Journal de Mathématiques Pures et Appliquées, 81(4):343- 378. 2002.
Kaushik Bhattacharya, Bamdad Hosseini, Nikola Kovachki, and Andrew M Stuart. Model reduc- tion and neural networks for parametric PDEs. The SMAI journal of computational mathematics, 7:121-157, 2021.
Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherical fourier neural operators: Learning stable dy- namics on the sphere. In International conference on machine learning, pp. 2806-2823. PMLR, 2023.
Haïm Brezis and Thierry Cazenave. A nonlinear heat equation with singular initial data. Journal D' A'Aallyse Mathematique, 68()::2773304 1996.
The Anh Bui, Piero D' Ancona, and Fabio Nicola. Sharp L estimates for Schrödinger groups on spaces of homogeneous type. Revista Matemática Iberoamericana, 36(2):455 484, 2020.
Thierry Cazenave. Semilinear Schrödinger Equations, volume 10. Courant Lect. Notes Math., Amer. Math. Soc., 2003.
Thierry Cazenave and Alain Haraux. An introduction to semilinear evolution equations, volume 13. Oxford University Press, 1998.
Gengxiang Chen, Xu Liu, Qinglu Meng. Lu Chen, Changqing Liu, and Yingguang Li. Learning neural operators on Riemannian manifolds. National Science Open, 300)20004 2024.
Ke Chen, Chunmei Wang, and Haizhao Yang. Deep operator learning lessens the curse of dimen- sionality for PDEs. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application dynamical systems. IEEE transactions on neural networks, 6(4):911-917 1995.
Jan W Cholewa and Anibal Rodriguez-Beraal On the Cahn- equation in H (RR)). Journal of Differential Equations, 253 (33(1)))367 2012.
Mourad Choulli, Laurent Kayser, and El-Maati Ouhabaz. Observations on Gaussian upper bounds for Neumann heat kernels. Bulletin of the Australian Mathematical Society, 92(3):429-453 2015.
Albert Cohen. Numerical analysis of wavelet methods. Elsevier, 2003.
Dinh Düng. Optimal adaptive sampling recovery. Advances in Computational Mathematics, 34: 1-41, 2011.
Edward Brian Davies. Heat kernels and spectral theory. Number 92, Cambridge university press, 1989.
Edward Brian Davies. Heat kernel bounds for higher order elliptic operators. Journées Équations aux dérivées partielles, pp. 1-11, 1995.
11

---

Published as a conference paper at ICLR 2025
Beichuan Deng, Yeonjong Shin, Lu Lu, Zhongqiang Zhang, and George Em Karniadakis. Approx- imation rates of DeepONets for learning operators arising from advection srrection-diifusion equations. Neural Networks, 153:411-4266 2022. ISSN 0893-6080.
R. A. DeVore, G. Kyriazis, D. Leviatan, and V. M. Tikhomirov. Wavelet compression and nonlinear n-widths. Advances in Computational Mathematicx, 1:197-214, 1993.
Ronald A Vore and Vasil A Popov. Interpolation of Besov spaces. Transactions of the American Mathematical Society, 305(1):397-414, 1988.
M. I. Dyachenko. The rate of u-convergence of multiple Fourier series. Acta Mathematica Hungar- ica, 68(1-2):55-70, 1995.
Charles Fefferman. On the convergence of multiple Fourier series. Bull. Amer. Math. Soc., 77(5): 744-745, 1971.
AG Georgiadis, J Johnsen, and Morten Nielsen. Wavelet transforms for homogeneous mixed-norm Tricbel- -izorkl-Lizorkin spaces. Monatshefte für Mathematik, 183(4):587-624, 2017.
Y. Giga and T. Miyakawa. Solutions in L" of the Navier-Stokes initial value problem. Arch. Rational Mech. Anal., 89:267-281, 1985.
Ingo Gühhring Gitta Kutyniok, and Philipp Petersen. Error bounds for approximations with deep ReLU neural networks in WAP norms. Analysis and Applications, 18(05):803-88) 2020.
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet-based operator learning for differen- tial equations. Advances in neural information processing xystems, 34:24048-24062, 2021.
Nguyen Dinh Huy and Nguyen Anh Tuan. Global solutions to a modified Fisher-KPP equation. Discrete and Continuous Dynamical Systems S, 17(3):1329-1346 2024.
Slim Ibrahim, Rym Jrad, Mohamed Majdoub, and Tarek Saanouni. Local well posedness 2d semilinear heat equation. Bulletin of the Belgian Mathematical Society-Simon Simon Stevin, 21(3):535- 551, 2014.
Masahiro Ikeda, Koichi Taniguchi, and Yuta Wakasugi. Global existence and asymptotic behavior for semilinear damped wave equations on measure spaces. Evolution Equations and Control Theory, 13(4): 11()):11111125 2024.
Norisuke Ioku and Takayoshi Ogawa. Critical dissipative estimate for heat semigroup with a quadratic singular potential and critical exponent for nonlinear heat equations. Journal of Differ- ential Equations, 266(4):2274-222) 2293, 2019.
Norisuke Ioku, Giorgio Metafune, Motohiro Sobajima, and Chiara Spina. LP-L" estimates for homogeneous operators. Communications in Contemporary Mathematics, 18(3):1500377, 2016.
Haydi Israel. Well-posedness and long time behavior of Allen-Cahn type equation. Communica- tions on Pure and Applied Analysis, 12(6):2811 1-2827, 2013.
Tsukasa Iwabuchi. The semigroup generated by the Dirichlet Laplacian of fractional order. Anal. PDE, 11(3):683-703, 2017.
Tsukasa Iwabuchi, Tokio Matsuyama, and Koichi Taniguchi. Boundedness of spectral multipliers for Schrödinger operators open sets. Revista Matematica Iberoamericana, 34(3):1277-1322. 2018.
Tsukasa Iwabuchi, Tokio Matsuyama, and Koichi Taniguchi. Bilinear estimates in Besov spaces generated by the Dirichlet Laplacian. Journal of Mathematical Analysis and Applications, 494 (2):124640, 2021.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for Fourier neural operators, Journal of Machine Learning Research, 22(290):1-766 2021.
12

---

Published as a conference paper at ICLR 2025
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to PDEs. J. Mach. Learn. Res., 24(89):1-97,, 2023.
Nikola B Kovachki, Samuel Lanthaler, and Andrew M Stuart. Operator learning: Algorithms and analysis. arXiv preprint 1rXiv:2402.15715 2024.
Anastasis Kratsios, Takashi Furuya, Matti Lassas, Maarten de Hoop, et al. Mixture of experts soften the curse of dimensionality in operator learning, arXiv preprint arXiv:2404.0m 09101, 2024.
Samuel Lanthaler. Operator learning with PCA-Net: upper and lower complexity bounds. Journal of Machine Learning Research, 24(118):167,, 2023.
Samuel Lanthaler and Andrew M Stuart. The curse of dimensionality in operator learning. arXiv preprint arXiv:2306. 15924, 2023.
Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for DeepOnets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applica- tions, t(1)tma 2022.
Samuel Lanthaler, Zongyi Li, and Andrew Stuart. The nonlocal neural operator: Universal ap- proximation. arXiv preprint arXiv:2304. 13221, 2023.
Shanshan Li, BoTang and Hajjun Yu. Better approximations of high dimensional smooth functions by deep neural networks with rectified power units. Communications Computational Physics, 27(2):379-41 2019. ISSN 1991-7120.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli. Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXlv:2010.08899. 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differ- ential equations. arXiv preprini arXiv:2003.0344 2020b.
Lu Lu, Pengzhan Jin, and George Em Kariiadakis. DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprini arXiv:1910.0319 2019.
Carlo Marcati and Christoph Schwab. Exponential convergence of deep operator networks for ellip- tic partial differential equations. SIAM Journal on Numerical Analysis, 61(3):1513-1555, 2023.
Tanya Marwah, Ashwini Pokle, Zico Kolter, Zachary Lipton, Jianfeng Lu, and Andrej Risteski. Deep equilibrium based neural operators for steady-state PDEs. Advances in Neural Information Processing Systems, 36:157 1055571115557 1533,, 2023.
Yves Meyer. Wavelets and operutors. Cambridge university press, 1992.
N Navaneeth, Tapas Tripura, and Souvik Chakraborty. Physics informed WNO. Computer Methods in Applied Mechanics and Engineering, 418:1 16546, 2024.
EI-Maati Ouhabaz. Analysis of heat equations on domains. Princeton University Press, 2005.
KK Pandey and P Viswanathan. Some elementary properties of Bernstein-Kantorvvicl operators on mixed norm Lebesgue spaces and their implications in fractal approximation. Nonlinear Analysis, 239: 113416, 2024.
Francisco Ribaud. Cauchy problem for semilinear parabolic equations with initial data in (!"())) spaces. Revista Matemática Iberoamericana, 14(1):1 1998.
Barry Simon. Schrödinger semigroups. Bulletin of the American Mathematical Society, 7(3):447- 526, 1982.
Per Sjölin. Convergence almost everywhere of certain singular integrals and multiple Fourier series. Arkiy för Matematik, (1)566990, 1971.
13

---

Published as a conference paper at ICLR 2025
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 15:1596-1611, 2022.
Rodolfo H Torres and Brika L Ward. Leibniz's rule, sampling and wavelets on mixed Lebesgue spaces. Journal of Fourier Analysis and Applications, 21:1053-1076, 2015.
Hans Triebel. Function spaces and wavelets on domains. Number 7. European Mathematical Soci- ety, 2008.
Tapas Tripura and Souvik Chakraborty. Wavelet neural operator for solving parametric partial differ- ential equations in computational mechanics problems. Computer Methods in Applied Mechanics and Engineering. 404:115748 15783, 2023. ISSN 0045-7825.
Fred B Weissler. Semilinear evolution equations in Banach spaces, Journal of Functional Analysis, 32(3):277-296, 1979.
Fred B Weissler. Local existence and nonexistence for semilinear parabolic equations in LP, Indiana University Mathematics Journal, 29(1):79-102, 1980.
Przemyslaw Wojtaszczyk. A mathematical introduction to wavelets. Number 37. Cambridge Uni- versity Press, 1997.
Eberhard Zeidler. Nonlinear Functional Analysis and its Applications I: Fixed- Point Theorems. Springer New York, NY, 1986.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A EXAMPLES OF & (ASSUMPTION I)
This appendix provides some typical examples of C satisfying Assumption
A.1 DIRICHLET LAPLACIAN
Let D be bounded domain in IR with d E N. Let L = - Die denote the Dirichlet Laplacian on (D). Then the semigroup (S-Api (t)]:2 corresponds the solution operator of the initial boundary value problem of the linear heat equation
O.u Au u(0) 40
in (0, T) x D, in D,
0
(0, () 0D,
where OD denotes the boundary of D. The integral kernel GDr (t, x, y) of S ADi (t) has the Gaus- sian upper bound
0 S GDir(I, S (4Tl)- exp (- -ー-
(-
= G(1, v),
4t.
for any and almost everywhere y (see e.g. Iwabuchi et al. (2018) Proposition 3.1)). Then, by combining this bound with the Young inequality, we have
1
g(l, vvll(())
Lon (R.)
Ct. -
- || (B:) Ct -
for any ʃ (D), where Q1 S 92 and / is the zero extension of / to Rd. Therefore, we see that C = ADir satisfies Assumption with v = d/2.
A.2 NEUMANN OR ROBIN LAPLACIAN
Let D be a bounded Lipschitz domain in Rd with d 2 We consider the Robin Laplace -A, on T,2 (D) associated with quadratic form
(f [ Vj- Vg + [ rfgds.
for any f.g (D), where y is function ON R. Note that y means Robin boundary condition, and particularly, the case of y = means the zero Neumann boundary condition. Hence, 7,2 (D)) Assume that y T, [a and , 0. We denote by Gt,t, y) and GNeu x, y) the integral kernels of semigroups generated by AA, and Neu. respectively. Then Gt,,, v) and GNeu (t, x, y) have the Gaussian upper bounds:
-An is the Neumann Laplacian - -eeu
S GNeu 2, Ct exp (-1= vl
GD,,(,,,,))
for any < and almost everywhere x, y E and for some C, C' > 0. The first and second
inequalities follow from domination of semigroups (see e.g. Ouhabaz (2005 Theorem 2.24)), and the proof of the last inequality can be found in Choulli et al. (2015). These bounds yield that A
and Ay satisfy Assumption I with d/2 in the same way Appendix A.1
A.3 SCHRÖDINGER OPERATOR WITH SINGULAR POTENTIAL
Let D be bounded domain in Rd with d 2 1. We consider the Schrödinger operator - -V+V with the zero Dirichlet boundary condition on OD, where V V(x) is real-valued measurable function on D such that
V - V. V1 2,0, V4 (D) and V- Ka(D).
15

---

Published as a conference paper at ICLR 2025
We say that V- belongs to the Kato class Ka(D) if
V-(v) yld-2 dy 0
lim sup
for d 3,
r-+0 Df D D0(02-1/lls0
lim sup
log(1x yl VVV- dy= =0 0 for 2,
m-0 wED Dn((2-a/er)
sup
(y) dy < 0
for d =
=ED (on[(=-llls
(see Simon 1982 Section A.2)). It is readily seen that the positive part V+ is almost unrestricted, and the negative part V- allows for the singularity such as (a) 1/|z|" with S 2 d and 0 I < 1 if d = 1. Then it is known that the integral kernel Gy of semigroup generated by Schrödinger operator
+ V satisfies the Gaussian upper bound
1Gv (t,x,y) & Ct exp (-1e
(-1---))
(10)
C't
for any 0 < < and almost everywhere x, J E D and for some C, C" > (see e.g. Iwabuchi et al. 20188 Proposition 3.1)). Therefore, L - satisfies Assumption way as Appendix A.1
+
] with d/2 in the same
A.4 UNIFORMLLY ELLIPTIC OPERATOR
We consider the following linear parabolic PDEs:
0iuu + Lu 0 in (0, D, in D,
(11)
u(()) u0
where
d
L: Σ Oze (axs02,) + Σ Or + Co. kjj=1
k=1
with real-valued functions 0g.ji bx, on D and with domain D(C) that a closed subspace of (D). Note that the choice of domain D(C) corresponds to the choice of boundary condition. For example, D(C) (D) and H (D) mean the zero Dirichlet boundary condition and the zero Neumann boundary condition, respectively. We assume that satisfies the uniformly elliptic condition:
ak.j by, co E (D) for all 1 < k,j < d:
There exists a constant C > 0 such that
d
(n, 2 Cies,
AJ=1
for any & E Rd and almost everywhere x E D.
In addition, we also assume the following on D(C):
if d 2 3, 2" is any number in V is continuous embedded into 2 (D), where 2 2d d
(2, oo) if = 2, and 2" oo if d = 1;
*
If u E D(C), then max(0, u), min |u|) sign E D(C);
If u E D(C), then eau E D(C) for any real-valued function 7) E C such that r) and Vni are bounded on D.
Then it is known that the integral kernel of semigroup generated by L satisfies the same Gaussian upper bound as (T0) for any 0 t and almost everywhere x, E D (see Ouhabaz (2005 Theorem 6. 6.)).. Therefore, the uniformly elliptic operator L satisfies Assumption with d/2.
16

---

Published as a conference paper at ICLR 2025
A.5 OTHER EXAMPLES
Various other operators, such as the Laplacian on the d-dimensional torus (periodic boundary con- dition), the fractional Laplacian (-A)a/2 (v = d/a), the higher-order elliptic operator (v = d/m with the highest order m) and the Schrödinger operator with Dirac delta potential (v 1/2 with d = 1), are possible, but these will not be mentioned here. For more information, see c.g. the books
Davies 1989 Ouhabaz 2005) and the papers (Bui al. 2020
Davies 1995
Ikeda et al. 2024
wabuchi 2017) and references therein.
B PROOF OF PROPOSITION
Let D be bounded domain in Rd with d N. parabolic PDEs
We consider the Cauchy problem of nonlinear
0,uu Lu F(u) (0) = uo
in (0, D,
in D.
This problem can be formally rewritten as the integral form
u(t) Sc(t)uo + J. ((((t-))((((t)dd
(P))
Throughout this appendix, we always assume that and satisfy Assumptions and respectively. We study local rell-posedness (LWP) for the integral equation (P). The LWP for P means the existence of local in time solution, uniqueness of the solution, and continuous dependence on initial data for (P') and it has long been extensively studied (cf. [Brzzis & Cazenave (1996); Weissler (1979 1)...).
this appendix, we provide the LWP results that can be used as the basis of our study of neural operator and its approximation theorem. We use the space (0, T; (D))
solution space. For convenience, we set
= a((,,,,)) = 1(p-1) B = B(ppp = T-p+1
(12)
We have the following result.
Proposition 3. Let d E N. E (0) and v > O. Assume that r, satisfy
8 E lp, oo]
and
(13)
Let M > 1 and E (D) be such that
+ 5M < M,
(14)
where p and 8 are defined by
p == P(T, [1 ) = D| |1uolle.,
8 5(T, , .M) = 2(0 +1 2
respectively. Define the map = hug by
([u](t)= = Sc(t)uo f Sclt (((())))(((())dd dr,
for E [0, T) and the complete metric space X(T, M) by
x BB = (u E', [ S MM,
equipped with the metric
d(u, v) = 1|u-
Then & is a contraction mapping from to itself. Consequently, there exists a unique function X such that b(u) u.
17

---

Published as a conference paper at ICLR 2025
Proof. The proof is based on the standard fixed point argument. Let take the parameters r, satisfying
r, [1. oo],
1,
T 21, 0<8<1 1 and a+ 0.
(15)
p p
0.00 > >, Take M
and un E L (D) such that
p+ 6M S M and 8< 1.
(16)
Let X. Then
o (ulll (ol $ (15c(t))all CF L'' ( (((t)))) dr
(T(0,2:20)
I II.
As for the first term, it follows from Hölder's inequality and ① that
IS D (0
(0.T, S CclD T No 1olle> P.
As for the second term, it follows from I that
/ (t 7) (1-r))((B-))lluual ()]])((((()))-((((()) dr
C
L(((0,T))
CCCCP 1/' (t-r (t-)))Il(()1( dr
(.((()T)))
CcCCrll gllor
where 2 and s/p are required and the functions f.g are defined by
for 0 <T
f(t) fr())
0
otherwise,
and
( [((())
for <t otherwise.
g(t) = (t) =
By Young's inequality, we have
|l * gller() I [l. , (R) |199| (R)
(a.5-1 + 1) T (0.77:)))))
I (a3 1) - =To+0 MP.
where we require r/p 1,0 < < 3 and a + > 0 and we calculate
|lfl (x)) (a. +1) T
Summarizing the estimates obtained above, and then using
14 we have
|| [ S p CcCF (as-1 + +1
a M = = 8 S M,
which implies that & is mapping from X itself.
Next, we estimate the metric d( (u1]. 4/u)]). Let 41 E X. By using and Hölder's inequality, have
d(O(ui), ((uz)) CcCr 1 (1 ( ()- max (uc(t) m2(F)]ll + dr
ma
Lr((0,7))
/
< 2C max im1,2
(1- ( dr Lr((0,T))
20 max |f h4 hiller
18

---

Published as a conference paper at ICLR 2025
where f is the same function as above and h; is defined by h:(1) (1) Il (+) (T) (2(T))l
for <tsT, otherwise.
By Young's inequality and Hölder's inequality, we have d( d($(ui), T ((2)) v 2CcCr
max
|/ hell
(=1,2
- CC
max
|
1 (3)
| (R))
1=1,2
$ 2CcCr (a.9-1+1)
Ta+8
max 11
Uz (0,,;; 1| (a.1o-(..T:11))
- 20 (ag-1=11 - T M d(u1, u2) ód(u1, u2).
From the second inequality < in (16), we show that + is a contraction mapping from into it- self. Therefore, Banach's 's fixed point theorem allows to prove that there exists uniquely function u E X such that u = o(u]. Finally, noting the condition [1) is equivalent to (33) and the condition
[66 is also equivalent to we conclude Proposition
Since (1/r, +-) is always positive under the condition 13 for aorany M > 0 and uo 1 (D)) we can take T' > 0 sufficiently small so that (4) holds. As a result, we have the following LWP result.
13 Then, for any E (D), there exist a time T T(ug) > 0 and unique solution E L" (0, (O, (D)) (P). Moreover for any E L (D), the solutions to (P") with u(0) Mu and v(0) = uo satisfy the inequality
Corollary (Proposition Dp. Assume that T, satisfy
||u - 1 I C'Iluo vo Lo
where (iii((((no)) [.
Proof. The former immediately follows from Proposition The latter is also shown in similar way to the proof of Proposition 3 In fact, let u, E X be solutions to (P' with initial data uo and Uo respectively. Then
||u c & T - vollo- + +ollu
T sufficiently small so that 1, then we obtain
Cc|D T |uo - vol..=0
| ul--vle-00.1210 <
The proof is finished.
C APPLICATIONS TO FNOS AND WNOS
In this appendix, we apply Theorem [ to FNOs and WNOs for some operator L. Let us recall that, to ensure our quantitative approximation theorem (Theorem Db. the families & = (pn), and 2b == (mm Im of functions need to approximate the Green function G of the linear equation 0uu + Lu (i.e. the integral kernel G of semigroup generated by L) such that
G(t T,I,8)
Σ M.nEA
Cm.nt C (t,,))
(17)
Gx T,
Σ
Con (T, (t,
m,nEAx
for 0 < <<< T and x, E D. where A is an index set that is either finite or countably infinite and AN a subset of A with its cardinality |Ax| N N. The convergence (17) means the sense that
Ee(N)= = | T,Ir G - T, (ooz) +0
(18)
19

---

Published as a conference paper at ICLR 2025
and
(N((N) [G 3) - GN 2t,,,,,))12; (5(0,7;44)) 0
(19)
as NN - oo for some r, E [1, oo] satisfying the condition for the arguments in this appendix:
The following conditions are required
|| - T, p. (..:)))))
<0,
(20)
||G (T(..T:1g))
<00.
(21)
Remark 5. Let us give some remarks on 20 and (1).
(a) It seen that 20 and always hold with
oc under Assumption (see Lemmal 7 in
Appendix D
(b) For example, the operators & appeared in Appendix A.4. satisfy
Aastsssssyy the Gaussian upper bound 10P. and hence, the corresponding Green functions satisfy (20) and 21 if
d 2
<2 and
(22)
r
8
r
C.I FOURIER NEURAL OPERATORS
Before stating FNOs, provide remark on partial sums of multiple Fourier series. From the perspective of convergence issues, the rectangular partial sum is preferable to the spherical partial sum. In fact, for f E (TT the Fourier coefficient of f is defined by
On fm) [ f(x)e (xx)e-2rin 22rén dx,
(n1, n2, ... na) E Zd
where T We define the rectangular partial sum of multiple Fourier series of f by
= Rd /Z' is the d-dimensional torus, x == (x1, T2... rz) and nx == n1 + ndJd-
(x(f(z) = Σ
CnE 2riin
2rina
NEN.
ImaleN
Then we have the following:
Theorem Sjölin 1971 Fefferman 1971)). Let q > Then
lim SN (z) f(x) a.e. and
lim Sx(f)=] = in L((T)) N
N N
for any f E (TT"..
This theorem can be easily extended periodic functions f a general d-dimensional rectangle with usual modifications. In contrast, the spherical partial sum of multiple Fourier series f E L (TT) converges to L only if q and for its pointwise convergence, it remains open problem even when = 2.
Next, mention the approximation theorem FNOs. Here, apply the Fourier expansion with respect to both time and space æ to define FNOs. This type of FNO is used in the implementation approximate some PDEs (see e.g. Kovachki
22023 Section 7.3)).
For simplicity, we consider only the case d = 1, r 8 2 and C is one of Appendixes A.
A.A. A.4
(which implies v d/2). Then the condition (22) satisfied and Theorem can be applied to FNOs under the condition 3 i.e., < < 242 if we can show that (88) and 19 hold for the Fourier basis.
In fact, consider the nonlinear parabolic PDEs P where D is bounded interval in R. As G is not a periodic function, a zero extension of G is considered here, so that the multiple Fourier series can be applied. We take as a bounded interval which includes and with its length L, and we define the zero extension of G = G(t - -T to [-, x [-T,27 x D x D by
G(l T,
if < T, 2. D
G(t T. v)
0
otherwise.
20

---

Published as a conference paper at ICLR 2025
Then, thanks to 20 and 21). G has the multiple Fourier series
G(t - T,
Σ N N M.NEAN
lim
Cm,ne 52
82
2minge
E
for almost everywhere in (1,, T,, E [-T,2T] x [-T,27] x D x ò and it also converges in (2( 2T] x
2 x D x D), where AN == {(m, n)) = (m1, m2, n1, n2) E z
[m1), |m2), |mll
fiul.11a |n2l S /N1/4) and Cp. is the Fourier coefficient given by
Cm,n
-T J.I.
G(t T, 2. V)e
2rintt
2mingr
2ninge
e dtdrdwdy.
2mingg
Hence, putting (t, x) 2 2minge and 3m (T,
2
2
37
3T
we have the expansion
(17) of G with (18) and (19). Therefore, by choosing these = frn In and
( Im in
Definition D have the following result the FNO as a corollary of Theorem |
Corollary 3.
2, L be one of Appendixes A.7
A.7.A.4 A.4 (v 1/2),
{pajo and
= (mm ) as above, and F satisfy Assumption 2 with < < 7/3. Then for any R > there exists a time T > O such that the following statement holds: For any e E (0, 1), there exist depth
L, the number of neurons II, rank N and FNO I' E NO HH, ReLU such that N.p.w
sup
[[++ (cw))l Se.
B (R)
Moreover, L = L(T) and H = HT)) satisfy
L(T) S C(log(( 12 and H(T) S Cc (log(e 11)2, where a constant depending on v, M, F, D,P, p, d, r, 8, and L.
C.2 HAAR WAVELET NEURAL OPERATORS
A wavelet is an oscillatory function with zero mean, used to analyze signals functions locally at different scales. There are various types of wavelets, such as Haar, Meyer, Morlet, and Shannon and the systems constructed by these wavelets unconditional bases for (R with 1 < q < so (see e.g. Meyer 1992); Wojtasze7yk 1997)) Wavelets can also be defined on domains other than on Rdd (see e.g. Triehel (2008). ). Several results wavelets for mixed Lebesgue spaces are also known (see e.g. Georgiadis et al. (2017); Pandey & Viswanathan 2024); Torres & Ward (015)).
In this appendix, we use the fact that the multiparameter Haar system is an unconditional basis for Qa) E (1, oo) dd by Pandey & Viswanathan
the mixed Lebesgue spaces (8, 1)4) with q (2024 Theorem 4.9).
The set of dyadic intervals in [0, 1] is defined by
h
ke +++1
D:= J,,x 21-1"
:0 Losk<22-1 .JEN
23-1
For I E DD we denote by Ieef and Fright the left and right halves of I, respectively, and define the Haar function hú by h1 = Xhea XTsiaa- Here, X7 is the characteristic function of an interval I. The set of dyadic hyper-rectangles in [0. 1]d is defined by
Ra = (I = 11 12 x la 11,2 La E D).
For I E R.,, we define
bu (z) == h4 (o) ),, (x2) (xa)
for x (2 ...., v.) [0, 1]d. Then the multiparameter Haar system defined by {hz I
Ra} is an unconditional basis for the mixed Lebesgue spaces (8, 1)d) with q = (q), (q qa)
(,) (see Pandey & Viswanathan 20024 Theorem 4.8)).
Let G be the zero extension of the Green function G to a function on 00, 222+2 (= [0, 10 1|1++++dd
with respect to f, Ir, y. Then we can apply Pandey & Viswanathan 2024 Theorem 4.8) to G and then
G( - (, = lim
Σ
czbz (1) (z) (T) (v)
No000
IER2442.N
21

---

Published as a conference paper at ICLR 2025
in the sense of mixed Lebesgue norms of ((0, 1)2d+2) with all q (q) (9.....,92) qu) (1, 20) where R2d+2.N is a subset R2d+2 with its cardinality [R2442.N| = N, CI are wavelet co- efficients, and I = It In x I, x IN E D x D4 x D x D Hence, putting Cm, = CI. PP (t, = hr, (t) (x) and Vm (m (T hr, (T) (v) with m (11, (N (0)) Na and n (k2, j2) E (O)) x No, we have the expansion (17) of G with (18) and (19 where Av is an index set of (1 k2. j2) corresponding to R2442,N with its cardinality AA N.
Therefore, by choosing these ip {rnJn and v result on the Haar WNO as a corollary of Theorem
f Jm in Definition | we have the following
Corollary 4. Let E N. Suppose L is one of Appendixes A.4 (v d/2) and F satisfies Assumption 2 Let r,8 E (1, satisfy the conditions (3) and (22) and let & {en)a and
- fm m Im be as above. Then, for any R > there exists a time T O such that the following statement holds; For any E (0, 1), there exist a depth L, the number of neurons H, rank N and Haar WNO T ENO L.,,H.Rell such that
N.p.0
(1o+ ((W))ILT Se.
sup B (It)
Moreover, L L(T) and H HT) satisfy
L(T) < C(log(e 1))² and H(T) S Ce + 1)) )))
where C' > 0 is a constant depending on v, M, F, D, d, r, R and L.
Note that the previous papers of WNOs such as Gupta et al. (2021); Tripura & Chakraborty (2023)
have not employed Haar wavelet in the context of learning solution operators for PDEs. Here, just chosen the Haar wavelet as a concrete example of basis expansion of the Green function, and the discussions choosing other wavelets could be possible.
D PROOF OF THEOREM
Suppose that L and F satisfy Assumptions and respectively, and the parameters r, satisfy the condition (3). The conditions for the parameters R. T, M, M' that appear in the proof are specified here. Let take R, M, M' > 0 and 7 E (0, 1] such that
C=|D (az +1)-+ MP S M,
2C +00B++2CcT(++000 CpM') M', T- M,
(23)
where and 3 are the constants defined in 12 The first condition in 23 ensures the LWP (T) (Propositions) [ and [ See Proposition and (4] Appendix for more details. The second condition ensures that the approximate mappings 4N,, and Ne the proof map from B ( (M') into itself. The third condition ensures (0, (M') C (I (M). We note to take the parameters in the following order: For any R, M 0, we can take sufficiently large M' and a sufficiently small so that 23 holds.
To begin with, we prepare the following lemma. Lemma 1. Under Assumption
we have
| T, (.22),111111001)1 1000: ( < CCT,
|| 1))1; 1011: SCc.
for any T E (0, 1].
Proof. From Assumption I and Iwabuchi et al. (2021 Lemma B.1), we see that
||G(1 T, H. w\ 1
||S
+) Cc.
22

---

Published as a conference paper at ICLR 2025
which implies that
(10.T1))))))) S CCT.
Similarly, as we also have The proof of Lemma] is finished.
1.t.,,,,1 [|sc(t)](====== Cc.
( $ Ce.
We divide the proof of Theorem into four steps.
Step We define &N by
@x[u[(t,x)) = / GN (t. + [(], J. GN(t - T, T. Fu((,, drddy
Since En(N), Ey (N) - as N + co with r', s 2 and D c Rd bounded, we see that
T,, ||G(L -,, v) G - (, 7,2,,)llo40o.T.44 (2(0.TT11)) ( - 0,
[|G(t,,))) - (t x, v) |1: (7(0,T:42) +
as N oo, and hence, there exists subsequence {N*tKEN such that
||GN, (t 7, -1 |, 11 |[GN, (1, + [IG(1,,,))),
for almost everywhere / (0, 7) and x E as + co. Then, for any e 0, there exists h; E N such that for any k 2 ke,
|I (t- (, (.(0TT11)))) ( 20CTT,
| (1. (1,,,)) LT 2C2,,
E.(NN). (Nx)
where the first and second inequalities follow from Lemma and the last follows from Assump- tion 3
Lemma 2. For any ug E B1 (R) and k 2 ke the following statements hold:
NN. map from B (1.T(..,)
itself.
(ii) There exists constant C₁ such that
[ &N. (wl. & Cie,
for u E (L=(0.T;L=) Here, C₁ = DD CrMP.
Proof. Let un E By R)) and 2 k. We first prove (i). By Hölder's inequality and the second condition in 23 for any (M' estimate
|
S |I w)( (0.T,1))))) |
|volla>
+ +||IGH. - -1,,,,))ll44.... (1 [((W))[>()))))))
$ 2C M < M'.
23

---

Published as a conference paper at ICLR 2025
Next, prove (ii). By Hölder's inequality and the third condition in 23 for any (MTT we have
[fu](t,x) - CN (t), & [G((,,,,)) - GN2 (t, v) L2 || T, T,
+|IG(t
GN2 (t 2, (.ll ((4)))oroo
which implies that
| 2 Iw
[| v) GN((, W.ll Lroor
| Ilvllle-
+ +1110((-7,,,1) -,,((
GN(( -, 7, 7, (omp/// [F(W)ll
s c ||oolll ((l
| R+ +
= Ccc
where we note that ( (MM') C (L Therefore, (ii) is proved. Thus, the proof of Lemma is finished.
Step We define the map Net by
[e) = / G (t, ()oo
( I. GN, (t und u))drdy, (24)
where Fret Assumption
is ReLU neural network. Here, as (MMM 11,00 (-M', ,M' from we can apply the approximation result by Gühring et al. (2020) Theorem 4.1) to see that there exists a ReLU neural network F Feet" - such that
F Fnet L (-MM)) S C,
(25)
where the depth I. = [( ((Feet) and the number H = (Fneet neurons are estimated as
[[(Feet) S C Clogg (6-1). (H(nett) S C (c-l),
(26)
where C = C(M', F) > 0 is constant depending on M', F.
Lemma 3. For any ug E B1 (R) and k: 2 ke, the following statements hold:
(i) nee is a map from B1 (1 to itself.
(ii) There exists constant C2 > such that
|1 [w] 4 Ne. [w]l S Cze.
for u E (M'' Here, C2 = 2C2| 1
Proof. The proof follows the same lines with Lemma 2 Let uo By- (R) and k 2 k. By Hölder's inequality and Assumption for any u E B (0,T (M'))
(M'), we estimate
b Ny net (w]l=~ < | (1, 2, 1
|
+|IIGN, T, 2,,,,,,,,) (0,1; [FFe((u)ll
$ 2C2R 2CcT(1 Cr < M',
24

---

Published as a conference paper at ICLR 2025
where we used the inequality
[1ee [l) - ( (PPillloooo))))))))
Sc+ Cp S <1+CPMM.
Hence, (i) is proved.
Next, we prove (ii). By Hölder's inequality, for any E (L=(00T:)>))(M)) we estimate
ΦN, (x,,x x) - + ned [u]
S Sl (1 - -T, a, () (400
(=]("))(==)) [T] - (ne & 2C.
Hence, we obtain
No. [u]
4 ma 2Cc C2e.
Therefore, (ii) proved. Thus, the proof of Lemma] Bis finished.
Step 3. We define I B (R) (o.T,,)))) by
T(wg) $ [J] Net [0] for un E B1
(27)
Lemma 4. Let J uo E B (R)
[log(1/e] 1 N. Then there exists constant C3 such that for any log(1/A)
|| (uo) ((oo)1l S Cze,
where C3 M + Ci+C2
Proof. By the triangle inequality, we have
| (wo) (. S (wo) ক
+ | (1
As to the first term, since + B (L- - (M) 8-contractive and a = (+o) the fixed point of & from Proposition 2 have
| (ua) [00 (0) = || - 1 / [0] (.
< 8 ||u
0 01 & M8 s Me.
(28)
As to the second term, we see that
||00
||01//100 [0
16-11 Ne.nes) [0]
($LU-s) [3] (l_ [0] [01 (0111-10z2.))
(29)
أー || 1b o T Ne. [0] $ me (7(00224))
(30)
j=1
+ 11 1. - + @Nanuluy-uwllloooruy Σ 8
(31)
where uy.N = $ bi
Nr net and 1 E B, (M') C (Ir(Oor)s)MMM by the third condi-
tion 23 By Lemma and Lemma 3
we see that
+ [u N] 1 N.. wll
[ 2 11--Nuuy--wwllllorrza [1
NN (u I.NII.
(C1 C2)c.
25

---

Published as a conference paper at ICLR 2025
Hence,
8(C + c2)e= Ci+C,
[0] - (oo S
Σ (C++
Σ j=0
1- c. (32)
Therefore, by (28) and 32)
we conclude that
|| (un) - (un) ( | (uo) (0., [| -(uo))l=-0.T..1)
M C1 C2
-CE.
Thus, the proof of Lemma is finished.
Final step. Finally, it is sufficient to represent the approximate operator I' given in 27 as a neural operator in the form of Definition T and provide its quantitative estimates.
Lemma 5.
I TE L. N.9.0
where I. L(T), H = H(T) satisfies
L(T) S C(log(E +)² and HTT) S CE C 1))
where C > is a constant depending on v, M, F, D, d, r, 8, R, L, and N = N(T) satisfies
Ec(N) Se and (N( S.
Proof. Let N 2 Ng. Then we have
Ec(N) & Eo(NN) SG Ec(N) & E(((NK) Sc.
We that
Φ No. not (t)
-f ( 1/') /
(,, (u(T, ()),yy
= Σ
Cn.m (w (0.-), uo)px(txx)
Σ n.mEAN
Cn,m (w'm: Fnet (u)) (t,
n,mEAN
Σ
En 20)
Σ n.MEAN
En.m (t
=
A,mEAN
where
Cn,m if m E ANo otherwise.
Cn.m
We denote by
in (t,t,)) = Σ
En (1hm (0, +), un) ().
n.MEAN
Then we see that
(t,o)((,x) Ne. [0](t, vz(t,x),
where to 0 and
1 (1, = (t, +
Σ
Cn WWm: Frce (nc((1)))p j= j=0...,, 1.
B.MEAN
We define K (L) - (' by
(K (0)
10),))))
Σ
((0)) (0, u0)
ūū (t,
ū (t,
A.MAA
26

---

Published as a conference paper at ICLR 2025
where (10)
Cn,m Cn,m
R2x1,
We also define x
E (=))),;;='0))" by
Σ
(00) Camm (t/m Feet
1 (t,z) x) ==
= Σ
p00
(nt
R,MEAN
0
nEAN
where
((0)
N.n (
(00) (Wim Prnc(())) R2211
Then, we see that
(1(t,2)) ūi (t,
(K N un (t, 10 (t, =
which corresponds to the input layer.
Next, we define
W = 00 1) R222
and
Σ M.mEAN
Eam (Wm u1 (n)
(K)))((x): =
Σ
A.MEAN
0
for
(u1, U2) (=()) where Ennm
(Cum
0)
1222
We also define
Faet : R + RR2 by
Poee(w) (c) = (Fnee(ur))
(ui u2) E IR?
u
112
which is a ReLU neural network with (26) (Note that ReL.U neural networks represent the identity map). Then we write
(i + KN) Fret
Vi ū1
(t,x) W
(Faa)(y)) ūn (1(4xx))
+ KN (Fuesls)) (t,
th (t, +
Σ
En.m Wimm Fnee (1
A.MEAN
ūi (1,2))
()-1(t,z)) for j = 1...., - 1, Un (1,
which corresponds to the hidden layer.
Finally, denoting by
W := (1,0) E R
which corresponds to the last layer, we obtain
I(wo)= = W' a ( + KN) Fnet .... (W KN) Fnet o (R + +b00)
By the above construction, we can check that I' E NO H.R.U Moreover, the depth L(I') and the number HT) of neurons of the neural operator I' can be estimated as
N.p.
I(T) S J. I(Fnet) S log(<-1) (o S (l))(c-1))²,
H(L) & J- (Fnet) $ log(e - logz (11 S e (log(e 11)?
Thus, the proof of Theorem 1 is complete.
27

---

Published as a conference paper at ICLR 2025
E PROOF OF COROLLARY 1
By the triangle inequality, we estimate
Umax mmax
(un) (M0)----
Therefore, as '=" is finite, it is enough to prove that
|| (M) (o) p (u))ll
(33)
CC | [--1 |
(uo)((( 1)7) - r (u)) (1))(()))))l
for some C>> 0 and for = N². For 1, the estimate (33 is already obtained in Theorem
For x: 2...., k², we have
[m] (wo) - T [x| (10))1=((--1))TTTTT
||T (T |-- (wo)) r
| (rl= (wo)) -u) (orr
By Corollary 2 (or Proposition the first term can be estimated as
| r+*n+t
r (1000)---000---|aaa00000---0---
< l (ux) 1)7) rfx (w) 1)T
By Theorem the second term in RHS can be estimated by Ce since r rlx (2o) E B (R). Thus,
the estimates 33) are proved for all 1....,K".
F PRESERVING THE POSITIVITY
We define the ReQU activation function by
ReQU(t) = (0,
t R.
Corollary 5. Suppose that C and F satisfy Assumptions ! and respectively, and the parameters T, satisfies the condition 3 Assume that F e C² C', R is polynomial and satisfies that
F(t) - 0
ift 5 0.
(34)
Moreover, assume that there exists No E such that the truncated expansion Gy defined in As- sumption 3 of the Green function G satisfies
GN, xr, v) 0, 0<t<T, x,y D,
for any N No. Then, there exists E N. N. that I preserve the positivity. i.e., [(uo) 0 uo E (D)) and wa E 0.
ReQUU R.RQU that the statement in Theorem 1 holds such
Proof. We can show that there exist I L.O.H. satisfying the statement of Lemma N.p.w E is ReQU neural network and exactly represents the polynomial F (see
where
GN. and Frer:
e.g., Li & Yu 2019 Theorem 2.2)). Note that l(um)) 4 JJ]
Nu. net [0] where $ NN is a map defined
in (24) and depends on ug. We will show that by induction
Nu.net [0] 0 if uo
J]
0.
Let $ 0. First, we see that $ net [0]
Nx.net [0)
1S 0.
Assume that Tbb
No met [0]
VIA
ji < J - 1). Since F = Fnet satisfies 34 we see that
T [1++1] Ne.net [0] $ net. (1 1
GN,, ((x S( GN (1- (,,, v) (5)
N. Nor [0](T] v)) 5
Thus, the required result is proved.
28

---

Published as a conference paper at ICLR 2025
G ADDITIONAL REMARKS
In this paper, we focused on parabolic PDEs with power-type (or somewhat general) nonlinear terms. Below, we provide additional remarks on Assumptions | and and some remarks on applicability and limitation.
Regarding Assumption 1 Assumption foor C is a condition satisfied by the solution operators of many linear parabolic PDEs. In particular, it generalizes the rate of time using the parameter v, and it is known that many important operators satisfy Assumption []] as stated in Appendix [A] On the other hand, There exist examples such that Assumption T operators with inverse square potentials, i.e., C
does not hold. For instance, Schrödinger c (where e is real number greater [=|
than the negative of the best constant of Hardy's inequality), are known to satisfy the inequality in Assumption when d/2 and d 3, provided that 2d S q1 S 92 2ーz (this range is optimal. See Ioku et af. (2016) and Ioku & Ogawal (019)). Thus, due to the constraints the range of qı and 92. certain operators may fail to satisfy Assumption I By extending the range of parameters in Assumption to Jmiu S 91 S 92 I Juax. it is possible to generalize the assumption to include such examples. In this case, 9min and Imax, in addition to v, would influence the conditions for well-posedness and approximation eтror estimates.
Regarding Assumption 2 There are various examples of nonlinear terms, such as F(u) Vul? or F(u)= = =(ellll - 1), that do not satisfy Assumption 2 However, we believe that the arguments this paper can be extended such nonlinear terms as well. For example, the heat equation with F(u) Vul", known as the viscous Hamilton-Jacos equation, has its well well-pooednnes established via a fixed-point argument (see Ben-Artzi et al. (2002)). Similarly. this is also true for F(u) =
u(cl=l- )) or more general exponential-type nonlinear terms (see Ibrahim et al. (0014). The reason why our paper focused on Assumption is because the results on well-posedness vary significantly depending on the nonlinear term and it is difficult to unify these results. Therefore, this work, we focused on power-type nonlinear terms. As mentioned in Remark smooth nonlinear terms often be represented by their leading term in power-type form through Taylor expansion, highlighting the importance of power-type nonlinear terms from this perspective.
first step in
Regarding other equations. We expect that similar arguments can be applied to broader class of PDEs. For instance, initial(-boundary) value problems for various PDEs, such as the nonlin- Schrödinger equations and nonlinear wave equations (see Cazenave (2003); Cazenave & Ha- raux (I (1998)), nonlinear damped wave equations (see Ikeda et al. (2024)), the KPP-Fisher equation Huy & Tuan (2024)), the Allen-Cahn equation (see ( the Cahn-Hillirad equation
(see
(see Cholewa & Rodriguez- Bernal (2012)) and the Navier-Stokes equations (see Giga & Miyakawal
(1855), )), can be expressed as integral equations. By appropriately choosing the initial data space and the solution space. their well coleddassss can be shown by using the fixed-point argument. On the other hand, for PDEs that involve nonlinearity in the principal part such as p-Laplacian equations 0,u Ap F(u) with Ap div(|Vu Vu² and * 2, it may not be possible to represent them in the form of integral equations. Such equations might not be effectively handled by our
approach.
29


Reviews:
[{"review_id": "q31iqMn6VL", "reviewer": "Reviewer_r9yU", "summary": "This research focuses on a new method for approximating the solution operators of nonlinear parabolic PDEs using neural operators. The authors aim to bridge the gap between theoretical understanding of neural operators and their practical application as PDE solvers by developing a quantitative approximation theorem. This theorem demonstrates that neural operators, specifically designed with a structure inspired by Picard's iteration, can approximate solutions of parabolic PDEs efficiently without experiencing the exponential growth in model complexity often associated with general operator learning. The authors provide examples of how their method can be applied to Fourier neural operators (FNOs) and wavelet neural operators (WNOs), highlighting the potential for this approach to solve a wide range of nonlinear PDEs.", "questions": "1. Can the authors provide numerical experiments to validate the theoretical results and compare the performance of their neural operator with traditional PDE solvers or other neural operator architectures?\n2. The dual of $L^\\infty$ is more complex than that of $L^1$. The authors should consider revising or removing this aspect throughout the paper.\n3. Could the authors address how the proposed method handles periodic boundary conditions, which are essential for applying FNOs? Additionally, could they discuss the role of the trace operator and how to ensure sufficient regularity for the boundary conditions when defining the neural operator $\\Gamma$?\n4. Can the authors elaborate on how the proposed framework can be extended to handle more general PDEs, specifically the Navier-Stokes equations mentioned in the abstract with their specific projection operator and boundary conditions?\n5. In PDE theory, extending local solutions to global solutions is a challenging and significant issue. Although the paper acknowledges the challenge of extending local solutions to global ones, it does not provide concrete solutions or insights. Could the authors elaborate on this challenge and clarify the practical relevance of Corollary 1, especially regarding its error estimate for long-time solutions?\n6. How does the choice of basis functions $(\\phi$ and $\\psi)$ in the neural operator affect the approximation error rates? Could the authors provide specific examples to illustrate these rates for different basis functions?\n7. Regularity of the solution space is crucial in numerical approximation. However, this paper does not incorporate this aspect into an error analysis. Could the authors characterize the impact of function regularity on the error estimate?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents a quantitative approximation theorem for approximating solution operators of nonlinear parabolic PDEs using neural operators.\n2. The authors show that the depth and number of neurons in their neural operators do not grow exponentially with the desired accuracy, potentially mitigating the \"curse of parametric complexity.\"\n3. The proof is constructive, leveraging the connection between Picard's iteration and the forward propagation through the layers of the neural operator.\n4. The proposed framework may be potentially generalizable to other nonlinear PDEs solvable by Picard's iteration, including the Navier-Stokes, nonlinear Schrodinger, and nonlinear wave equations.", "weaknesses": "1. The theoretical results are not supported by numerical experiments, making it difficult to assess the practical performance and efficiency of the proposed neural operator architecture.\n2. The paper focuses on a specific class of parabolic PDEs amenable to analysis via Banach's fixed point theorem. The applicability to more general PDEs, particularly the Navier-Stokes equation mentioned in the abstract, is not demonstrated.\n3. The paper does not adequately address the challenge of handling periodic boundary conditions, a crucial aspect for applying FNOs.\n4. The paper does not discuss the role of the trace operator and how to ensure sufficient regularity for the boundary conditions when defining the neural operator $\\Gamma$.\n5. The assumptions on the operator L and the nonlinearity F are quite strong, potentially limiting the scope of applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6zZTeiCtxg", "reviewer": "Reviewer_kN84", "summary": "The authors derive the approximation rate of solution operators for the class of nonlinear parabolic partial differential equations (PDEs), contributing to the quantitative approximation theorem for solution operators of nonlinear PDEs. They show that neural operators can efficiently approximate these solution operators without the exponential growth in model complexity, thus strengthening the theoretical foundation of neural operators. An innovative link between Picard iteration and neural operators is found and the authors also leverage solid PDE theory (using Duhamel's principle). This work provides a very good theoretical foundation for neural operators (and generalizes previous works on Neural operators such as Fourier Neural Operators, etc.).", "questions": "1. I do not think it is easy to find a Green's function or design one, which I see as the main drawback of the paper. Perhaps it is not that hard for the linear parabolic PDE. (addressed in the Questions below). How can one go about finding these (for specific cases or in general) or is this mainly constructive? Or do you only need to characterize the regularity of the Green function and how can you do that, if that is the case?\n2.  Could you clarify if the error being log ((1/eps)^{-1}) implies exponential dependency?\n3 (a). How can I interpret K^{(l)}_N? W are weights, b are biases, but I am not sure what K (the rank) is (unless it is specific to the Neural Operator?)\n(b) Can you clarify (or add in the paper to make it more self-contained (see point 1 in Weakness) what is the rank of a neural operator? I do not believe it is defined in your paper (I checked other sources to understand). \n4. It seems the first 2 sentences of your Discussion are contradictory: you mention it is a limitation that your method only applies to parabolic PDEs, but the next sentence says your approach is not restricted to parabolic PDEs. Perhaps you could clarify that it is not truly a limitation then, and is just a first step?\n5. For the long-time solution, how reasonable is it that u(T) again falls into  Ball(L_\\infty, R) ? Does that limit the expressiveness of the solution of the PDE?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a highly rigorous and technical paper. The details are presented clearly and I am able to understand the authors' motivation and thought process in developing the theory. I think this is a great theoretical contribution that has solid foundations in PDE theory to better understand Neural Operators (of which there are varieties like Fourier, Wavelet, etc.) and unifies all of them in this fairly decent class of PDEs. I hope to see extensions of this work (that the authors mentioned there would be) to more general classes of PDEs but this is a significant start. Using neural operators as solvers for PDEs has a wide array of applications, particularly in scientific computing and other \"AI4Science\" domains. What is interesting is the method and analysis they use to bypass, by appropriately selecting the basis functions within the neural operator, the exponential growth in model complexity. They also do not seem to have too many unreasonable assumptions (this I'm not highly confident about).  Error rates with respect to truncation N make sense. There is a lot of scope for future explorations with different architectures and PDEs.", "weaknesses": "1. The paper could provide a bit more intuition and background on Neural Operators to be a bit more self-contained. This could be presented in the Supplementary.\n2. There are no numerical experiments (but for this paper I do not think this is a big weakness given that I believe the main contribution of theory is solid). Nevertheless, it would be nice to see in future (if it is even possible to implement all these different basis expansions, etc.)\n3. I do not think it is easy to find a Green's function or design one, which I see as the main drawback of the paper (if there is to be a practical extension for Deep Learning). Perhaps it is not that hard to find one for the linear parabolic PDE. (addressed in the Questions below).\n4. Minor: typo on line 182 \"elliptic operato,r \" \nI cannot think of any more now, but it depends on my questions below as well as I may not have understood everything.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zUcKWwISZb", "reviewer": "Reviewer_HAsy", "summary": "The paper defines a neural operator based on Picard iteration and provides a quantitative approximation theorem for learning the solution operator of a general class of non-linear parabolic PDEs. The paper starts with a local well-posedness result for the considered PDEs, then introduces the considered neural operators, states the quantitative approximation theorem, provides a sketch of the proof and discusses extensions to longer time-horizons. The supplementary material contains proofs and shows how the result can be applied to Fourier Neural Operators (FNOs) and Haar Wavelet Neural Operators (WNOs).", "questions": "### Additional points requiring clarification\n1.\tThe paper mentions that Theorem 1 avoids the curse of dimensionality. However, this is not entirely correct: it seems that in the statement of the theorem, the involved constant $C$ may still grow exponentially in $d$?\n2.\tTheorem 1 requires $N$ to be chosen in such a way that $C_G(N) \\leq \\varepsilon$. Is it possible to quantitatively relate $N$ and $\\varepsilon$, e.g., in the FNO case?\n3.\tP.3: You state that under suitable conditions on $\\mathcal{L}$ and $F$, the problems (P) and (P’) are equivalent, if the function $u$ is sufficiently smooth. Could you give some precise references here?\n4.\tAppendix A does not fully clarify why the considered operators satisfy Assumption 1. For example, A.1 states a bound on the Green function, but does not make explicit why this bound implies Assumption 1. For readers less familiar with PDE theory it would be useful to make explicit (at least for one of the sections A.1, A.2, …) why this bound implies that Assumption 1 is satisfied. \n5.\tSimilar results to Proposition 1 are well-known in the literature. Can you relate Proposition 1 to the literature more precisely, outlining differences to existing results in the literature? \n6.\tFourier neural operators: why do you only consider the case $d=1$ here? \n7.\tCould you clarify if WNOs with Haar wavelets, as defined in Appendix C.2, have been used in previous papers?\n8.\tWhat is the role of $\\tilde{D}$ in Appendix C.1? Can’t we just set $\\tilde{D}=D$?\n\n\n### Additional comments:\n\n-P.4, l182: it should be operator, not operato,r", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is well-written and structured. The results hold in quite general settings: Assumption 1 on the differential operator $\\mathcal{L}$ of the PDE is rather general and includes, e.g., Laplace operators with different types of boundary conditions. Assumption 2 on non-linearity seems more restrictive and mainly limited to power-type non-linearities, but the obtained results are useful even in the case $F=0$, which is also covered by the setting. Assumption 3 is crucial to make the approach work, based on an expansion of the underlying Green's function. In the supplementary material the paper specializes the setting to certain types of Fourier neural operators and Haar wavelet neural operators, demonstrating the usefulness of the result.", "weaknesses": "The setting is formulated very abstractly, making reading potentially difficult for readers less familiar with general PDE theory. The paper does not discuss how the constructed neural operators could be implemented. In relation to existing neural operators (FNOs, WNOs) it remains unclear in what generality they are covered. Relation to existing literature is not always clear. I make these points more specific in the questions posed below.\n\nGenerally, I think this could be a valuable contribution, enhancing our understanding of mathematical foundations of neural operators. However, currently there seems to be a mistake in the proof of Theorem 1, affecting the construction of the neural operators and potentially making implementation infeasible. Consequently, at this point I need to recommend rejection, but I would be willing to increase my rating if this issue can be addressed convincingly. \n\n### Problem in the proof of Lemma 5\n\nThe main issue I currently see is that there appears to be a problem in the proof / formulation of the neural operators. The issue concerns the use of the time horizon $T$ instead of time variable $t$. In the definition of $\\Phi_N$ in lines 264-269, in the first line the time integral is from $0$ to $t$ (the variable of the function to be approximated). In the next line, the integral with respect to $\\tau$ is replaced by the inner product $\\langle \\psi_m, F(u) \\rangle$. However, this means you are taking the integral up to $T$, not just up to $t$. \nThe same problem can be found in the proof of Lemma 5 in the Appendix (lines 1245-1248), which does not seem to be correct in its current form. It seems that, to address this, the neural operator architecture would need to be modified, by making also the inner products $\\langle \\psi_m, u \\rangle$ in the definition of the operator $K^{(\\ell)}_N$ depend on the time variable $t$.\nHowever, this would make a crucial difference from the perspective of computational effort required to evaluate the neural operator. In the current notation, to evaluate  $K^{(\\ell)}_N$  only one inner product needs to be evaluated (the same for all times $t$). It seems that to fix the issue mentioned above, for each time $t$ a different inner product would need to be evaluated, which may be computationally infeasible (as at any given hidden layer, you are computing inner products with output functions from previous hidden layers). Here a thorough discussion is required.\nEven without this issue, a more thorough discussion on implementation would be needed: The approach crucially relies on evaluating inner products, by definition of  $K^{(\\ell)}_N[u]$ . To compute such inner products (between the hidden layer functions and the basis functions $\\psi_m$) in practice, additional discretization is required at each layer; for each function evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qHG0WbhbxQ", "reviewer": "Reviewer_pUKd", "summary": "This paper studies the approximation property of the neural operator for representing the solution operator for nonlinear parabolic equations. The key idea is that, the solution of the equation, written as the Duhamel’s integral, can be obtained from Picard iteration, with exponential convergence. The authors use a deep neural operator to approximate the Picard iteration process, where each layer approximates one step of Picard iteration. Overall, the paper is well written with clear explanation for all the ideas.", "questions": "My primary concern is with the ReLU network approximation, specifically in the correspondence between the neural operator structure defined on page 6 and Lemma 2. In Eq. (24), you define the mapping, but it remains unclear how this mapping aligns with the neural operator structure described earlier. Please explain how the neural operator structure in page 6 result in eq (24). A clearer explanation or justification is needed to ensure consistency between these sections.\n\nAdditionally, I noticed that the notation $\\Phi$ represents one step (either exact or approximated) of the iteration. However, on page 6, within each iteration, the activation is only used once, which is a shallow neural network with a single hidden layer. The reference to Guhring et al. (2020, Theorem 4.1) pertains to deep neural networks, which might not directly apply to this context. I suggest finding a more appropriate reference for the approximation result, specifically one that addresses shallow networks.\nAnother minor question is that, I think your remark “the curse of dimensionality that occurs in conventional neural network approximations does not appear here” in page 8 does not make sense. As addressed by yourself, the curse of dimensionality is hidden in Assumption 3, the representation of the Green’s function of the linear equation.\n\nAnother minor question is that, I think your remark “the curse of dimensionality that occurs in conventional neural network approximations does not appear here” in page 8 does not make sense. As addressed by yourself, the curse of dimensionality is hidden in Assumption 3, the representation of the Green’s function of the linear equation.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Same as summary.", "weaknesses": "Need for clarification on ReLU approximation. Detail in Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)