Deep Operator Learning Lessens the Curse of Dimensionality for PDEs
Authors: Ke Chen
Chunmei Wang
Haizhao Yang
Venue: Accepted by TMLR ()

Abstract: Deep neural networks (DNNs) have achieved remarkable success in numerous domains, and their application to PDE-related problems has been rapidly advancing. This paper provides an estimate for the generalization error of learning Lipschitz operators over Banach spaces using DNNs with applications to various PDE solution operators. The goal is to specify DNN width, depth, and the number of training samples needed to guarantee a certain testing error.  Under mild assumptions on data distributions or operator structures, our analysis shows that deep operator learning can have a relaxed dependence on the discretization resolution of PDEs and, hence, lessen the curse of dimensionality in many PDE-related problems including elliptic equations, parabolic equations, and Burgers equations. Our results are also applied to give insights about discretization-invariant in operator learning.

Published in Transactions on Machine Learning Research (10/2023)
Deep Operator Learning Lessens the Curse of Dimensionality for PDEs
Ke Chen
kechen & undd edu
Deportment of Mathematics University of Moryland, College Park
Chunmei Wang
Department Mathematics University of Florida.
Haizhao Yang Department of Mathematics University of Moryland, College Park
Kzyang edu
Reviewed on OpenReview: /ttps:///ppnrevieeee net/ forum? 2uBFzuT2DN
Abstract
Deep neural networks (DNNs) have achieved remarkable success in numerous domains, and their application to PDE-related problems has been rapidly advancing. This paper provides an estimate for the generalization error of learning Lipschitz operators over Banach spaces using DNNs with applications to various PDE solution operators. The goal to specify DNN width, depth, and the number of training samples needed guarantee a certain testing analysis Under mild assumptions on data distributions operator structures, shows that deep operator learning can have a relaxed dependence on the discrelization resolution of PDEs and, hence, lessen the curse of dimensionality in many PDE-related problems including elliptic equations, parabolic equations, and Burgers equations. Our results are also applied to give insights about inseretization-inaaiiann in operator learning.
1 Introduction
Nonlinear operator learning aims learn a mapping from a parametric function space the solution space of specific partial differential equation (PDE) problems. has gained significant importance in various fields,
including order reduction Peherstorfer & Willcox (2016), parametric PDEs Lu al. (2021b); Li al. (2021)
inverse problems Khoo & Ying (2019), and imaging problems Deng et al.
[2020]; Qiao et al. (2021 Tian
et al. 2020). Deep neural wetworks (DNNs) have emerged as stale-of-the-art models in numerous machine
learning tasks Graves et al. (2013: Miotto et al. (2018); Krizhevsky et al. (0017), attracting attention for
their applications to engineering problems where PDEs have long been the dominant, model. Consequently.
deep operator learning has emerged as a powerful tool for nonlinear PDE operator learning Lanthaler et al.
(2021) : Khoo Ying (2019). The typical approach involves discretizing the computational domain and representing functions vectors that tabulate function values on the discretization mesh. A DNN is then employed to learn the map between finitedimennsioall spaces. While this method has been successful in various applications Lin et al. 2022 Cai et al. (2021), its computational cost is high due to its dependence on the mesh. This implics that retraining of the DNN is necessary when
(2022); Li et al.] (2021); Nelsen & Stuart)
using a different domain discretization. To address this issue, Li el al. (2021): Lu et al. [2022); Ong el al.
(202)) have been proposed for problems with sparsity structures and discretization innattiionnnvaainceee properties. Another line of works for learning PDE operators are generative models, including Generative adversarial models (GANs) and its variants Rahman al. (2022); Botelho et al. (2020): Kadeethum al. (2021) and
diffusion models Wang al. (002).. These methods deal with discontinuous features, whereas neural

---

Published in Transactions on Machine Learning Research (10/2023)
network based methods are mainly applied to operators with continuous imput and output. However, most of generative models for PDE operator learning are limited to empirical study and theoretical foundations are in lack.
Despite the empirical success of deep operator learning in mumerous applications, its statistical learning theory still limited, particularly when dealing with infiitteddimnnsooaal ambient spaces. The learning theory generally comprises three components: approximation theory, optimization theory, and generalization theory. Approximation theory quantifies the expressibility of various DNNs as surrogates for a class of operators. The universal approximation theory for certain classes of functions Cybenko (1989); Hornik [1991] forms the basis of the approximation theory for DNNs. It has been extended to other function classes, such as continuous functions Shen et al. (2019); Yarotsky (2021]; Shen et al. (2021), certain smooth functions Yarotsky (0018 Lu et al. (021aa]; Suzuki (2018): Adcock et al. (2022). and functions with integral
representations Barron (1993): Ma et al. (2022). However, compared to the abundance of theoretical works on approximation theory for high-dimensional functions, the approximation theory for operators, especially between infinite-dimensionall spaces, is quite limited. Seminal quantilative results have been presented
in Kovachki et al. [2021): Lanthaler et al. (2022).
In contrast to approximation theory, generalization theory aims to address the following question:
How many training samples are required to achieve a certain testing error?
This question has been addressed by numerous statistical learning theory works for function regression using neural network structures Bauer & Kohler (2019) : Chen et al. (2022); Farrell al. (2021): ;: Kohler & Krzyžak 2005 ; Liu et al. (2021;: Nakada & Imaizumi (2020); Schhmitt-Hieber (2020). In a d-dimensional learning problem, the typical error decay rate is on the order of n
n 0(174) as the number of samples increases. The fact that the exponent is very small for large dimensionality d is known as the curse of dimensionality (CoD) Stone (1982). Recent studies have demonstrated that DNNs can achieve faster decay rates when dealing with target functions or function domains that possess low-dimensional structures Chen et al. (2019)
) Cloninger & Klock (2020); : Nakada & Imaizumi (2020); (Schmidt-Hieber (2019); [hen et. al. (0019)
In such cases, the decay rate becomes independent of the domain discretization, thereby lessening the CoD
Bauer & Kohler (2019); Chkifa et al. (2015): Suzuki (2018). However, it is worth noting that most existing
To the best of our knowledge, previous results de Hoop al. (2021) : Lanthaler et al. (002 Lu et al. (2021b); Liu et al. (2022) provide the only
works primarily focus functions between finite-dimensional spaces.
generalization analysis for infinite-dimenn dimensooii functions. Our work extends the findings of Liu et al. (2022) by generalizing them to Banach spaces and conducting new analyses within the context of PDE problems.
The removal of the inner-product assumption crucial our research, enabling us to apply the estimates to various PDE problems where previous results do not apply. This is mainly because the suitable space for functions involved in most practical PDE examples are Banach spaces where the inner-product is not well-defined. Examples include the conductivity media function in the parametric elliptic equation, the drift force field in the transport equation, and the solution to the viscous Burgers equation that models continuum fluid. See more details in Scction 3
1.1 Our contributions
The main objective of this study is to investigate the reasons behind the reduction of the CoD in PDE- related problems achieved by deep operator learning. We observe that many PDE operators exhibit a composition structure consisting of lincar transformations and clement-wise nonlinear transformations with small number of inputs. DNNs are particularly effective in learning such structures due to their ability to evaluate networks point-wise. We provide an analysis of the approximation and generalization errors and apply it to various PDE problems to determine the extent to which the CoD can be mitigated. Our contributions can be summarized as follows:
o Our work provides a theoretical explanation to why CoD is lessened in PDE operator learning. We extend the generalization thoory in Liu al. (2022) from Hilbert spaces to Banach spaces, and apply to several PDE examples. Such extension holds great significance as it overcomes a limitation in previous works, which primarily focused Hilbert spaces and therefore lacked applicability
2

---

Published in Transactions on Machine Learning Research (10/2023)
in machine learning for practical PDEs problems. Comparing to Liu et al. 2022) estimate circumvented the inner-product structure at the price of a non-decaying noise estimate. This is a tradeoff of accuracy for generalization to Banach space. Our work tackles a broader range of PDE operators that are defined on Banach spaces. In particular, five PDE examples are given in Section 3 whose solution spaces are not Hilbert spaces.
Unlike existing works such as Lanthaler et al. 2022 which only offer posterior analysis, we provide an priori estimate for PDE operator learning. Our estimate does not make any assumptions about the trained neural network and explicitly quantifies the required number of data samples and network sizes based on a given lesting error criterion. Furthermore, we identify two key structures low- dimensional and low-complexity structures (described in assumptions 5 and G are commonly present PDE operators. We demonstrate that both structures exhibit a sample complexity that depends on the essential dimension of the PDE itself, weakly depending on the PDF discretization size. This finding provides insights into why deep operator learning effectively mitigates the CoD.
respectively) that
o Most operator learning theorics consider fixed-size neural networks. However, it is important to account for neural networks with discretization invariance properties, allowing training and evalua- tion on PDE data of various resolutions. Our theory is flexible and can be applied to derive error estimates for discretization invariant neural networks.
1.2 Organization
In Section [ we introduce the neural network structures and outline the assumptions made on the PDE operator. Furthermore, we present the main results for generic PDE operators, and PDE operators that have low-dimensional structure or low-complexity structure. At the end of the section, we show that the main results are also valid for discretization invariant neural networks. In Section 3 we show that the assumptions satisfied and provide explicit estimates for five different PDEs. Finally, in Section 4 we discuss the limitations of our current work.
2 Problem setup and main results
Notations. In a general Banach space X, we represent its associated norm as 1-Ix Additionally, we denote Ex the encoder mapping from the Banach space X' to Euclidean space Rdx, where dx denotes the encoding dimension. Similarly, we denote the decoder for X' as D IRdx - X. The Ω notation for neural network parameters in the main results section 2.2 denotes a lower bound estimate, that is, x = N(y) means there exists a constant C > such that æ 2 Cy. The O notation denotes an upper bound estimate, that is, x = Oly) means there exists a constant C >>0 0 such that x S Cy.
2.1 Operator learning and loss functions
We consider a general nonlinear PDE operator + X' u - y over Banach spaces X' and V In this context, the input variable u typically represents the initial condition, the boundary condition, or a source of a specific PDE, while the output variable v corresponds to the PDE solution or partial measurements of the solution. Our objective
train a DNN denoted as (lu;0) f) to approximate the target nonlinear operator t using a given data set S {(u,, 0z), 0; = ((u,) +,, 1,...,,200 2n). The data set divided into ST = [[u,, u;), 0; = ((u:) + s, 1,...,n) n) that is used to train the encoder and decoders, and a training data set Sy = [[u, v1), v, ((4) + E, n + .n....2n.. Both ST and S are generated independently and identically distributed (i.id.) from random mcasure y over X' with e; representing random noisc.
In practical implementations, DNNs operate on finite-dimensional spaces. Therefore, we utilize empirical eecoder-decoder pairs, namely E : X' + dx and D Rdx + X, to discretize u E X. Similarly, we employ empirical encoder-decoder pairs, EG : y + Rdy and D3: Rdy + y, for E y. These encoder- decoder pairs are trained using the available data set S or manually designed such that D o Ex = Iax and DT P.y. R Tay- A common example of empirical encoders and decoders is the discretization operator, which
3

---

Published in Transactions on Machine Learning Research (10/2023)
target operator
encoder Ex
DNN T
decoder Dj
input function u
output function
encoded vector
encoded vector
Figure 1: The target nonlinear operator T H is approximated by compositions of an encoder Ex: DNN function D, and decoder DJ- The finite dimensional operator r is learned via the optimization problem equation
maps a function to a vector representing function values at discrete mesh points. Other examples include finite element projections and spectral methods, which map functions to coeflicients of corresponding basis functions. Our goal is to approximate the encoded PDE operator using a finite-dimensional operator I that & D3 [ Ex- Refer to Figure 1 for illustration. This approximation is achieved by solving the following optimization problem:
NNN E argmin
To E3 (w:) (v:)
(1)
TEFx
Here the function space FNN represents a collection of rectified linear unit (ReLU) feedforward DNNs denoted as fx),, which are defined as follows:
f(x) = Wior.r 01 0.. (z) + br. 0,(x) = oW,, + =1-=1.... 1.
(2)
where a the ReLU activation function (xx max(r, 0}, and W; and b; represent weight matrices and bias vectors, respectively. The ReLU function evaluated pointwise on all entries of the input vector. In practice, the functional space FNN is selected compact set comprising all ReLU feedforward DNNs. This work investigates two distinct architectures within FNN. The first architecture within FNN is defined as follows:
FN = = [2,, fal"= for each & = I,..., (x( is in the form of 2
(3)
with L layers, width bounded by p, llfiello S M, WW K. |billoo Σ Ilwill 1bb S K},
||]] = max,, W.yl. |b|| = max; [ba] for any function f, matrix W, and vector b with || l/o denoting the number of nonzero elements of its argument. The functions in this architecture satisfy parameter bounds with limited cardinalities. The second architecture relaxes some of the constraints compared to the first architecture; i.e.,
where |f = max, [x)),,
Fd L.P. M) [I'
fal for each k = L,..., d, fa(x) is in the form of with layers, width bounded by p. fl < M}.
(4)
When there is no ambiguity, we use the notation FNN and omit its associated parameters.
We consider the following assumptions on the target PDE map b, the encoders Ex, EG,, the decoders D D and the data set S our theoretical framework.
Assumption 1 (Compactly supported measure). The probability measure 7 is supported on compact set Qx C. X. For any Ωx, there exists Rx such that ||u]]x Rx. Here, || l|x denotes the associated norm of the space X.
Assumption 2 (Tipschitz operator). There exists Lo 0 such that for any un u2 fxx
[[((w)) < & Lo ul - uzllx-
Here, || Iby denotes the associated norm of the space y
4

---

Published in Transactions on Machine Learning Research (10/2023)
Remark 1. Assumption 1 and Assumption 2 himlly that the images Φ(u) are bounded by Ry Lx for all u E Ωx. The Lipschitz constant T.p will be explicitly computed in Section 3 for different PDE operators.
Assumption 3 (Lipschitz encoders and decoders). The empirical encoders and decoders Ex Dx Ey, D' satisfy the following properties:
Fx (0x) = 0, (0) = 0x (3g) = D3(0) 0y,
where 0 denotes the zero vector and Ox, Oy denote the zero function in X' and V respectively. Moreover, assuine all empirical encoders are Lipschitz operators such that
[E Epuull2 Lg ||u uzlle, P = X,Y'
where || ll2 denotes the Euclidean L norm, || || denotes the associated norm of the Banach space P, and Lxg is the Lipschitz constant the encoder Ep. Similarly, we also assume that the decoders D%, P X are also Lipschitz with constants LDz.
Assumption (Noise). For I..... 2n, the noise e, satisfies
1. is independent of us;
B[ey] 0;
3. There exists 0 such that ||=/lly a.
Remark 2. The above assumptions on the noise and Lipschitz encoders imply that ((E((((u)) + E.) (B)(())) & Lgy.
2.2 Main Results
For a trained neural network NNN over the data set S, we denote its generalization error as
Egen (TNN) = EsE [I o NNN o (u) [
Note that omit its dependence on S in the notation. We also define the following quantity,
Enoise, = E..
[[ITT dx (u) u||ee] Eg [ITTS w1[ + o n-
where 1xx D' Ex and 4a DJ Ei denote the encoder-decoder projections X' and y respectively. Here the first term shows that the mucoder/decode projection error EsE, [ dx (w) =ll3 for X' amplified by the Lipschitz constant; the second term is the incoder/decoo projection error for )) the third term stands for the noise; and the last term small quantity n - will be shown later that this quantity appears frequently in our main results.
Theorem 1. Suppose Assumptions
4 hold. Let NNN be the minimizer of the optimization problem equa- tion with the network architecture FNN(dy, L,p, K, M) defined in equation with parameters
L In( n2 (m(=)))
dy ),), p=
dy
2+1x
n 24dx
V Ry, K = a(pL), k == ((M?) M
where the notation Q contains constants that solely depend on LEY. LDg. LE LD Rx and dx. Then there holds
stdx xनं (1 + I2-dx)
Eyen NN) I ddy
In3 dy
+ In2 +nn2n) n)
Enoise,
Here S contains constants that solely depend on Lgz, L.ng, L,g. L.Dz, Rx and dx-
5

---

Published in Transactions on Machine Learning Research (10/2023)
Theorem 2. Suppose Assumptions 1 hold ture. Let NNN be the minimizer of the optimization problem equation with the networrk urchileclure FuN (Ny.dy, I, M) defined in equation 4 with porameters,
4-dx x
M2 2 Vdybey Ry, Ry, and Lp 2
dy
n x
(5)
Then we have
btdx
Egen (TNN) S L2 log Lo dd 2ixx x log n Enoise,proj
(6)
where X contains constants that depend on dx, LEy LEY, Log. Luz and Rx-
Remark 3. The aforementioned results demonstrate that by selecting appropriate width and depth for the DNN, the generalization error can be broken down into three components: the generalization error of learning the finite-dimensional operator r. the projection error of the encoders /ecps////c.ddss.../// and the noisc. Comparing to previous results Liu et al. (2022) under the Hilbert space selting, our estimates show that the noise term in the generalization bound is non-decaying without the inner product structure in the Banach space setting, This is mainly caused by circumventing the mnnr-pproduct structure via triangle incqualitics in the proof. As the number of samples n increases, the generalization error decreases exponentially. Although the presence of dx in the exponent of the sample complexity initially appears pessimistic, will demonstrate that it can be eliminated when the imput data fix of the target operator exhibits low-dimensional data structure or when the target operator itself has a low-complexity structure. These assumptions are often satisfied for specific PDE operators with appropriate encoders. These results also imply that when dx is large, the neural network width does not need to increase as the output dimension dy increases. The main difference between Theorem and Theorem 2 lies in the different neural network architectures FNN L, p, K, K, M) and (dy p. M). As consequence, Theorem 2 has smaller asymptotic lower bound (2n of the neural network width p in the large dx regime, whereas the asymptotic lower bound is 22(n) in Theorem 1
Estimates with special data and operator structures
The generalization error estimates presented in Theorems 2 are effective when the input dimension dx is rel- atively small. However, in practical scenarios, it often requires mumerous bases to reduce the encoder decoder projection error, resulting a large value for dx- Consequently, the decay rate of the generalization error as indicated in Theorems 2 becomes stagnant due to its exponential dependence on dx-
Nevertheless, it often assumed that the high-dimensional data lie within the vicinity of low-dimensional manifold by the famous "manifold hypothesis.. Specifically, assume that the encoded vectors lie a dg-dimensional manifold with du & dx. Such data distribution has been observed in many applications, including PDE solution set, manifold learning, and image recognition. This assumption is formulated as follows.
Assumption 5. Let du < dxx E N. Suppose there exists encoder Ex & Rxx such that (Ex(u) u E 2x) lies in smooth dn-dimensional Riemannian manifold M that is isometrically embedded in Rdx, The reach Niyogi et al. (2008) of M is 1 0.
Under Assumption 5 the input data set exhibits a low intrinsic dimensionality. However, this may not hold for the output data set that is perturbed by noise. The reach of a manifold is the smallest osculating circle radius on the manifold. A manifold with large reach avoids rapid change and may be easier to learn by neural networks. In the following, we aim to demonstrate that the DNN naturally adjusts to the low-dimensional characteristics of the data set. As result, the estimation error of the network depends solely on the intrinsic dimension do, rather than the larger ambient dimension dx. We present the following result. to support this claim.
Theorem 3. Suppose Assumptions and Assumption 3 hold. Let NNN be the minimizer of the optimiza- tion problem equation 1 with the network architecture FNN (dy, T.,p, M) defined in equation 4 with parameters
L = Q(ī.og i) ((dxdy(logg) M 2 Vdyl. Ry
6

---

Published in Transactions on Machine Learning Research (10/2023)
where L, > O are integers such that Lp 2
dy
d n T=2ng
dp
Then we have
a+da 2440
Egen (T'Nn S Li (og(Lv) )
n 2fd log" + Emoise,
(8)
where the constants in S and 92(-) solely depend on do, logdx. Rx, LEy, LEg. Lax, LDg, T, the surface area of M.
It is important to note that the estimate equation 8 depends at most polynomially on dx and dy. The rate of decay with respect to the sample size no longer influenced by the ambient input dimension dx. Thus, our findings indicate that the CoD can mitigated through the utilization the 'manifold hypothesis. To effectively capture the low-dimensional manifold structure of the data, the width of the DNN should be on the order of O(dx). Additionally, another characteristic often observed in PDE problems is the low complexity of the target operator. This holds true when the target operator is composed of several alternating sequences of lew linear and nonlinear transformations with only small nunber of inpuls. We quantify the notion of low-complexity operators in the following context.
Assumption 6. Let 0 < do S dx. Assume there cxists Ex, Dx, Ey, Dy such that for any 2x we have
dyy (u) ogo Ex(u),
ogogo
where g Rdx + Ry is defined as
g(a) = [V a), ,ay (V2 a)]
= .....dy. See an where the matrix is V& E Rdy and the real valued function ge IR + R for illustration 44
In Assumption 6 when do = 1 and 91 -... = gdy g(a) is the composition of a pointwise nonlinear transform and a linear transform on a. In particular, Assumption holds for any linear maps.
Theorem 4. Suppose Assumptions 14 and Assumption 2 hold. Let I'NN be the minimizer of the optimiza-
tion problem with the network architecture FNN(dy, L, p. M) defined 4 with parameters
(1)
Lp= Ω
di
191200
n 4420
M 2 V /yy..Eg Ry
Then we have
B+dg
(T'n( S L3g log( log(Lo)dd 2-
aide log + projs
(9)
where the constants in S and 11-) solely depend on do. Rx Ry, LEg. LEg. LD LDg.
Remark Under Assumption our result indicates that the CoD can be mitigated to a cost ( - because the main task of DNNs learn the nonlinear transforms gl. :dy that are functions over 1R
In practice, a PDE operator might be the repeated composition of operators in Assumption O This motivates more general low-complexity assumption below.
A ssuumttion 7. Let 0 < di,..., de I dx and 0 < to...., I min (dx dy} with to dx and la = dy. Assume there exists Ex, Dx, Ey, Dy such that for any u E Ωx, we have
Hy o @(u) = Dy Gh o...o G¹ Ex(u),
where G :4:1 + R4 is defined as
(a) = [oi((V ............ ((V))
where the matrix is V ddx and the real valued function is 9) Rd R for j = 1, i 1,
See an illustration in 45

---

Published in Transactions on Machine Learning Research (10/2023)
Theorem 5. Suppose Assumptions 1 and Assumption 7 hold. Let INN be the mindmizer of the optimiza- loon 1 with the nelwork urchilecture FNN (oyddy, kT,, M) defined in equation with porameters
Tp= 2
dy
Andmus max n1 4+
Amax
M 2 Ey Ry,
V
where dmnex = max(d;) and lmax = max
Then we have
Stdmax
Egen (T'NN) I Lig log(Lo maaz n 2idmer log ri + Enoise,
where the constants in S and (2(-) solely depend on k. dmaas manx Rx, LEy, LEy, LDz. LDg.
Discretization invariant neural networks
In this subsection, we demonstrate that our main results also apply to neural networks with the discretization invariant property. neural network is considered discretization invariant it can be trained and evaluated on data that are discrctized in various formats. For example, the input data u, u,i= I...., n may consist of images with different resolutions, u, (11))))....) (x,)), representing the values of uy sampled different locations. Neural networks inherently have fixed input and output sizes, making them incompatible for direct training on data set {(u: v:), i .n....n] where the data pairs (u, E Rdi different resolutions d,, = 1,..., n. Modifications of the encoders are required to map inputs of varying resolutions to a uniform Euclidean space. This can be achieved through linear interpolation or data-driven methods such as nonlinear integral transforms Ong et al. (2022).
Rd) have
Our previous analysis assumes that the data (u,, 14) X y mapped to discretized data (u,, vz) E Rdx dy using the encoders FX and Eÿ- Now, let consider the where the new discretized data (u,, v.) E R R vectors tabulating function values as follows:
u; = [u,(=f) (x(x))
u,(x1,)] v. = [o=(=t) (;(xb)
(10)
The sampling locations x² = l=f....,.",, s., are allowed to be different for each data pair (u,, v.). We can now define the sampling operator on the location x² as Px u + u(x'), where u(x') [u(af) u(zz) ... u(x²,)] For the sake of simplicity, spaced grid points, denoted as 8, (r, + 1)d, where + 1 represents the number of grid points in each dimension. To achieve the discretization invariance, we consider the following interpolation opera- tor Tx. u(x') + û, where ū represents the multivariate Lagrangian polynomials (refer to Leaf & Kaper (1974) for more details). Subsequently, we map the Lagrangian polynomials to their discretization on a uniformly spaced grid mesh x Rdx using the sampling operator Px. Here dx = (r + 1)d and r is the highest degree among the Lagrangian polynomials û. We further assume that the grid points x of all given discretized data are subsets of &. We can then construct discretization-ivvariat encoder as follows:
assume that the sampling locations are equally
Ex = Px o lx Px..
We can define the encoder Fly a similar manner. The aforementioned discussion can be summarized the following proposition:
Proposition 1. Suppose that the discretized data {(u, =I..._, ,n} defined in equation 10 are images of sampling operator Pa applied to smooth functions Vi), and the sampling locations x² are equally spaced grid points with grid size h. Let â E Rdx represent equally spaced grid points that are denser than all (r+1)a. Define the encoder Ex= = E' Pao Ly o Pas, and decoder D'x = D2 = l
x', = 1....,,,, n with dx = =x= Then the encoding error can be bounded as the following:
Ey [ (w) ()) w < Ch2 Ch2 Eg [IITT dy (v) v S Ch2 Ch1"1101122111
(11)
where C is an absolution constont, I''d = D' o E' and Hy DJ, Ezy.
Proof. This result follows directly from the principles of Multivariate Lagrangian interpolation and Theorem 3.2 in Leaf & Kaper 1974).
8

---

Published in Transactions on Machine Learning Research (10/2023)
Remark 5. To simplify the analysis, we focus the - norm in equation However, it is worth noting that TP estimates can be easily derived by utilizing TP space embedding techniques. Furthermore, C" estimates can be obtained through the proof of Theorem 3.2 in Leaf & Kaper (1974). By observing that the discrctization invariant encoder and decoder in Proposition I satisfy Assumption 2 and Assumption I we can conclude that our main results are applicable discret ization invariant neural networks. In this section, we have solely considered polynomial interpolation encoders, which require the input data to possess a sufficient degree of smoothness and for all training data to be discretized on a finer mesh than the encoding space Rdx. The analysis of more sophisticated nonlinear encoders and discretization invariant neural networks is topic for future research.
In the subsequent sections, we will observe that numerous operators encountered in PDE problems can be expressed as compositions of low-complexity operators, as stated in Assumption or Assumption 7 Consequently, deep operator learning provides mcans to alleviate the curse of dimensionality, confirmed by Theorem or its more general form, as presented in Theorem 5
3 Explicit complexity bounds for various PDE operator learning
In practical scenarios, enforcing the uniform bound constraint in architecture (3) is often inconvenient. As result, the preferred implementation choice is architecture 4 Therefore, in this section, we will solely focus on architect ure ( In this section, we will provide five examples of PDEs where the input space X' and output space y are not Hilbert. For simplicity, we assume that the computational domain for all PDEs is = 1].. Additionally, we assume that the input space X' exhibits Hölder regularity. In other words, all inputs possess bounded Hölder norm || llce, where
0. The Hölder norm is defined |[ʃ|ec- || + max flcn.,, where & k a, k is integer, 0 < < and | Ic represents the a-Hölder semi-norm Iflco.- = supryy [(x))/(w)), It can be shown that the output space y also admits ||-vv-
Hölder regularity for all examples considered in this section. Similar results can be derived when both the input space and output space have Sobolev regularity. Consequently, we can employ the standard spectral method as the mccoer/decooes for both the input and output spaces. Specifically, the encoder E maps u E X' to the space Ph: which represents the product of univariate polynomials with a degree less than r. As result, the input dimension is thus dx = dimpy rd We then assign LP-norm (p > 1) to both the input space X' and output space y. The encoder/decoder projection error for both X' and y can be derived using the following lemma from Schultz 1969).
Lemma 1 (Theorem 4.3 (ii) of Schultz 1969 Let integer k 2 0 and 0 < 1. For any e 1]]( with k + a, denote by f its spectral approzimation in PT, there holds
Il - fl1zo S Callffleorr*
We can then bound the projection error
II' dx u u al(p((-1111)) J-1.1-- lu dx 02241uulB.r 10
Therefore,
[ dx xdx -ul-ull2 11l2 I C2220/0 1 1u
(12)
Similarly, we can also derive that
[1 (w) w/ w/l 1 S C222d/011
Iull # L3.
(13)
given that the output w = (u)) is in Ct for some t t>0 0.
In the following, we present several examples PDEs that satisfy different assumptions, including the low- dimensional Assumption 5 the low-complexity Assumption and Assumption 7 In particular, the solution operators of Poisson equation, parabolic equation, and transport equation are linear operators, implying that Assumption G is satisfied with g;'s being the identity functions with do 1. The solution operator of Burgers
9

---

Published in Transactions on Machine Learning Research (10/2023)
equation is the composition of multiple numerical integration, the pointwise evaluation of an exponential
function 03 (+) = exp(-), and the pointwise division (a b) = a/b. It thus satisfies Assumption 7 with d1 = and d2 = 2. In parametric equations, we consider the forward operator that maps a media function a(z)
to the solution u. In most applications such forward maps, the media function a(z) represents natural images, such as CT scans for breast cancer diagnosis. Therefore, is often assumed that Assumption holds.
3.1 Poisson equation
Consider the Poisson equation which seeks u such that
=
(14)
where x E Rd, and [u(x)] + O as la]| + co. The fundamental solution of equation 14 is given as
In|a for d = 2, for d 2 3,
(x)
|2|2-d
wa
where wa is the surface area of a unit ball in R4. Assume that the source f(x) is smooth function compactly supported in Rd. There exists a unique solution to equation given by u(x) = f Notice that the solution map f +> u is a convolution with the fundamental solution, u(x) v * f. To show the solution operator is Lipschitz, we assume the sources f.g E C* (Rd) with compact support and apply Young's inoquality to get
- | (Ra) = I|D (anp(lulllll - ()) = 1| D* () S | - o
(15)
where q.q 2 1 so that 1/p + 1/9- = Here Ω is the support of f and g.
For the Poisson equation 14h on an unbounded domain, the computation is often implemented over a truncated finite domain Q. For simplicity, we assume the source condition f is randomly generated in the space (*) from random measure y. Since the solution is convolution of source f with smooth kernel, both f and u are in (I2).
We then choose the encoder and decoder to be the spectral method. Applying equation 12 the encoder and decoder error of the input space can be calculated as follows
B [TT dx (f) - flli=(as) I Cd.dx E, [I [1)))
Similarly, applying Lemma and equation 15 the encoder and decoder error of the output space is
EsRyyy [ (u) uI [1]2 I Capdy 2 F,, [| sllén()) s Cap.ndy 2 F,, [I1/IlEH(o]]
Notice that the solution u(y) - See (y x)) fx) (z) is a linear integral transform of f, and that all linear maps special cases of Assumption with y being the identity map. In particular, Assumption 0 thus holds true by setting the column vector Vx the mumerical integration weight of V yk), and setting 9x 's as the identity map with du = 1 for k = 1.... ...... .dy By applying Theorem 4 we obtain that
,d.
EsLE DD's o INNN EX - + S nd 2/3 logn (a² +n 11 +r - [IL]TE<())
(16)
where the input dimension dx = dy = rd and S contains constants that depend on dx, d, p and [2).
Remark 6. The above result equation 16 suggests that the generalization error is small if we have a large number of samples, a small noise, and a good regularity of the input samples. Importantly, the decay rate with respect to the number of samples is independent from the encoding dimension dx or dy.
3.2 Parabolic equation
We consider the following parabolic equation that seeks u(x t) such that
{
u - An = in Rdd x (0, oo),
(17)
u u=9 on Rd x (t - 0}.
10

---

Published in Transactions on Machine Learning Research (10/2023)
The fundamental solution to equation 17 is given by A(r,t) = (4xt)-d/22= d/2e- - "." for x E R > 0. The solution map g()) H -( can be expressed as a convolution with the fundamental solution u(-; 7) = AC,, T) * g. where T the terminal time. Applying Young's inequality, the Lipschitz constant is ||A(-) where can explicitly calculate this number in 3D as IA(,TT)l parabolic equation (170, we consider truncated finite computation domain Q x [0, and assune an initial condition C* (S2). Due to the similar convolution structure of the solution map compared to the Poisson equation, we can obtain a similar result by applying Theorem
1 S co. As an example,
p.. For the
E NN - ((9) & n
2/3 lognt logn+(o²2 + + E [10
(18)
where the encoding dimension dx = dy = r the symbol "S" denotes that the expression on the left-hand side is bounded by the expression on the right-hand side, where the constants involved depend on dx, d, p, and [S]. The reduction of the CoD in the parabolic equation follows a similar approach as in the Poisson equation.
3.3 Transport equation
We consider the following transport equation that seeks a such that
[n + a(x) Vu = 0 in (0, Red x(0,,)) uo(2) in Rd
(19)
where a(x) is the drift force field and u1(x) is the initial data. For convenience, we assume that the drift force field satisfies (ODE), we consider the initial value problem dt a(x(()), x(0) = x, which admits a unique solution for any x Rd, f + xt)) (x( E C² (R; Applying the Characteristic method, the solution of equation 19 is given by 2t) == (P (x))) If we further assume that un is randomly sampled with bounded H norm, > 3d 2, then by Theorem 5 of Section 7.3 of Evans (2010), we have u E (10, oc); Rd). More specifically, we have
(²( n (..)(( By employing the classical theory of ordinary differential equations
dzit)
| (BC) I (Re))
where Ca,T, > 0 is constant that depends on the media a, terminal time T,, and the support Ω of the initial data. Since the initial data has C* regularity, by equation 12 the encoder /eccoer projection error of the input space is controlled via
Euo [ (uo) - uo 10122 s Cap.ndx $ E, [10018400)
Similarly, for the projection error of the output space, we have
EsEuwber [ITT (u) u ullirco)]
Cd. $ Euo [Iu(TT]]e= s Capa.r $ Buo [1uollir
We again use the spectral ancoder/dccodee so dx = dy rd Notice that solution u(T,x) uol (x)) is a translation of the initial data un by p which is linear transform. Let V Rdx be the corresponding permutation matrix that characterizes the translation by PP then VTT is the k-th row of V. Then by setting 9i'ss 's as the identity map, Assumption t holds with do = 1. Apply Theorem g to derive that
EsE.IIDD NNN (u) - (1)) S 3d 2/3 logn + (0² + n nnn-1)) +r -E, [1tolle= + || (00)
(20)
where < contains constants that depend on d. p. a, I and Ω. The CoD in transport equation is lessened according to equation 20 in the same manner as in the Poisson and parabolic equations.
11

---

Published in Transactions on Machine Learning Research (10/2023)
3.4 Burgers equation
We consider the 1D Burgers equation with periodic boundary conditions:
Ue + uur KUrF in R x (0, oc),
u(x, 0) - =0((x) uu-m1,) -, t) = t)
(21)
where K > 0 is the viscosity constant, and we consider the solution map uo(-) + uTT.)). This solution map can be explicitly written using the Cole-Hopf transformation -2xes where the function is the solution to the following diffusion equation
KUZr
F( 0) =o(x) = exp (- S= no(s)ds)
The solution to the above diffusion equation given by
v(x -2x JR O,KC(a, T) (ooly)dy Jz (.
(22)
where the integration kernel K. is defined K(,,,, 1) VARNt exp shock formed in the solution of viscous Burger equation, the solution may form a large gradient in finite time for certain initial data, which makes it extremely hard to be approximated by NN. We assume that the terminal time T' is small enough so a large gradient is not formed yet. In fact, it is shown in [eywood & Xie (1997) (Theorem 1) that if T S C'Iuool H then [|u[[ I) S C']luollI We then assume an initial data uo randomly sampled with uniform bounded H' norm. By Sobolev embedding, we have
(a- y)²
Although there will be
4mt
| S Clluullmm T)lcoaaa S C C'Juollmt,
By 12 we can control the mncoder/decodee projection error for the initial data
E Eus d (wo) uolle-(0) s Ca En. [1luo]]
Since the terminal solution u(,)T) has same regularity as the initial solution, by 13 we also have
Euo [ (u( u(, < Candyy' Eng [llu(, 1)1l S Ca Bug [ll
Similarly, we can choose dx = dy = r. The solution map is a composition of three mappings uo + up, 00 + o(: T) and T), - 7). More specifically, vo(x)) exp (- 1 J uo(s)ds) so we can set V Rdxx1 as the numerical integration vector on [-T, mz) and (x) = exp( for all 1....,ddy For the second mapping Up ++ v(.T) T) (c.f. 22), we set V22 E Rdxx2 where the first row is the numerical integration with kernel O,K and the second row the numerical integration with kernel K,, and we let 92(,,))
2x
=2s for all k = 1.... ,dx. For the third mapping u -2x0, we can set V E Rdxxx2 where the first row is the k-row of the numerical differentiation matrix, and the second row is the Dirac-delta vector at Iv. and we let (2 y) =2xe for all & = 1,... ,dy. Therefore, Assumption 7 holds with dmax = and mmax = dx = dy = r. Then apply Theorem to dcrive that
Es NN (uo) u(-,T) S 5/2 1/2 logn + (a² + 1 + r- r-'E, [Iuol]rr<(0]]
(23)
where & contains constants that depend on p.r and T. The CoD in Burgers equations is lessened according to equation 23 well as in all other PDE examples.
12

---

Published in Transactions on Machine Learning Research (10/2023)
3.5 Parametric elliptic equation
We consider the 2D elliptic equation with heterogeneous media in this subsection.
-dv(((()VVau<)) 0, in Q CR,
(24)
u f. on 02.
The media coefficient a(z) satisfies that S a(z) < 3 for all z E Ω where and 3 are positive constants. We further assume that a(x) E (2)). We are interested NN approximation of the forward map + : a + u with a fixed boundary condition f, which has wide applications in inverse problems. The forward map is Lipschitz, see Appendix A.2 We apply Sobolev embedding and derive that u E (2)1/2(1). Since the parameter a has C regularity, the sucoder/decoded projection error of the input space controlled
|1 ax - allix0)] I C2dx EE, [Iaaéva]]
The solution has? Hölder regularity, so we have
Eg EE [
(u) - ulli-(0)) Eg a (w) (oo)-wllennn ul( Cydy & Eng [IulBonracs] S Cpa.8.
We use the spectral encoder /ecoder and choose dx dy r2. We further assume that the media functions a(x) are randomly sampled on smooth do-dimensional manifold. Applying Theorem 3 the generalization error is thus bounded by
Es DD' o NNN o (+) (w))lB S dz
TT n ममक log n + (o² + n=) r--r- EE [Iaale +r-
where S contains constants that depend p, S2 a, and f. Here do is constant that characterized the manifold dimension of the data set of media function a(x). For instance, the 2D Shepp-Logan phantom Gach et al. (2008) contains multiple ellipsoids with different intensities thus the images in this data set lies on a manifold with a small dg- The decay rate in terms of the number of samples n solely depends on dg. therefore the CoD of the parametric elliptic equations is mitigated.
4 Limitations and discussions
Our work focuses on exploring the efficacy of fully connected DNNs surrogate models for solving general PDE problems. We provide an explicit estimation of the training sample complexity for generalization error. Notably, when the PDE solution lies in low-dimensional manifold or the solution space exhibits low complexity, our estimate demonstrates a logarithmic dependence on the problem resolution, thereby reducing the CoD. Our findings offer a theoretical explanation for the improved performance of deep operator learning in PDE applications.
However, work relics on the assumption of Lipschitz contimuity for the target PDE operator. Conse- quently, estimates may not be satisfactory if the Lipschitz constant is large. This limitation hampers the application of our theory to operator learning in PDE inverse problems, which focus on the solution- to-parameter map. Although the solution-to-parameett map is Lipschitz in many applications (e.g., electric impedance tomography, optical tomography, and inverse scattering), certain scenarios may feature an expo- nentially large Lipschitz constant, rendering our estimates less practical. Therefore, our results cannot fully explain the empirical success of PDE operator learning in such cases.
While our primary focus is on neural network approximation, determining suitable encoders and decoders with small encoding dimensions (dx and dy) remains challenging task that we did not emphasize in this work. In Section 2.2 we analyze the naive interpolation as a discretization invariant encoder using fully connected neural network architecture. However, this analysis is limited lo cases where the training data is sampled on an equally spaced mesh and may not be applicable to more complex neural network architectures or situations where the data is not uniformly sampled. Investigating the discretization invariant properties of other neural networks, such as IAE-net Ong et al. (2022). FNO Li et al. (2021), and DeepONet, would be interesting avenue for future research.
13

---

Published in Transactions on Machine Learning Research (10/2023)
Acknowledgements
K. C. and H. Y. were partially supported by the US National Science Foundation under awards RMSS-2063333 and the Office of Naval Research Award N00014-23-1-2007. C. W. was partially supported by the National Science Foundation under awards DMS-2136380 and DMS-2206332.
References
Ben Adcock, Simone Brugiapaglia, Nick Dexter, and Sebastian Moraga. Near-optimal learning of banach- valued, high-dimensional functions via deep neural networks. arv preprint or Xrxi::2211.100 2022.
Martin Anthony, Peter L Bartlett, Peter L Bartlett, et al. Neural network learning: Theorelical foundations, volume 9. cambridge university press Cambridge, 1999.
Andrew R Barron. Universal approximation bounds for superpositions of sigmoidal function. Transactions on Information theory, 39(3):930-94) 945, 1993.
1EEE
Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Niarly-tight ve-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, 20(1):2285-22 2201,, 2019.
Benedikt Bauer and Michael Kohler. On deep learning as a remedy for the curse of dimensionality in nonparametric regrossion. The Annals of Statistics, 27(4):2261-222) 2019.
Sergio Botelho, Ameya Joshi, Biswajit Khara, Vinay Rao, Soumik Sarkar, Chinmay Hegde, Santi Adavani, and Baskar Ganaapauhysubrrmmniian Deep generative models that solve pdes: Distributed computing for training large data-free models. In 2020 IEEE/ACM Workshop on Machine Learning in High Performance Computing Enwironments (MLHPC) and Workshop on Artificial Intelligence and Machine Learning for Scientific Applications (A14S), pp. 50 63. IEEE, 2020.
Shengze Cai, Zhicheng Wang, Lu Lu, Tamer A Zaki, and George Em Karniadakis. Deepm&mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks. Journal of Computational Physics, 436:110296, 2021.
Minshuo Chen, Haoming Jiang, Wenjing Liao, and Tuo Zhao. Efficient approximation of deep relu networks for functions on low dimensional manifolds. Advances in neural information processing systems, 32, 2019.
Minshuo Chen, Haoming Jiang, Wenjing Liao, and Tuo Zhao. Nonparametric regression on low-dimensional manifolds using deep relu networks: Function approximation and statistical recovery. Information and Inference: Journal of the IMA, (1)4:1203-1155) 2022.
Abdellah Chkifa, Albert Cohen, and Christoph Schwab. Breaking the curse of dimensionality sparse polynomial approximation of parametric pdes. Journal de Mathématiques Pures Appliquées, 103(2): 0000-28,,40 2015.
Alexander Cloninger and Timo Klock. Relu nets adapt to intrinsic dimensionality beyond the target domain. arXiv preprinl Xix:2008 02545, 2020.
George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 2(4):303 314, 1989.
Maarten V de Hoop, Nikola B Kovachki, Nicholas Н Nelsen, and Andrew M Stuart. Convergence rates for learning linear operators from noisy data. arXiv preprint arXiv:2108. 12515, 2021.
Mo Deng, Shuai Li, Alexandre Goy, Iksung Kang, and George Barbastathis. Learning to synthesize: robust phase retrieval at low photon counts. Light: Science Applications, 961):1-16, 2020.
Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
14

---

Published in Transactions on Machine Learning Research (10/2023)
Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and inference. Econometrion, (9(1):181-21) 213,,11,,-00 2021.
H. Michael Gach, Costin Tanase, and Fernando Boada. 2d & 3d shepp-logan phantom standards for mri. In 2008 19th International Conference on Systems Engineering, pp. 521-520 TEFE, 2008.
Alex Graves, Abdel-rahman Mohamed, and GeofTrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference acoustics, speech and signal processing, pp. 6645-6649, Iece, 2013.
John G Heywood and Wenzheng Xie. Smooth solutions of the vector burgers equation in nonsmooth domains. Diferential and Integral Equations, 10(5):961 974, 1997.
Kurt Hornik. 1991.
Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251 --7
Teerator Kadeethum, Daniel O'Malley, Jan Niklas Fuhg, Youngsoo Choi, Jonghyun Lee, Hari S Viswanathan, and Nikolaos Bonklas. A framework for data-driven solution and parameter estimation of pdes using conditional generative adversarial networks. Nature Computational Science, 1 829, 2021.
Yuehaw Khoo and Lexing Ying. Switchnet: neural network model for forward and inverse scattering problems. STAM Journal on Scientific Computing, (165):33182-33201 2019.
Michael Kohler and Adam Krzyčak. Adaptive regression estimation with multilayer feedforward neural networks. Nonparametric Statistics, 17(8):891 913, 2005.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art-No, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for deeponets: A deep learning framework in infinite dimensions, Tronsactions of Mathematics and Its Applications, 6(1):tnac001, 2022.
Gary K Leaf and Hans Kaper. -erorr bounds for multivariate lagrange approximation. SIAM Journal on Numerical Analysis, 11(2):363 381, 1974.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizadencsheli, Burigede liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Represenlations, 2021. URL httpp::/oppaareriieeneee//eee Id=c8P9NQVtmn0
Chensen Lin, Zhen Li, Lu Lu, Shengze Cai, Martin Maxey, and George Em Karniadakis. Operator learning for predicting multiscale bubble growth dynamics. The Journal of Chemical Physics, 154(10):104118, 2021.
Hao Liu, Minshuo Chen, Tuo Zhao, and Wenjing Liao. Besov function approximation and binary classifica- tion low-dimensional manifolds using convolutional residual networks. In International Conference Machine Learning, pp. 6770-6780. PMLR, 2021.
Hao Liu, Haizhao Yang, Minshuo Chen, Tuo Zhao, and Wenjing Liao. Deep nonparametric estimation of operators between infinite dimensional spaces. arXiv preprint wxXim:2211002.1, 2022.
Jianfeng Lu, Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation for smooth functions. SIAM Journal on Mothematical Analysis, 53(5):5465 5506, 2021a.
Lu LU, Pengzhan Jin, Guofei Pang. Zhongqiang Zhang, and George Em Karniadakis. Learning nonlincar operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218 229, 2021b.
15

---

Published in Transactions on Machine Learning Research (10/2023)
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, 2022.
Chao Ma, Lei Wu, et al. The barron space and the flow-induced function spaces for neural network models. Constructive Approzimation, 55(1):369 406, 2022.
Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley. Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics, 19(6):1236 1246, 2018.
Ryumei Nakada and Masaaki Imaizumi. Adaptive approximation and genoralization of deep ncural network with intrinsic dimensionality, J. Mach. Learn. Res., 21(144): 38, 2020.
Nicholas H Nelsen and Andrew M Stuart. The random feature model for input-output maps between banach spaces. SIAM Journal on Scienlific Compuling, 43(5):A3212 A3243, 2021.
Partha Niyogi, Stephen Smale, and Shmuel Weinberger. Finding the homology of submanifolds with high confidence from random samples. Discrele & Computalional Geomelry, (9(1):419944) 2008.
Yong Zheng Ong, Zuowei Shen, and Haizhao Yang. Integral autoencoder network for learning. The Journal of Machine Learning Research, 23(1): 12996-13000, 2022.
Benjamin Peherstorfer and Karen Willcox. Data-driven operator inference for nonintrusive projection- based model reduction. Computer Methods Applied Mechanics and Engineering, 306:196-2143 2016.
Chang Qiao, Di Li, Yuting Guo, Chong Liu, Tao Jiang, Qionghai Dai, and Dong Li. Evaluation and development of deep neural networks for image super-resolttio in optical microscopy. Nature Methods, 18(2):194 202, 2021.
Md Ashiqur Rahman, Manuel A Florcz, Anima Anandkumar, Zachary E Ross, and Kamyar Azizzadenesheli. Generative adversarial neural operators. arXiv preprint arXiv:2205.0801 2022.
Johannos Sehmidt--Hibbes Deep relu network approximation of functions manifold. arXiv preprint arXiv:1908.0063 00695, 2019.
Johannes Schmidt-HHebbr Nonparametric regression using deep neural networks with relu activation func- tion. The Annals of Statistics, 48(4):1875-1899 2020.
Martin Schultz. /*-multivariate approximation theory. SIAM Journal on Numerical Analysis, 6(2):
161-183, 1969.
Zuowei Shen, Haizhao Yang, and Shijun Zhang, Deep network approximation characterized by number of neurons. arXiv preprint arXic:1906.0544 05497, 2019.
Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network with approximation error being reciprocal of width to power of square root of depth. Neural Computation, 33(4):1005 1036, 2021.
Charles J Stone. Optimal global rates of convergence for nonparametric regression. The annals of statisties, pp. 1040 1053, 1982.
Taiji Suzuki. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. arXiv preprint ar Xiv:11880. 180.0003,,,,,,,, 2018.
Chunwei Tian, Lunke Fei, Wenxian Zheng, Yong Xu, Wangmeng Zuo, and Chia- Wen Lin. Deep learning on image denoising: An overview. Newral Networks, 131:251 275, 2020.
Ting Wang, Petr Plechac, and Jaroslaw Knap. Generative diffusion learning for parametric partial differential equations. arXie preprint arXiv:2305.1 2023.
Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103 114, 2017.
16

---

Published in Transactions on Machine Learning Research (10/2023)
Dmitry Yarotsky. Optimal approximation of continuous functions by very deep relu networks. In Conference on learning theory, pp. 639-649 649. PMLR, 2018.
Dmitry Yarotsky, Elementary superexpressive activations. In International Conference on Machine Learning. pp. 11932-11 11940. PMLR, 2021.
17

---

Published in Transactions on Machine Learning Research (10/2023)
A Appendix
A.1 Proofs of the main theorems
Proof of Theorem 1 The L squared error can be decomposed as
Eg [I NNN (x( (1))1
<2EgBuw D3 o NN o (u(() - Dÿ o Eÿ (3) + +225 [I a EG B(u) +(2) [5]
where the first term I = 2EsBuwy D3 NN (x) D3 Eÿ o (1))1] is the network estimation error in the D' space, and the second term I1 21 [1D3 EB x(u) - is the empirical projection error, which can be rewritten as
II 2E5 Een # [|[TT dy (w) w133]
(25)
We aim to derive an upper bound of the first term I. Finst, note that the decoder D3 is Lipschitz (Assumpption
We have
= En [ID NNN Ex(()) Dÿ E3 (())l
S by EsEuwy [[]TNN o Ef(() E3G o ((u))(²).
Conditioned on the data set SJ, we can obtain
Esg E Bunry
[I
F (u)
F3 (
2n
=2R
[ NN (u:) Fi 3
n
i=++1
(26)
+ E.s, Buny
[[TNN e (u) - E3 (1)) 2Es,
[ o (xs))) EG $(u=)11
i=n+1
-Ii + T2:
where the first lerm Ti 2Rs, E NNN (xs))) - EB o ((u))l includes the DNN ap-
proximation error and the projection error in the X' space, and the second term Tr Es, Eus, [| (u) Ez +(m)) Ti captures the variance.
To obtain an upper bound of Ti, we apply triangle inequality to separate the noise from Ti
2n Σ
2n
Ti S 2Esa
NNN (x())) (o:)11
2ES,
Σ
E
+(u,)
E3(1:)1
72
71
Using the definition of I'NN, we have
Ti < 21s,,
inf Σ I'ENN im++1
2n
To Ex -
+ 21 Rsa
1
2n Σ
i
Using Fatou's lemma, we have
2n
2n
Ti S 4Es;
inf
Σ
[TE - ES o ((u))
+ 77 Es n
Σ
||
LEF n
imn+1
imn+1
2n
2n
< 4 inf Esp rEFNN
[o - E3 o #(u))1
+ 611 Esa
||
(27)
i=n+1
im++1
2n
1
4 inf En [ (u) TEFxx
Eÿ {(1)]|] + + Es²
Σ
1|E
n
i=++1
18

---

Published in Transactions on Machine Learning Research (10/2023)
To bound the first term the last line of Equation equation 27 consider the discrete transform I
F + DX E FNN. Note that it is a vector filed that maps Rdx to Rdy, and by Assumption 1 2 and
each component h(-) is a function supported on [-B, Bjáx with Lipschitz constant M = LDR Lg
where B- = Rxb This implies that each component h has an infinity bound || S R = BM LDz Lo LoLERXXLE
We now apply the following lemma to the component functions of r
Lemma 2. For any function f ([-1, ()), and E (0, 1), we assume that Il S There exists a function f FNN(l1, K, K, M) such that
- flloo <ee
flloo
where the porameters of FNN chosen
L -((( + d) Inc" + + Ind dd),
p p= -= =d- (2)d(/)). /n),
(28)
K 1 f K (MM3), M Oldd n).
Here all constants hidden in 22-)) do not dependent on any parameters.
Proof. This is a direct consequence of proof of Theorem in Yarolsky 2017 for Fr.d-
Let h4 : IRdx + RR = I,....dd dy be the components of 17 then apply Lemma 2 to the rescaled component
1 Sh,(B (B) -) with n = 1. It can be derived that there exists h2 E FN, L, p. K, K, M) such that R
h,())))) - hz(x)) S E
max xE[-1,1]6x
R
with parameters chosen as in equation 28 with n 1, dx, and E. Using a change of variable, we obtain that
[h((x) Rh. I S R
ME| DE|-BA B
B
Assembling the neural networks Rha( ( together, obtain an neural network i'g FNN(dd, L, K, K. M) with p dyō, such that
[ - lo S E1,
(29)
Here the parameters of Fxn(dy, L,p, K, M) are chosen as
L R(dx In " p p= -(dys dx Li dx 20))
(30)
K = 0(pL), K = S(M3). M 2 VdyLEy Ry.
Here the constants in Ω may depend on LDz. LEy LEy and Rx. Then we can develop an estimate of Ti follows.
inf Ey o Ex) E3 (u)
PEFNN
<En [|| (u) Eÿ (1)]
(31)
<2E, [ o (x) m3 Ex (w)l 2E, [[T Ex(u) Eg o (1))
<2dye + 2F [ o (x) ES o (
where we used the definition of infinimum in the first inequality, the triangle inequality in the second inequality, and the approximation equation 29 in the third inequality. Using the definition of Φ, we obtain
inf E. [|[ E (w) EG (1)))
rEFNN
=2dye + 2E, [[ o t D3 (x) - ES ((u)
(32)
<2dye + +2I L&E,E, [I Ex(u)-u13) -u13]]
2dyye + 22 12 E, [ (u) -u13]]
19

---

Published in Transactions on Machine Learning Research (10/2023)
where we used the Lipschitz continuity of & and E in the inoquality above. Combining equation 32 and equation 27 and apply Assumption 4
we have
Ti < Sdysi Ey E.
xx,xx (u) =11xx| + gy
(33)
To deal with the term T2, we shall use the covering number estimate of FNN (Ny(dy, L. K,K, M), which has been done in Lemma 6 and Lemma in Liu et al. (2022) direct consequence these two lemmas is
35dy Log RB log.
T2 -
FNN 1l 1l00 |loo)
+ +6
n
4dyL
dz K
(Ind + + In(BB) + L + Lmpp) +
o 1 L3
(Ino~1+ +I(B)) + L Inx + Llup) +
where we used Lemma and from Liu et al. (022) for the second inequality. The constant in X depends on Ey and Rx. Substituting parameters K, B, x from equation 30 the above estimate gives
Ta S Lyddyn pL (In 8- LLnB Lnn + Linp) +
X LB L2 pL + -2) +
2 Li - (L'' (Ino-1)3) +
where used the fact Ind-1 S L with the choice equation 34
The constant in X depends Leg, Log, LEg Log, Rx and dx. We further substitute the values of and L from equation 30 into the above estimate
T2 T L3
12-dx dzn -eTdx ( (In (In6-1)3) (1n +
Combining the Ti estimate above and the T3 estimate in equation 33 yields that
Ti + T2 Sdye + T2z 1 ((In 13 + (In )2))
En [||TT ax (u) u1122] 0² 8.
In order to balance the above error, we choose
8=n
E1 = dy. x n 240x.
(34)
Therefore,
Gidx
Ti + T2 Sdy
214x
7
I=ux (1 + L2-dxx
(In ") ) + (lan)²
dy
(35)
+ E
II dx (u)
u u112 + 2 + n
where we combine the choice in equation
34
and equation 30 as
dx n2 )), p= p= Q(d L = 2(n(
-T n =Tdx
dr
K = M(pL), k 9(M3) M VdyLEy Ry
Here the notation Ω contains constants that depends on Lgy. Lng. LE Luy. Rx and dx.
Combining equation 25 and equation
we have
Gtdx
EsEnwy [I o TNN o Ex(4) (n))
2T
(1 + 1.2 2- dx)x))
(an T
+ (Inm)²
L E, [1 (u) w 2112 + Es Ewwwrs [ w113]]
0² + -1
20

---

Published in Transactions on Machine Learning Research (10/2023)
Proof of Theorem Similarly to the proof of Theorem 1, we have
Es D3 o INN o (x) +(u)l} - I1+11,
and
1 DD (Ti + T2),
where Ti and T2 are defined in equation 26 Following the same procedure in equation 27
we have
T S 4 inf E. TEFNN
[[T o (x) - EG ((4)l 6ES,
|
i=n+1
To obtain an approximation the discretized target map r = FJ o $ o D2, we apply the following lemma for each component function T7.
Lemma 3 (Theorem 1.1 in Shen et al. (2019)). Given f E C(1 11 for any L PENt there exists function implemented by ReLU PNN with width 3143 max (el j p + 1}, and depth 12L 14 2d such that
If - ~l & 19 2/d L
where wy(+) the modulus of continuity.
Apply Lemma to each component. h2 of d can find neural network h, E FNN(1,, L, M) such that
h hilloo S CLaE1,
where L 0 are integers such that Lp 1x- and the constant C depends dx. Assembling the neural networks hy together, can find neural network In in FxN(dy, M) with dyō, such that
F - 17 S ClaE1.
Similarly to the derivations in equation equation 31 and equation 32 we obtain that
Ti 5 dy
L
|| dx (x) (u) - u1f + t²
(36)
where the nolation I contains constants that depend on dx and Lug. To deal with term T2, we apply the following lemma concerning the covering number.
Lemma [Lemma 10 in Liu al. (022
Under the conditions of Theorem 2, we have
T2 35dy R2
8 Ly Ry
log
FNN, 2n 2n) + 68.
Combining Lemma
with equation 36 we derive that
I ICCLLL d + 16L Dy. L E. [ (u) - ul/2]] + 12L
70L 70 dy
(37)
log N
FNN (dy, L, M) 2n + 12L.2g.
n
4dyLgg Ry
By the definition of covering number (c.f. Definition 5 in Liu et al. (022)) we first note that the covering number of (NN(dy, I.p.M) M) bounded by that of FNN L.p. M):
8
N 4dyL.g Ry FNNldy, L,p, M), 2n
<CANN
FNN (1, M), M))
4dyLeg Ry
Thus it suffices find an estimate on the covering number of LN, L, M). A generic bound for classes of functions provided by the following lemma.
21

---

Published in Transactions on Machine Learning Research (10/2023)
Lemma 5 (Theorem 12.2 of Anthony et al. (199 Let F be class of functions from some domain f to [-M, M.. Denole the
of F by Pdim FF). For any 8 >> 0, we have
2cMm
Pdimn(F)
((6, F) -
(38)
FF)
for m > PFi
The next lemma shows that for DNN FNN(1, L,p, M), its pseudo-dimension of can be bounded by the network parameters.
Lemma 6 (Theorem 7 of Bartlett et al. (019)). For any network architecture FNN with L layers and U parameters, there erists an universal constant C such that
Pdim(FNx) s CLU log(U).
(39)
For the network architecture (1, L, M), the number of parameters is bounded by U = Lp². We apply Lemma 5 and B to bound the covering number by its parameters:
Ady Ry FNN(dy, L,,, 2n < Cidyy" log (V²L) (l0g
8
Rgd Lub T.2p2 log( log(T.p2)
log.N
+ log 1 + log n
0-
(40)
when 2n > C2p² L2 log(p²L) L) for some universal constants C₁ and C2. Note that p, L are integers such that pL- = [ -dx/ dx therefore we have
8 log.V 4dyLgy Ry L. 2n S d dx log(dy=T*) (log Lo log(dye-1) + log 8 + logn)
(11)
where the notation S contains constants that depend on Rx, dx and Lg3.
Substituting the above covering number estimate back to equation 37 gives
LI + E. [ dx (w) - u1l2e] + o
Linn -1 "OSETde log(dys; (log La log(dye ) + logo +logn) + 6,
where the notation X contains constant that depends on Læz. LDg, LEg LDz Rx and dx. Letting
E1 dy n 240x 8 n
we have
s-ax "x n - log (n)
21 n
2न22 + 12
[|[T
x dx (n) u113e] + (0² + - + I log Tap I.pd
(42)
stdx 2+dx
LLe log Lg yd
ननंर logn (0² + n L2 || (u) u u113e]
where S contains constants that depend Lez, Loz. L Lxg Lng, Rx and dx- Combining our estimate
equation 42 and equation 25 we have
sidx 2-
Ds I'NN (x) ((u)l33 L log( Lop )d
- logn + (o² + n1
1 En
||112 (u) u113e1 Es Ee [I|IT (0).dy(w))) (w) w113
22

---

Published in Transactions on Machine Learning Research (10/2023)
Proof of Theorem Under Assumption 5 the target finite dimensional map becomes r By ① Dx M Rdy, which is Lipschitz map defined on MM Rdx. Similar to the proof of Theorem 2 the generalization error is decomposed as the following
Eg Dy o NNN o Ex(u) - T T2 + II,
(43)
where T2 and II defined in equation 26 and equation 25 respectively. Following the same procodure in equation 27 obtained that
2n ||
Ti S inf Ex TEZNN
[|[ E (u) - E3 o (2)) 6E
Σ
1=
We then replace Lemma by the following modified version of lemma 17 from Liu et al. (022) to obtain an FNN approximation T
Lemma 7 (Lemma 17 in Liu et al. (202))) Suppose assumplion 5 holds, and assume that || ~0 I B for
all E M. For any Lipschitz function with Lipschitz constant R M, and any integers L, > 0, there ewists f FNN(1, M) such that
fllo S CRL p do
where the constant C solehy depends on do, B, T and the surface area of M. The parameters of Fxx(1, L. M) are chosen as the following
L= -[ L) O(dxplog p) M = R.
The constants in Ω depend do B,T and the surface aren of M.
Apply the above lemuna to each component of Ey $ o Dx and assemble all individual neural networks together, we obtain a neural network 1 E F(dy, L,P, M) such that
[ - - $ Lac,
Here the parameters = (( 1.), p (ddxdy log M I(T+)) with ip = (()). The notation and Q contains constants that solely depend do, Rx, LEx, and surface area of M. The rest of the proof follows the same procedure as in proof of Theorem 2
Proof of Theorem 4 The proof is similar to that of Theorem 2 with a slight change of the neural network construction, so we only provide a brief proof below.
While Assumption holds, the target map X' r y can be decomposed as the following
R
91
R
Vi
X
x
R
dx
R
R
Ryy
DB
y
(44)
Vay
dR
9dy
R
Notice that each route contains a composition of a linear function V. and nonlinear map Di IRd -> RR The nonlinear function Ji can be approximated by a neural network with a size that is independent from dx, while the linear functions Vi can be learned through a linear layer neural network. Consequently, the function h = Vi o Is can be approximated by a neural network hy E FNx (1, 1. M) such that
[h: halloo S CT.s
23

---

Published in Transactions on Machine Learning Research (10/2023)
where L, are integers with Lp [s] do/2 2 and the constant C depends on do- Assembling the neural networks h4 together, can find neural network 17 in FNN(dy, 1, M) with p dyβ, such that
r bo S C
The rest of the proof follows the same as in the proof of Theorem
The proof is very similar to that of Theorem & has the following structure:
Proof of Theorem +
Under Assumption 7
the target map
R
R
R
IR
V1
V?7
V
V/2
D3
N'
R
dx
R
d1
R
R4
Rada
IR
R
Rdy
y.
Vi1
V/2
R d
96
R
R
9
I
(45)
where the abbreviation notation ... denotes blocks G -3.....** G The neural network construction for each block Go is the same as in the proof of Theorem Specifically, there exists neural network Hi FNN(1, 1140, such that
- H'' S CLa,E1, for all i i= 1,...,k
Concatenate all neural networks H, together, obtain the following approximation
| G Hh H' Clae1
The rest of the proof follows the same as in the proof of Theorem 2
A.2 Lipschitz constant of parameter to solution map for Parametric elliptic equation
The solution u to equation 24 is unique for any given boundary condition f so we can define the solution map:
Sa: S H' + H 3/2
To obtain an estimate of the Lipschitz constant of the parameter-to-solutio map o compute the Frechét derivative DS with respect to a and derive an upper bound of the Lipschitz constant. It can be shown that the Frechét derivative
D0 S 18
where satisfies the following equation
vii((((x)V,,s(<))) = div(oVu) in Ω
1/s = 0, on a..
The above claim be proved by using standard linearization argument and adjoint equation methods. Using classical elliptic regularity results, derive that
| ha/2 & Cildiv(oVu))l H- 1/2
<C||l||>> / C1 1 Il
where C' solely depends on the ambient dimension d 2 and a. 8. Therefore, the Lipschitz constant is Cll.
24


Reviews:
[{"review_id": "AxTULdUrad", "reviewer": "Reviewer_SeaA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of the paper lies on the proposal of the analysis, which is quite comprehensive and inspirational as I see. \n\n This paper proposed a few major theorems which explains the claimed \"lessening the curse of dimensionality\". Authors layouts the phenomenon and organizes the analysis very well, with very intuitive illustration and very detailed explanations. Also, the examples to demo the prevalence of \"low-dimensional structure\" and/or low sample complexity in many cases are very well written, and it serves as strong evidence to support the paper's main results. \n\nThere are limitation of this work and authors also stated some of them in the \"limitation\" section. While a few good examples were presented to demonstrate the generalization and common applicability of the theories, there are somes questions/concerns.\n\n$\\mathbf{1}.$ The Lipschitz constant does have a big impact in general for the difficulty of training deep neural networks. Are authors aware of cases where, given a very large constant, some deep neural networks of solving PDEs are very difficult to train? Or, instead, one could bypass such an issue by construction of a smaller Lipschitz constant to learn PDE operators. \n\n$\\mathbf{2}.$ It might not be necessary to conduct some empirical study in this paper. But, since this works targets at analyzing DNN (for PDEs), would it be better off conducting some experiments to support the claim in practice?\n\n$\\mathbf{3}.$ A trivial item:\nNotation could be improved with some minor editing, e.g., the composition function symbols on page 3; also it would be better to add more explanations between theorems (or to explain each terms or some of them one by one to help audience gain better understanding).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an analysis to explain how using deep neural networks to learn a class of PDEs have demonstrated success in the field. in particular, the merits of DNN in reducing the sample/computational complexity for a class of PDEs whose operator functions are Lipschitz continuity and exhibits low-dimensional structure. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the \"weakness\" in the above section.", "broader_impact_concerns": "I do not see any concerns in terms of ethical implication of this work, and authors acknowledges the limitation of this work in the end of the paper. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cCiheRw1cv", "reviewer": "Reviewer_DUAy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is well-structured, the math seems correct, and the results are of interest to the neural networks/PDEs community. The example applications in PDEs are compelling.\n\nWeaknesses: From a technical point of view, the proofs of the main theorems are small modifications of Liu et al. (2022), which provides similar results but when $\\mathcal{X}$ and $\\mathcal{Y}$ are Hilbert spaces (have inner product structure).\n\nThe statements of some of the theorems are difficult to read in the current version (see my comments below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* This paper studies the sample complexity of learning an operator $\\Phi$ from a Banach space $\\mathcal{X}$ to a Banach space $\\mathcal{Y}$ using a neural network.\n\n* The procedure is to learn encoders and decoders $E_{\\mathcal{X}} : \\mathcal{X} \\to \\mathbb{R}^{d_X}$, $D_{\\mathcal{X}} : \\mathbb{R}^{d_X} \\to \\mathcal{X}$, $E_{\\mathcal{Y}} : \\mathcal{Y} \\to \\mathbb{R}^{d_Y}$, $D_{\\mathcal{Y}} : \\mathbb{R}^{d_Y} \\to \\mathcal{Y}$, as well as a neural-network transformation $\\Gamma : \\mathbb{R}^{d_X} \\times \\mathbb{R}^{d_Y}$. The neural network is chosen via empirical risk minimization (ERM) on the square loss.\n\n* It is shown that ERM can avoid the curse of dimensionality if the operator to $\\Phi$ has structure: it is a composition of low-dimensional functions.\n\n* Several example applications to PDEs are given, where the spaces $\\mathcal{X}$ and $\\mathcal{Y}$ are function spaces that lack an inner product and hence previous theorems do not apply.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### A) Comments about statements of theorems: [critical]\n\nI had some difficulty understanding the statements of the theorems at a formal level. However, I am confident that the authors can clarify my understanding by answering my questions below.\n\n1. In Theorems 1 and 3 what does it mean to solve the optimization problem for L = O(...), p = O(...) etc? Does it mean A, B, or C below?\n\n* Possibility A) there are constants C_1,C_2, etc... depending on L_{E_Y^n}, etc... such that if we solve the optimization problem with $L = \\lfloor{C_1 ln(n/d_Y)}\\rfloor$, etc... we get the bound\n\n* Possibility B) there are constants C_1,C_2, etc... depending on L_{E_Y^n}, etc... such that if we solve the optimization problem over all networks with $L <= \\lfloor{C_1 ln(n/d_Y)} \\rfloor$, p <= C_2 (...) etc.. we get the bound\n\n* Possibility C) something else?\n\n2. Another nit-pick is that in Theorem 3 there is reference to integers $\\tilde{L}$, $\\tilde{p}$ such that $\\tilde{L}\\tilde{p} = \\lceil d_Y^{-3{d_0}/(4+2d_0)} n^{d_0 / (4+2d_0)} \\rceil$. Should I interpret the theorem as applying to the optimization problem for any pair of integers satisfying this? Also, technically speaking what if $\\lceil d_Y^{-3{d_0}/(4+2d_0)} n^{d_0 / (4+2d_0)} \\rceil$ is prime, and either $\\tilde{L}$ or $\\tilde{p}$ has to be 1? (I have a similar comment about Theorem 2, equation (5), and Theorems 4 and 5.)\n\n3.  Does Thm 3 also depend on Assumption 5? If not, why does $\\sigma$ appear in the bound?\n\n### B) Comments about the theorems: [critical]\n\nI also have some questions about qualitatively interpreting the theorems. To summarize what confuses me: the theorems are stated as applying for a very specific choice of hyperparameters (width, depth, sparsity, weight bounds) without much explanation for that choice. I don't know how much of that choice of hyperparameters is arbitrary, and how much is flexible.\n\nFor example:\n\n1. In Theorem 1, is there something special about d_X = 2? It seems that for d_X < 2, the width p must tend to zero as d_Y tends to infinity? But for d_X > 2, the width p can tend to infinity as d_Y tends to infinity? Why should there be such a behavior at d_X = 2? In comparison, in Theorem 2 there seems to be some special behavior at d_X = 4?\n\nIs there anything going on here, or is the \"interesting\" regime of the theorems only for large d_X?\n\n2. It would help the reader if the authors pointed out exactly what is different about Theorem 1 and Theorem 2. I don't understand how the sparsity and parameter L_{infinity} norm bounds are mattering. For example, why is M allowed to grow as sqrt(d_Y) in Theorem 2 but not in Theorem 1?\n\nSimilar questions about the hyperparameter choices in Theorems 3,4,5.\n\n3. The bounds in the theorems are challenging to read. One suggestion to streamline the theorems and make them significantly easier to read and compare to each other would be to define something along the lines of\n$\\Pi_{err} = \\sigma^2 + n^{-1} + L_{\\Phi}^2 E_u[||\\Pi_{X,d_X}^n(u) - u||^2_X] + E_S[E_{w\\sim \\Phi_{\\\\#} \\gamma}[||\\Pi_{Y,d_Y}^n(w) - w||^2_Y]$\n, and then add $\\Pi_{err}$ to each of theorems 1,2,3,4,5, since it is a shared term between each of these theorems. Similarly you could define something like\n$RISK(\\Gamma_{NN}) = E_S[E_u[||D_Y^n \\circ \\Gamma_{NN} \\circ E_X^n(u) - \\Phi(u)||^2_Y]]$\nand reuse $RISK(\\Gamma_{NN})$ in each of the theorems.\n\n### C) Requested clarifications on discretization section [not critical]\n\n1. What does it mean for all grid points x^i of all given discretized data to be denser than \\hat{x}? Do you mean that the \\hat{x} discretization is a subset of the x^i discretization?\n\n### D) Comments on \"Explicit complexity bounds for various PDE operator learning\" [not critical]\n\n1. In Section 3.1, it may help to write more explicitly with an equation how Assumption 7 is met. It took me some effort to understand this.\n\n2. What does the \"manifold dimension of the data set of media function a(x)\" mean in Section 3.5? Is it always <=2 because Omega \\subset R^2?\n\n### Typos/very minor: [not critical]\n* “discretization-invariant in operator learning”\n* “are Banach space” -> \"spaces\"\n* Overloaded \\phi notation\n* “and provides explicit” -> \"and provide explicit\"\n* \"hold ture\"\n* In Theorems 4,5 there is no O(\\cdot) notation, but it's referred to in the statement.\n* In discussion after Theorem 3, please clarify what \"algebraic dependency\" means. Does it mean that the bounds depend at most polynomially?\n* Bottom of page 7: \"where the sampling locations $x^i = [x_1^1,...,x_{s_i}^i]$\n* Top of page 8: mismatch between r^i and r_i\n* Missing unclosed parenthesis in (15)\n* \"is give by\" -> \"is given by\"\n* Equation (20), should have $\\lesssim$ instead of $\\leq$\n* \"Assumption 7 hold\" -> \"Assumption 7 holds\"\n* \"see Appendix A.2\" -> \"see Appendix A.2.\"\n* On page 3, the set $\\\\mathcal{S}_1$ is referred to a training set, and $\\mathcal{S}_2$ as a test set. However, judging from the proof in the appendix it seems that $\\\\mathcal{S}1$ is used to create the encoders/decoders, and $\\\\mathcal{S}2$ is used to learn $\\\\Gamma$. Am I misinterpreting this?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "emJTrnqkEN", "reviewer": "Reviewer_3b39", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. This is an interesting topic to investigate as we have seen several works in ML (like consistency mode in diffusion) that need to implement  ODE/PDE solvers using neural networks (NNs). The work on this topic could provide some theoretical guidance in network design. \n2. The analysis that explains how NNs alleviate the CoD in the PDE solving problems is interesting. Although this is not a very surprising result, I think it should be useful to present everything rigorously. \n3. The paper is well-written and easy to follow for most of the part. \n\n## Weaknesses\n1. The authors claim that ``the removal of the inner-product assumption is crucial .. enabling us to apply the estimates to various PDE problems'',  and they mentioned in Sec 1.1 that they will give some demonstration in Section 3. However, it seems like the authors do not explicitly discuss this issue in Section 3. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on providing theoretical evidence to explain why many PDE operators can be effectively approximated by deep neural networks (DNNs). In comparison to the existing work, this paper generalizes the setting from Hilbert space to Banach spaces that do not have an inner-product structure. Additionally, by assuming inherent low dimensionality and complexity in the target PDE operators (which may be embedded in a high dimensional space), they show that the sample complexity only depends on the low inherent dimensionality, which justifies why DNNs can mitigate the curse of dimensionality problem (CoD). Additional results and discussions are given on the discretization invariance neural networks. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical changes\n1. As mentioned in weaknesses, there should be some discussion on why getting rid of the inner product structure is important. Besides, it will be helpful to also discuss how this relaxation can benefit the ML research (e.g., there are some problems that we cannot define inner products). \n\n## Minors changes\n1. p2, above Sec 1.1, 'To the best of our knowledge, the [typo] ... '\n2. Above Assumption 1, 'We consider ... on the target PDE map [should be $\\Phi$?], the encoder'\n3. In proposition 1, $C$ and $h$ are not defined. ", "broader_impact_concerns": "There aren't any major ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work continued the theoretical analysis of deep operator learning, where an encoder-decoder structure is first applied and then deep neural networks are employed to learn the resulting (lower dimensional) map. The results bear some similarity to those of Liu et al. (2022) and the main contribution lies on the analysis of several nonlinear PDE operators. All reviewers found this work well-written and recommended acceptance. The presented analysis and examples are interesting and may inspire further interaction between deep learning and PDEs. 

One revision we would like to suggest is to compare the authors' results with those of Liu et al. (2022) in much greater detail, under the Hilbert setting. It seems the lack of an inner product (in this work) was circumvented by a simple application of the triangle inequality, which probably will make the authors' result a bit looser (c.f., Remark 3). If this is the case, it is important to explicitly acknowledge this trade-off (looser but more general) so that the tighter results of Liu et al. (2022) will not be shadowed by the authors' seemingly more general result (for any Banach space). It will also explain why the current work could afford to remove Assumption 5, even under a more general setting. Lastly, given how the proof proceeds, it is probably wise to not over-claim the extension to a Banach space. [Of course, this is not to say the case studies in Section 3 are not interesting or important.]

Notation: first paragraph of Section 2, do the authors mean x <= Cy for the notation x = O(y)? [This would be the usual meaning of big-O. The current wording seems to suggest the converse, which should be denoted as big-Omega.]
Decision: Accept (poster)