Generic bounds on the approximation error for physics-informed (and) operator learning
Authors: Tim De Ryck
Siddhartha Mishra
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep learning
PINN
DeepONet
FNO
neural network approximation theory

Abstract: We propose a very general framework for deriving rigorous bounds on the approximation error for physics-informed neural networks (PINNs) and operator learning architectures such as DeepONets and FNOs as well as for physics-informed operator learning. These bounds guarantee that PINNs and (physics-informed) DeepONets or FNOs will efficiently approximate the underlying solution or solution-operator of generic partial differential equations (PDEs). Our framework utilizes existing neural network approximation results to obtain bounds on more-involved learning architectures for PDEs. We illustrate the general framework by deriving the first rigorous bounds on the approximation error of physics-informed operator learning and by showing that PINNs (and physics-informed DeepONets and FNOs) mitigate the curse of dimensionality in approximating nonlinear parabolic PDEs. 

Generic bounds on the approximation error for physics-informed (and) operator learning
Tim De Ryck
Siddhartha Mishra
Abstract
We propose very general framework for deriving rigorous bounds on the approxi- mation error for physics-informn neural networks (PINNs) and operator learning architectures such as DeepONets and FNOs as well as for physics-informed opera- tor learning. These bounds guarantee that PINNs and (physics-informedd) Deep- ONets or FNOs will efficiently approximate the underlying solution or solution operator of generic partial differential equations (PDEs). Our framework utilizes existing neural network approximation results to obtain bounds on more involved learning architectures for PDEs. We illustrate the general framework by deriving the first rigorous bounds on the approximation error of physics-infommn learning and by showing that PINNs (and informed DeepONets and FNOs) mitigate the curse of dimensionality in approximating nonlinear parabolic PDEs.
operator
Introduction
The efficient numerical approximation of partial differential equations (PDEs) is of paramount importance as PDEs mathematically describe an enormous range of interesting phenomena in the sciences and engineering. Machine learning techniques, particularly deep learning, playing an increasingly important role in this context. For instance, given their universal approximation properties, deep neural networks serve as ansatz spaces for supervised learning of variety of S and references therein. In this setting. large amounts of training data might be required. However, this data is often acquired from expensive computer simulations or physical measurements (33, necessitating the design of learning frameworks that work with limited data. Physics-informet neural networks (PINNs), proposed by [18 42 41 and popularized
(parametric) PDEs 19 77
40
3
by (67 68), are a prominent example of such a learning framework as the residual of the underlying PDE is minimized within the class of neural networks and in principle, little (or even no) training data is required. PINNs and their variants have proven to be a very powerful and computationally efficient framework for approximating solutions to PDEs, 69 and references therein.
Often in the context of PDEs, needs to approximate the underlying solution operator that maps one infiniteddimensional function space into another [27] 39). As neural networks can only map between finite dimensional spaces, learning frameworks need to be designed in order to approximate operators. These include deep operator networks (DeepONets) [ 40 and their variants as well as neural operators 39 which generalize neural networks to this setting. A variety of neural operators have been proposed, see but arguably, the most efficient form of neural operators is provided the so-called Fourier neural operators (FNOs) 44 Both DeepONets and FNOs have been very successfully deployed in
field of operator learning is emerging wherein novel
45 46
"Seminar for Applied Mathematics (SAM), D-MATH, ETH Zürich, Switzerland
Seminar for Applied Mathematies (SAM), D- D-MATH and ETH Al center, ETH Zürich, Switzerland
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

scientific computing 550 56 8
48
47 66 and references therein. Finally, one can combine PINNs
and operator learning to design physies-informet informed DeepONets/FNOs
From a theoretical perspective, one needs to provide a rigorous guarantee that the learning framework can approximate the underlying PDE solution (operator) to desired accuracy. More precisely, given an error tolerance E >> O, we need to rigorously prove that the approximation error of the neural network (operator) can be made smaller than s. For efficient approximation, one has to further ensure that the computational complexity (measured in terms of the size) of the learning architecture grows at most polynomially in E 1 In particular, exponential growth has be ruled out. As neural networks. DeepONets and FNOs are all universal approximators [T3] 4o 43
3N of the underlying functions or operators, it possible to show that the approximation error can be made as small as desired. However, these results do not guarantee efficient approximation as the underlying network size could still grow exponentially with decreasing error, see [77] for neural networks in very high spatial dimensions, 4 for DeepONets and 38 for FNOs. Hence, the real theoretical challenge in this context lies in proving efficient approximation results for the different learning architectures
in scientific computing. Such efficient approximation results have mostly been obtained for neural networks in the supervised learning setting e.g. [23
63 and references therein. In contrast, there is relative scarcity of such efficient approximation results for PINNs and operator learning with notable exceptions being [25] [5] [6 (for PINNs), A (for DeepONets) and B8 (for FNOs). Moreover, the underlying proofs in these works often a case-by-care basis and the overall abstract structure is not clearly identified. Finally, no similar rigorous approximation results for physics informed operator learning available till date.
36
3
This paucity of generic efficient approximation results for PINNS and operator learning for PDEs sets the stage for the current paper where our main contribution is to propose a very general framework (Section 3 for proving bounds on the approximation error for space-time neural networks, PINNs, DeepONets, FNOs and physics-informed DeepONets and FNOs for very general PDEs. Consequently, we obtain the first rigorous bounds for physics-infommn is based on the observation that error estimates for different types of neural network architectures can all be obtained from one another. As error estimates for neural network approximations of PDE solutions fixed time are the casiest to obtain, and hence constitute the largest proportion of currently available estimates, we devote particular attention to demonstrating how these available estimates can be used to obtain novel bounds on the approximation error for space-time networks, PINNs and (physics-ifformed) operator learning. Our results provide a roadmap for deriving mathematical guarantees for deep learning methods in scientific computing by simplifying the proofs, as the needed work essentially reduces to verifying a small number of assumptions. We demonstrate how the generic error bounds from Section 3 can be applied in practice in Section 3 among others by giving short alternative proofs for known results and also proving a number of novel results. In particular, we show in Section 4.1 that PINNs can overcome the curse of dimensionality for nonlinear parabolic PDEs such as the Allen-Cahn equation i.c., that the network size does not grow exponentially with increasing spatial dimension. Moreover, dimension-inee indeeenddenn convergence rates are also obtained for (physics-im -nnfo-ooddd)) DeepONets and FNOs, provided that the PDE solutions are sufficiently smooth. These are the first results of their kind. We note that many of the proofs and some examples are deferred to the supplementary material (SM).
operator learning in literature. Our framework
2 Preliminaries
2.1
Setting
Given T > 0 and D c Rd compact, consider the function u : [0, D - R. for m 2 1, that belongs to a function space H and solves the following (time-dependent) PDE,
La (t, 0 and u(z, 0) uo
V(t,x) E [0,T] x D,
(2.1)
where uy E y c (DD the initial condition and La: H. H- I2 x D) is a differential operator that can depend on a parameter (function) a E Z C L2()). In our notation, we will often suppress the dependence of L := L. on a for simplicity. Depending on the context, one might want to recover one of the following mathematical objects: for fixed a and up. one might want to approximate -(T,-) or u(:..) with a neural network; more challenging task would be to learn the solution operator G= : X + (2)) + u, where u E {uo, X 2),2) and Ω D or
2

---

Ω = [0,T] x D. We will use this notation consistently throughout the paper, see SM A.1 for an overview.
2.2 Approximating PDEs with neural networks
Neural networks A (feedforward) neural network ug R - Rd is defined as a concatenation
of affine maps A R Rd 2 + Wiz + bi and an activation function σ R - IR that is applied component-wist resulting in,
= AL ... 00. (1)).
(2.2)
The weights and biases the affine maps 0 = {W,, I are the trainable parameters. We will quantify the size of neural network by its depth(ue) := L and its width(ua) := max; di. In order to obtain a neural network that approximates the solution u of PDE 2.T at time 1 = T. one chooses the parameters of us is minimized. The training data is acquired from either measurements or potentially expensive simulations.
E such that discretization (quadrature) of J [() ||u(T) uell ug]]
PINNs Physics-informed neural networks (PINNs) are neural networks that are trained with a different, residual-based loss function. As the PDE solution u satisfies L(u) = 0, the goal of physics-informed learning is to find a neural network up : [0, x + I for which the PDE residual is approximately zero, L(wo) Rs O. To ensure uniqueness, one also needs to require that the initial condition satisfied i.e., (0, Rs un(r)), and similarly for boundary conditions. In practice ||ue (0. -) uo li (D) where additional terms can be added to (approximately) impose boundary conditions and augment the loss function using data. A desirable property of PIŇNs is that only very little or even no training data is needed to construct the loss function.
one minimizes quadrature approximation of (0) [[C(ue)l ([(=))](=(000rrxx
Operator learning In order to approximate operators, one needs to allow the input and output of the learning architecture to be infinite-dimensioall dimensional. A possible approach is to use deep operator networks (DeepONets), as proposed in M 49]. Given m, fixed sensor locations { C D and the corresponding sensor values {1(x)))} as input, a DeepONet can be formulated in terms of two (deep) neural networks: branch net 3 RT + I2 and a trunk net T D + R++1 The branch and trunk nets are then combined to approximate the underlying nonlinear operator as the following DeepONet Go : X + (D)) with (v() = To(y) + Σk (x) A second approach that of neural operators, which generalize hidden layers by including a non-local integral operator [45], of which particularly Fourier neural operators (FNOs) 44] are already well- established. The practical implementation (i.e. discretization) of an FNO maps from and to the space of trigonometric polynomials of degree at most N E N, denoted by L and can be identified with a finite-dimensional mapping that is a composition of affine maps and nonlinear layers of the form (x)) a (Wiu,- + FN (Pt(k)) Fx (k)),, where the (k) are coefficients that define non-local convolution operator via the discrete Fourier transform FN, see AS
Physics-informed operator learning Both DeepONets and FNOs trained by choosing suitable probability measure μ on X and minimizing a quadrature approximation of J = [g ((v))l22xa Generating training sets might require many calls to an expensive PDE solver, leading to an enormous computational cost. In order to reduce or even fully eliminate the need for training data, physics-informed operator learning has been proposed in [74] for DeepONets and 47 for FNOs. Similar to PINNs, the training procedure aims to minimize a quadrature approximation of J(0) [[C(G))] (X'xX))
3 General results
We propose a framework to obtain bounds on the approximation error for the various neural network architectures introduced in Section 2 Figure [ visuullizes how different types of error estimates can be obtained from one another. Every box shows the name of the network architecture, the form the relevant loss and the theorem which proves the corresponding estimate for the approximation error. Every arrow in the flowchart represents proof technique that allows one to transfer an error estimate from one type of method to another (see caption of Figure I for overview of those techniques).
3

---

Neural network (fixed time) [l((T) ug (T( (a)D) < Assumed be known
FNO
DeepONet
l
Goll <e
D.
Goll G <E Corollary F88
Theorem E7
Neural network (space-time)
||u Hell ((o < Theorem
B
B
PINN
Physics-ifformed FNO
Physics-informed DeepONet
[[uo))
2. < <
[( [(go))l <
[( L2 <
Theorem
Theorem .9
Theorem
Figure I: Flowchart of the structure of the results in this paper, with q {2, j. The letters reflect the techniques used in the proofs: A uses Taylor approximations (Section (.II, is based on finite difference approximations (Section 3.1 C uses trigonometric polynomial interpolation (Section 3.2
and D uses the connection between FNOs and DeepONets (Section B.2)
We give particular attention to the case where known that neural network can efficiently approximate the solution to a time-dependent PDE at a fixed time. Such neural networks are usually obtained by emulating classical numerical method. Examples include finite difference schemes, finite volume schemes, finite element methods, iterative methods and Monte Carlo methods, e.g.
IC 57 More precisely, for O, we assume have access to an operator Uf X x [0, 7 H that for any E (0, 7' maps any initial condition/parameter function E X to neural network U (v. t) that approximates the PDE solution (vv) 1) = u((,t)e (D), E {2, oo); at time 4, as specified below. Moreover, we will assume that we know how its size depends on the accuracy Explicit examples of the operator UE will be given in Section $ and SM C
36 63
Assumption 3.1. Let q E xc). For any B, 0, & || S there exist a neural network (=(1,, t) D R. and constant C s.t.
E N,
[0,T] T and any v E X' with
(ol)),)))zzzo SE and max rE(O.T]
[1, t) wll S C
(3.1)
Remark 3.2. For vanilla neural networks and PINNs one can set X' == (u), g(v) == u and v == uo or 0 a in Assumption 3.7 above and Assumption 3.4 below.
Under this assumption, we prove the existence of space-time neural networks and PINNs that efficiently approximate the PDE solution (Section B.T), as well as FNOs and DeepONets (Section
B.T),
3.2p and physics-informed FNOs and DeepONets (Section 3.3) Finally, we also prove a general result on the generalization error (Section) 3.4).
3.1 Estimates for (physics-informed) neural networks
We will construct a space-time neural network uo for which both ||ue - == D) and the PINN loss ([(ua))]l==0007)000 are small. To accurately approximate the time derivatives of u emulate Taylor expansions, whereas for the spatial derivatives we employ finite difference (FD) operators in proofs. Depending on whether forward, backward or central differences
used, FD operator might not be defined on the whole domain D, e.g. for f E C(O0, 1]) the (forward) operator [ == f( + h) - fxx) is not well-defined for x E (1 - h, 1]. This can be solved by resorting to piccewise-defined FD operators, c.g. forward operator on [0, 0.5] and backward operator on (0.5, 1]. a general domain Q one can find a well-defined piecewise FD operator 12 satisfies the following assumption, which is satisfied by many domains (e.g. rectangular, smooth).
Assumption 3.3. There exists a finite partition P of such that for all E P there exists e > and Up E B {a E Rdim(D) || < 1} such that for all x. E P it holds that x t + B C Ω.
Additionally, we need to assume that the PINN error can be bounded in terms of the errors related to all relevant partial derivatives, denoted by (kkka)) D. : 0f g.. ... 3.. for (k, a) e No+1

---

This assumption is valid for many classical solutions of PDEs. A few worked out examples can be found in D.3 (gravity pendulum) and ( flow).
Assumption 3.4. Let k, l E N. q E (2, C 0 be independent from d. For all v E X it holds,
[[GG()))) s C. poly(d)
Σ (K) &'shllaallise
al - -9o) [<(().T))))
(3.2)
In this setting, we prove the following approximation result for space-time networks and PINNs.
Theorem 3.5. Let s.r E N. let u E ((o,) x D) be the solution of the PDE 2. and ler Assumption 3. be satisfied. There exists a constant C(s,r) e, h > O there exists a tanh neural network ue : [0, [0,T) x D - IR for which it holds that,
0 such that for every M E N and
Iug w ( I C([[u]lcu - + +e).
(3.3)
and if additionally Assumption and Assumption 3.4 hold then,
||<(u))]l><()>)))) + U - w wll =011=000(0077)
(3.4)
CC. (C-ooyy() In (M + 2 + C).
Moreover, dopth(us) S C. depth(())) and width(ug) I CM - width(U1F)
Proof. We only provide sketch of the full proof (SM B.2 The main idea is to divide [0, 7] into M uniform subintervals and construct neural network that approximates Taylor approximation in time of u in each subinterval. In the obtained formula, we approximate the monomials and multiplications by neural networks (A.)) and approximate the derivatives u by finite differences and use (A.2) of SM A.2 to find an error estimate in C* ([0, T`, (D))))nnoom We use again finite difference operators to prove that spatial derivatives of are accurately approximated as well. The neural network will also approximately satisfy the initial/boundary conditions l|ue ull (.((a((.T)x S
Cpoly (d) I u| ([0.T) D). which follows from Sobolev trace inequality.
We note that the bounds 3.3 and 3.4) together imply that there exists a neural network for which the total error as well as the PINN loss can be made as small as possible, providing solid theoretical foundation to PINNs for approximating the PDE 22.
3.2 Estimates for operator learning
In this section, we use Assumption 3.1 to prove estimates for DeepONets and FNOs. First, we prove a generic error estimate for FNOs. Using the known connection between FNOs and DeepONets (SM Lemma B.6) this result can then easily be applied to DeepONets (Corollary 3.8 In order to prove these error estimates, we need to assume that the operator UE from Assumption with respect to its input function, as specified in Assumption 3.6 below. Moreover, we will take the d-dimensional torus as domain = Td [0. ) and assume periodic boundary conditions for simplicity in what follows, This is not restriction, as for every Lipschitz subset of IT there exists a (linear and continuous) T -e-ppoioo extension operator of which also the derivatives are T -prioodic
is stable
3ل Lemma 41].
Assumption 3.6. Assumption 3.1 is satisfied and let p E (2, For every e > 0 there exists such that for all v, E X holds that,
constant Ca >
[u u (v', (V',T))lzz S Caablle - 1/
(3.5)
In this setting, we prove a generic approximation result for FNOs.
Theorem 3.7. Let r E N, T > 0. let G C (T*) + C' (TT) be an operator that maps a function un to the solution u(-, I' of the PDE 2.I with initial condition up let Assumption 3.6 be satisfied and let p E (2, 3 {p}. Then there exists a constant C > such that for every 0, N E N
there is an FNO Go L3 (TT) + L2x(T of depth ((deth((U=)) and width O width((()) with accuracy,
|| - Gollz C(s + Caab BN BN--+d// + +CCBN' NT).
(3.6)
5

---

Proof. We give sketch of the proof, details can be found in SM Given function values of on uniform grid with grid size 1/N, we use trigonometric polynomial interpolation A.M to reconstruct and use this together with Assumption 3.1 to construct a neural network. The resulting approximation is then projected onto the space 2. of trigonometric polynomials of degree at most N E N, again through trigonometric polynomial interpolation.
L
recent result, Theorem 36] (SM Lemma B.6p. shows that any error bound for FNOs also implies an error bound for DeepONets, by choosing the trunk nets as neural network approximations of the Fourier basis. We apply this result with & ~ poly(1/N) to Theorem 3.7 to obtain the following generic error bound for DeepONets.
Corollary 3.8. Assume the setting of Theorem 3.7 Then for every > N N and ev- L²(D) with
ery corresponding FNO Go from Theorem 3.7 there exists a DeepONet 9 : X + width = O(N)), depth(3) = ((depth(Go), width(r) = ((N+11) and depth(T) S that
satisfies 3.6
3.3 Estimates for physics-informed operator learning
Using the techniques from previous sections, we now present the very first theoretical result for physics-informed operator learning. We demonstrate that if an eπror estimate for DeepONet/FNO and the growth of its derivatives are known (see SM D.I on how to obtain these). then one can prove an error estimate for the corresponding physics-informed DeepONet/FNO. For simplicity, the following result focuses only on operators mapping (D) but the generalization to e.g. C" (10, T) x D) is immediate by considering D' == [0, x D.
N, that satisfies Assumption 3.3 and Assumption 3.4 with E N. Let X E (0, os). let X, C()) > with S X and let a N be function such that for all p E N there is a DeepONen/FNO Go such that
Theorem 3.9. Consider an operator G X C (D))
[1 Go(u))lzP(D) S Cy-
and
Go((v) (r(D) S Cyp(r)
Vr EX
(3.7)
Then for all 3 E with 0 < 3 S
there exists a constant C > such that for all
E X and p E N it holds that
[<(G(()))]<>>)>>>>>>>)) S SC'P
(3.8)
Proof. For suitable D use SM Lemma B.1 with q 2 = (vv) and fz = Go(v) together with 3.7
find
(g(()) Go(u)) S ((r h (r()) (-
(3.9)
Let B E I with 0 < (((rr) We carefully balance terms by setting h = and
X -o((r) + 353 to find 3.8 Conclude using Assumption 3.4
L
Finally, we use Theorem 3.5 to present an alternative error estimate for a physics-informed DeepONet in the case that Assumption 3.1 is satisfied. As this assumption different from assuming access to an error bound for the corresponding DecpONet, it is interesting to use the techniques from the The proof of the following thcorem can be
previous sections rather than directly apply Theorem 3.9 found in BM4
C(s.r) (JO, T x T be operator that maps a function un to the solution u of the PDE 2.1 with initial condition un, let Assumption 3.1 and Assumption 3.6 satisfied and let p" {2, oc)
Theorem 3.10. s,r E N, >>>>>> O. let G O" T
{p}- There exists constant C > O such that for every Z, N,M E N, e, > 0 there is an DeepONet Go CT(T)) (2 I1 x T with z sensors with accuracy,
[[((v) - -v (P((O.T)XTT) S CM M M + O Ca Z-rd/pp CCB N ))
(3.10)
and if additionally Assumption and Assumption hold then, [[C(G(()))l S CC
*MM* N' C C --t///" +CCB N N=))),
for all v. Moreover, it holds that, depth = depth(UF") widthh3) - (3.11) depth(T) = 3 and width(T) O(MN In))))).

---

3,4 A posteriori bound on the generalization error
Although the main focus of this paper is on the approximation error for different neural network architectures, we now demonstrate that it is possible to provide similar bounds for other sources of error, such as the generalization error. We therefore prove a general a posteriori upper bound on the generalization error of the all the considered neural network architectures. Consider f (an operator or function) and the neural network architecture fo DD + R, O, which includes all architectures of Section 2.2 neural networks (D) 22 f = and fo ug); PINNs (D = Ω f= and fo (. operator learning (D f2 X, 9 and fe Gg) and physics-informed operator learning (D Q X, 0 and fe ((Ge) )). Given training set S = { Xn. where } are random variables on (according to a measure μ), the training error & and generalization error &g are,
IR
ET(0,5)
- 7
Eo(0)² ( d(())
(3.12)
where j is probability measure on D. The following theorem provides a computable posterioni error bound on the expectation of the generalization error for a general class of approximators. We refer e.g. 66 16 for bounds on de, and L.
Theorem 3.11. For R >> O and de N, let θ i-R, Ride be the set of trainable parameters, and for every training set S. let (S) E θ be an (approximate) minimizer of - (f, S)² assume that 0 + ( 5)2 and 0 (0) are bounded by > 0 and Lipschitz, continuous with Lipschitz constans & 0. I 20²e. (2222((2R5) /(222)0/2 then it holds that
2c (de 1) In (RRSV)).
E ((((
< E (T)0) +
(3.13)
n
Proof. The proof (SM B combines standard techniques, based on covering numbers and Hoeffd- ing's inequality, with an error composition from [6]
For any type of neural network architecture of depth L, width W and weights bounded by R. one finds that de LW (W + d). For tanh neural networks and operator learning architectures, one has that In(2) ~ T. (nddRW,,, whereas for physics-infommn
neural networks and DeepONets one
finds that In(2) ~ (I: + (ddRW) with le and as Assumption 3.4 [6] Taking this into account, one also finds that the imposed lower bound on is not very restrictive. Moreover, the RHS of (3.13) depends at most polynomially on L, W, RG d, k É and c. For physics-informed architectures, often depend exponentially
however, upper bounds
L [1 14
Remark 3.12. As Theorem 3.77 is an posteriori error estimate, one can use the network sizes of the trained networks for L, W and R. The sizes stemming from the approximation error estimates of the previous sections can be disregarded for this result. Moreover, instead of considering the expected values of &g and Ep (.13), one can also prove that such an inequality holds with
certain probability (see SM B.5
4 Applications
demonstrate the power and generality of the framework proposed in Section B applying the presented theory to the following case studies. First, we demonstrate how these generic bounds can be used to overcome the curse dimensionality (CoD) for linear Kolmogorov PDEs and nonlinear parabolic PDEs (Section 4.1 These are the first available results that overcome the CoD for nonlinear parabolic PDEs for PINN and dahysics-informen operator learning. Next, we apply the results of Section 3.3 to both linear and nonlinear operators and provide bounds the approximation error for thysics-informat operator learning.
4.1 Overcoming the curse of dimensionality
For high-dimensional PDEs, it is not possible to obtain efficient approximation results using standard neural network approximation theory 77 15 as they will lead to convergence rates that suffer from the CoD, meaning that the neural network size scales exponentially the input dimension.
7

---

In literature, one has shown for some PDEs that their solution at fixed time can be approximated to accuracy E > 0 with a network that has size O(poly(d) ")). with 5 > independent of and therefore overcomes the CoD.
Linear Kolmogorov PDES
We consider linear time-dependent PDEs of the following form.
Setting 4.1. Let 8, E N, uh E CB and let ((s,)) ([0,T) Rd) be the solution of
(())(x, = 0,,(a, 1 T
mxx)) V (x) () t) = -0, u(0. x) =
uo(x)
(4.1)
for all (x, E D [0, T) where Rd + Raxd and μ Rd - Red affine functions and for which [lullct.,2 grows at most polynomially in d. For every e > 0, there is neural network uo of width (ppoly(d)s-P) such that || - (R)) <..
Prototypical examples of such linear Kolmogorov PDEs include the heat equation and the Black- Scholes equation. In [33 / [6 the authors construct neural network that approximates uTT) and overcomes the CoD by emulating Monte-Carlo methods based on the Feynman- Kac formula. In []6] one has proven that PINNS overcome the CoD as well, in the sense that the network size grows as (ppoly(dpa) (dpp))dd)))))) - ), with Pa as defined in SM CC.0).. For a subclass of Kolmogorov PDEs is known that Pa poly(d), such that the CoD is fully overcome.
We demonstrate that the generic bounds of Section 3 Thhoreem 3.5) can be used to provide a much shorter proof for this result. SM Lemma C.6 verifies that Assumption 3.1 is indeed satisfied. The full proof can be found in SM
Theorem 4.2. Assume that Setting holds. For every 0,8 > 0 and d E N, there is tanh neural network uo of depth (depth((a)) and width Oppoly (2+33 sto #tl 14) such that,
+ ||ue - ull (=(00 (0,1) S
(4.2)
Nonlinear parabolic PDEs Next, we consider nonlinear parabolic PDEs as in Section 4.3 which
typically arise in the context of nonlinear diffusion-reaction equations that describe the change in space and time of some quantities, such as in the well-known Allen-Cahr equation 1
Setting 4.3. Let s, r E N and for 20 E X c C" (TT) let u E ((s..)) ([0, I] T be the solution of (x))(x,t) d,u(t,
-(((tx)))(( 00 u(0, X) uo(z),
(4.3)
for all (t) E [0,T] x D, with period boundary conditions, where F R + R is a polynomial and for which ( grows at most polynomially in d. For every > O, there is neural network ūo of widhh (poly(d) (d)= such that [|uo uo (m))) < e. Let ps, resp. j², be the normalized Lebesgue
measure on [0, T] x Td, resp. a(oo, T] x T
In 32 the authors have proven that ReI.U neural networks overcome the CoD in the approximation of u We have reproven this result in SM Lemma C.14 for tanh neural networks show that Assumption 3. is satisfied. Using Theorem 3.5 we can now prove that PINNs overcome the CoD for nonlinear parabolic PDEs. The proof is analogous to that of Theorem 4.2
Theorem 4.4. Assume Setting 4.3 For every o, 0 and d E N there is tanh neural network of depth (depth((o)) poly (d) (n)/e()))) and width Opoly (a)oolyddde(-+-0)0) (2+ mig T-1
such that,
|| ((mo) ((((TTT)TXX + ||u ----uoll uoll (1000 S..
(4.4)
Similarly, one can use the results from Section 3.2 to obtain estimates for (physics-informed)
DeepONets for nonlinear parabolic PDEs 4.3) such as the Allen-Cahn equation. In particular, innension-independeee
convergence rate can be obtained the solution is smooth enough, which
improves upon the result of 4X] which incurred the CoD. For simplicity, we present results for (2,r)
functions, rather than ((s,r) functions, we found that assuming more regularity did not necessarily further improve the convergence rate. The proof is given in SM B.4
Theorem 4.5. Assume Setting 4.3 and let G C(2.)) T] x T ug + u. For every O,E > there exists DeepONets Go and Gž such that || - -el = Se,
+ (T(TT)= uo + u(I) and G X +
| | Se.
(4.5)
8

---

Moreover, for Go we have O(s 4t2
sensors and,
(3) O
(d+m)(20))
depth((3) = O1I (l(()))))) depth(r) = 3,
(4.6)
width(r) = (E
1+1
whereas for Gg have O( x (3+-)
sensors and,
width(33 = ( (31 iata
depth(33 = (In((1E)),
(4.7)
width(T) = Oss [3-))d+1)
depth(T) = 3.
4.2 Error bounds for physics-informed operator learning
We demonstrate how Theorem 3.9 can be used generalize available enror estimates for DeepONets and SMD.I to estimates for their physics-informe counterparts.
and FNOs, c.g.
Linear operators In the simplest case, the operator G of interest is linear. In 43 Theorem D.2], general error bound for ReLU DeepONets for linear operators has been established, which still holds for tanh DeepONets. Using Theorem 3 it is then straightforward to prove convergence rates for physics-informi informee DeepONets for solution operators
linear PDEs 22.1.
Consider an operator 9 X + (2(T)): : + u as in Section 2. where v is the parameter/initial condition and u the solution of the PDE 2.J. Following K we fix the measure μ on (²( as Gaussian random field, such that allows the Karhunen-Loève expansion
> Xxek. 0, the X AO 1) are iid Gaussian random variables and {ee Jezz is the standard Fourier basis (SM A.5 In this setting, we can prove the following approximation result, the proof of which can be found in SM D.3 The result can be generalized to other data distributions μ for which convergence result for DeepONets can be proven, as
where |oo| S exp(-e[kl) with
Theorem 4.6. Assume the setting above and that of Assumption 3.4 and assume that G(v)
CR+1 (TT) for all v E X. For all B > O there exists a constant C > 0 such for any p E N there exists a DeepONet Go with p sensors and branch and trunk nets such that
||[(G))) (( S Cp Cp-
(4.8)
Moreover, size(T) I Cp
d+1 J' depth(T) size(B) S and depth(3) =
Nonlinear operators For nonlinear PDEs general result like Theorem 4.6 can not be obtained from the currently available tools. Instead one needs to use Theorem 3.9 for every PDE of interest on case-by-case basis. In the SM, demonstrate this for nonlinear ODE (gravity pendulum with external force, SM D.5) and an elliptic PDE (Darcy flow, SM D.6)..
Related work and discussion
This is the first paper to rigorously expose the connections between the different deep learning frameworks from Section 2.2 for generic PDEs. Until now, most available results focus on providing generic results for one specific method. In [3I and !Z! mate solutions to a generic ODE/PDE at fixed time to construct space-time neural networks. A generalization to PINNs is not immediate the proof involves the emulation of the forward Fuler method. We have overcome this difficulty by constructing space-time neural networks using Taylor expansions instead (Theorem 3.5) To bound the approximation error of PINNs one can use the generic error bounds in Sobolev norms of e.g. 25 26 for very general activation functions or the more concrete bounds [S]] for tanh neural networks. In both approaches, the only assumption is that the solution of the PDE has sufficient Sobolev regularity. As consequence, these results incur the curse of dimensionality and are not applicable to high-dimensione
uses neural networks that approxi-
PDEs. The authors of (15) analyze PINNs based on three theoretical questions related to approximation, stability and generalization. Other theoretical analyses of PINNs include e.g.
30 For DeepONets, convergence rates for advection-diffusion equations are presented in 17 and a clear workflow for obtaining generic ervor
72
estimates as well as worked out examples can be found in 43 Similar results are obtained for FNOs
in 38 A comprehensive comparison of DeepONets and FNOs is the topic of [50] To the best of

---

the authors' knowledge, no theoretical results for physics-informed informed operator learning are currently available. Unrelated to the approximation error, we also report generic bounds on the expected value of the generalization error of all the aforementioned deep learning architectures, in the form of an posteriori error estimate on the generalization error.
A second goal of the paper is to prove that deep learning-base -basdd frameworks can overcome the curse of dimensionality (CoD). PDEs for which the curse of dimensionality has been overcome include linear Kolmogorov PDEs e.g. [23] [6 nonlinear parabolic PDES 32 and elliptic PDEs A [0 557.
proved for elliptic PDEs that the Deep Ritz Method (20] can overcome the CoD. Since the Barron class is a Banach algebra IC it possible that our results, which mostly only involve multiplications and additions of neural networks, can be extended to Barron functions. For PINNs, it is proven that they can overcome the CoD for linear Kolmogorov PDEs [66]. We give an alternative proof of this result, improve the convergence rate (Theorem 4 and additionally prove that PINNS can also overcome the Col) for nonlinear parabolic PDEs (Theorem A.4.. DeepONets and FNOs can overcome the CoD in many cases 43 but we note that this does not yet include nonlinear parabolic PDEs such as the Allen-Cahn equation. In Theorem 4.3 we prove that dimension-ing indieeddeddeeedennt convergence rates can be obtained if the solution is sufficiently regular. Similar results are expected to hold for e.g. elliptic 57
By assuming that the initial data lies in a Barron class, the authors [52]
PDEs by using the results from
It is evident that the generic bounds presented here can only be obtained under suitable assumptions These should always be checked to prevent misleading claims about mathematical guarantees for the considered deep learning methods. We briefly discuss how restrictive these are and whether they can be relaxed. Assuming the existence of a neural network that approximates the solution of PDE at fixed time (Assumption 3.T is of course essential, but such a result can usually be obtained by emulating an existing numerical method. Proving a bound on the Sobolev norm of that network is always possible as we only consider smooth networks. Assumption many domains, including rectangular and smooth ones. Assumption 3.4 and Assumption 3.6 also hold for a very broad class of PDEs, much like the assumption on the size of the neural network approximation in Setting 4.1 and holds for most functions of interest. Therefore, the assumption that the PDE solution is C(o -reuuar seems to be the most restrictive. However, results like Theorem 3.5 could be extended to e.g. Sobolev regular functions by using the Bramble-Hilbert lemma instead of Taylor expansions. Another restriction is that we exclusively focused on neural networks with the tanh activation function. This was only for simplicity of exposition. All results still hold for other sigmoidal activation functions, as well as more general smooth activation functions, which might give rise to slightly different convergence rates. A last restriction is that the obtained rates not optimal, but this is not the goal of our framework. In particular, for PINNs for -i-dimmnsional PDEs it is beneficial to use e.g. 26
3x3 holds for
Optimizing the obtained convergence rates and comparing with optimal ones is one direction for future research. Previously mentioned possibilities include extending to more general activation functions and less regular functions. Another direction to make the connection between our results and that of [00] where they prove that Barron spaces are Banach algebras and use this to obtain cnneenioon-indpeeddee convergence rates for PDEs with initial data in a Barron class by emulating numerical methods.
Here, we have considered the approximation and generalization errors in the present analysis. is clear that the bounds on the generalization error may not be sharp, as in traditional deep learning. Obtaining sharper bounds will be an interesting topic for further investigation. Finally, there is no explicit bound on the training (optimization) errors. Obtaining such bounds will be considered in the future.
Acknowledgments and Disclosure of Funding
The research of SM was performed under project that has received funding from the European Re- search Council (ERC) under the Furopean Union's Horizon 2020 research and innovation programme (Grant Agreement No. 770880).
10

---

References
[1] S. M. Allen and J. W. Cahn. A microscopic theory for antiphase boundary motion and its application to antiphase domain coarsening. Acta metallurgica, 27(6): 1979.
[2] G. Bai, U. Koley, S. Mishra, and R. Molinaro. Physics informed neural networks (PINNs) for approximating nonlinear dispersive PDEs. arXiv preprint arXiv:2104.05584, 2021.
[3] A. Barth, A. Jentzen, A. Lang, and C. Schwab. Numerical Analysis of Stochastic Ordinary Differential Equations. ETH Zürich, 2018.
[4] C. Beck, L. Gonon, and A. Jentzen. Overcoming the curse of dimensionality in the numerical approximation of high-dimensional semilinear elliptic partial differential equations. arXiv preprint arXiv:2003.0000 00596, 2020.
[5] C. Beck, F. Hornung, M. Hutzenthaler, A. Jentzen, and T. Kruse. Overcoming the curse of dimensionality in the numerical approximation of AllenCCahn partial differential equations via truncated full-history recursive multilevel Picard approximations. Journal of Numerical Mathematics, 28(4):197-22 222,, 2020.
[6] C. Beck, A. Jentzen, and B. Kuckuck. Full error analysis for the training of deep neural networks, 2020.
[7] J. Bernr,, P. Grohs, and A. Jentzen. Analysis of the generalization error: Empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations. SIAM Journal on Mathematics of Data Science, 2(3):63 2020.
[8] S. Cai, Z. Wang. L. Lu, T. A. Zaki, and G. E. Karniadakis. Meep Inferring the electroconvection multiphysics fields based on operator approximation by neural networks. Journal of Computational Physics. 436:1 10296, 2021.
[9] T. Chen and H. Chen. Universal approximation nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEFE Transactions on Neural Networks, (1-77, 1995.
[10] Z. Chen, J. Lu, and Y. Lu. On the representation of solutions to elliptic PDEs in Barron spaces, arXiv preprint Xrx: 10.00733, 2021.
[11] A. Cohen, R. Devore, and Schwab. Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDEs. Analysis and Applications, 901)):1 11 47, 2011.
[12] G. Constantine and T. Savits. A multivariate Faa di Bruno formula with applications. Transactions of the American Mathematical Society, 348(2):503-520, 1996.
[13] G. Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314 1989.
[14] T. De Ryck, A. D. Jagtap, and S. Mishra. Error estimates for physies informed neural networks approxi- mating the Navier-Stokes equations. arXiv preprint arXiv:2203.099 09346, 2022.
[15] T. De Ryck, S. Lanthaler, and S. Mishra. On the approximation of functions by tanh neural networks. Neural Networks, 2021.
[16] T. De Ryck and S. Mishra, Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEs. arXiv preprint arXiv:2106.144: 2021.
[17] B. Deng, Y. Shin, L. Lu, Z. Zhang, and G. E. Karniadakis. Convergence rate of DeepONets for learning operators arising from advection-die -difuttion equations. arXiv preprint arXiv:2102. 10621, 2021.
[18] M. Dissanayake and N. Phan-Thien. equations. Communications in Numerical Methods in Engineering, 1994.
saarallnetwoor-basedd approximations for solving partial differential
[19] W. J. Han, and A. Jentzen. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in Mathematics and Statistics, 5(4):499-35) 0804999-00000 2017.
[20] W. E and B. Yu. The deep Ritz method: a deep learning- based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
[21] R. A. Fisher. The wave of advance of advantageous genes. Annals of eugenics, 7(4):355-36,, 1937.
[22] S. Goswami, M. Yin, Yu, and E. Karniadakis. A physics-informed variational DeepONet for predicting crack path in quasi-brittle materials. Computer Methods in Applied Mechanics and Engineering, 391:1 14587, 2022.
[23] P. Grohs, F. Horuung A. Jentzen, and P. Von Wurstemberger. A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations. arXiv preprint arXiv:1809.02362. 2018

---

[24] P. Grohs, F. Hornung, A. Jentzen, and P. Zimmermann. Space-time enror estimates for deep neural network approximations for differential equations. arXiv preprint arXiv: 1908 008830333,,.0 2019.
[25] I. Gühring, G. Kutyniok, and P. Petersen. Error bounds for approximations with deep ReLU neural networks in W norms. Analysis and Applications, 18(05):803 859, 2020.
[26] Gühring and M. Raslan. Approximation rates for neural networks with encodable weights in smoothness spaces. Neural Networks, 134:107-130, 2021.
[27] W. H. Guss and R. Salakhutdinov. On universal approximation by neural networks with uniform guarantees on approximation of infinite dimensional maps. arXiv preprint 1rXi::110..0155, 2019.
[28] P. Henry-Laborderre Counterparty risk valuation: A marked branching diffusion approach. Available at SSRN 1995503, 2012.
[29] P. Henry-Labordere, X. Tan, and Touzi. A numerical algorithm for class BSDEs via the branching process. Stochastic Processes and their Applications, 124(2):112-1144) 2014.
[30] B. Hillebrecht and B. Unger. Certified machine learning: A posteriori error estimation for physics-infommed neural networks. arXiv preprint arXiv:2203. 17055, 2022.
[31] F. Hornung, A. Jentzen, and D. Salimova. Space-time deep neural network approximations for high- dimensional partial differential equations. arXiv preprint 2rXiv:2006.02// 2020.
[32] M. Hutzenthaler, A. Jentzen, T. Kruse, and T. A. Nguyen. A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations. SN partial differential equations and applications, 1(2): 2020.
[33] M. Hutzenthaler, A. Jentzen, B. Kuckuck, and L.. Padgett. Strong -r-error analysis of nonlinear Monte Carlo approximations for high-dimensional semilinear partial differential equations. arXiv preprint vrXiv:2110.08297, 2021.
[34] A. D. Jagtap and G. E. Karniadakis. Extended physics-informed neural networks (XPINNs): general- ized space-time domain decomposition based deep learning framework for nonlinear partial diffcrential equations. Commanications in Computational Physics, 28(5):2002-200 2020.
[35] A. D. Jagtap, Kharazmi, and G. E. Karniadakis. Conservative thysics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse prohlems. Computer Methods in Applied Mechanics and Engineering, 365:1 13028, 2020.
[36] A. Jentzen, D. Salimova, and T. Welti. A proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients. arXiv preprint 1.Xiv:1809.0755
2018.
[37] A. N. Kolmogorov. Étude de I' l'équation de la diffusion avec croissance de la quantité de matière application à un problème biologique. Bull. Univ. Moskow, Ser. Internat. Sec. A, 1:1 1:1-25, 1937.
[38] N. Kovachki, S. Lanthaler, and S. Mishra. On universal approximation and error bounds for Fourier Neural Operators. arXiv preprint arXiv:2107.07556: 2021.
[39] N. Kovachki, Z. Li, B. Liu, K. Azizzadensheli, K. Bhattacharya, A. Stuart, and A. Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108. 0848/v3, 2021.
[40] G. Kutyniok, Petersen, M. Raslan, and R. Schneider. A theoretical analysis of deep neural networks and parametric PDFs. Constructive Approximation, pages 1-53, 2021.
[41] I. E. Lagaris. A. Likas, and P. G. D. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions Neural Networks, 11:1041-1049, 2000.
[42] I. E. Lagaris, A. Likas, and D. I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5):987- 1000, 2000.
[43] S. Lanthaler. S. Mishra, and G. Karniadakis. Error estimates for DeepONets: deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1):tnac001, 2022.
[44] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations, 2020.
[45] 7. Li, N. B. Kovachki, K. Arizzadenesheli, B. Liu, K. Bhattacharya, A. M. Stuart, and A. Anandkumar. Neural operator: Graph kernel network for partial differential equations. CoRR, ahs/2003.03485, 2020.
[46] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, A. M. Stuart, K. Bhattacharya, and A. Anandkumar. Multipole graph neural operator for parametric partial differential equations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems (NeurlPS), volume 33, pages 6755-6766. Curran Associates, Inc., 2020.
[47] Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzadenesheli, and A. Anandkumar. Physics- informed neural operator for learning partial differential equations. arXiv preprint arXi::211..0394, 2021.
12

---

[48] C. Lin, Z. Li. Lu, S. Cai, M. Maxey, and G. E. Karniadakis. Operator learning for predicting multiscale bubble growth dynamics. The Journal of Chemical Physics, 154(10):104115 18, 2021.
[49] L. Lu, P. Jin, G. Pang. Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence, 3(3):218 229, 2021.
[50] I.. Lu, X. Meng, S. Cai, Z. Mao, S. Goswami, 7. Zhang. and G. Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering. 393: 14778, 2022.
[51] L. Lu, X. Meng. Z. Mao, and G. E. Karniadakis. DeepXDE: A deep learning library for solving differential equations. SIAM Review, 63(1):208 228, 2021.
[52] Y. Lu, J. Lu, and M. Wang. A priori generalization analysis of the deep Ritz method for solving high dimensional elliptic partial differential equations. In Conference on Learning Theory, pages 3196-3241. PMLR, 2021.
[53] K. O. Lye. S. Mishra, and D. Ray. Deep learning observables in computational fluid dynamics. Journal of Computational Physics, page 109339, 2020.
[54] K. O. Lye, S. Mishra, D. Ray, and P. Chandrashckar, Iterative surrogate model optimization (ISMO): An active learning algorithm for pde constrained optimization with deep neural networks. Computer Methods in Applied Mechanics and Engineering, 374:113577 2021.
[55] Z. Mao, A. D. Jagtap, and G. E. Karniadakis. Physics-informed neural networks for high-speed flows Computer Methods in Applied Mechanics and Engineering, 360: 12789, 2020.
[56] 7. Mao, I.. Lu, O. Marxen, T. A. Zaki, and G. E, Karniadakis. DeepM& for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators. Journal of Computational Physics, 447:1 10698, 2021.
[57] T. Marwah, Z. Lipton, and A. Risteski. Parametric complexity bounds for approximating pdes with neural networks. Advances in Neural Information P Processing Systems, 14:11444-5555 2021.
[58] H. P. McKean. Application brownian motion to the equation of Kolmogorov-Petunn Eetrovskii-Psskune Communications on pure and applied mathematics, 28(3):323-31, 1975.
[59] S. Mishra and R. Molinaro. Estimates on the generalization error of physics- enysics-informed neural networks for approximating class of inverse problems for PDEs. IMA Journal of Numerical Analysis, 2021.
[60] S. Mishra and R. Molinaro. Physics informed neural networks for simulating radiative transfer. Journal of Quantitative Spectroscopy and Radiative Transfer, 270: 2021.
[61] S. Mishra and R. Molinaro. Estimates on the generalization error of physics informed neural networks (PINNs) for approximating PDEs. IMA Journal of Numerical Analysis, 2022.
[62] B. Øksendal. Stochastic differential equations. Springer, 2003.
[63] J. A. Opschoor, P. C. Petersen, and C. Schwab. Deep ReLU networks and high-order finite element
methods. Analysis and Applications, 18(05):715-777 2020.
[64] J. A. Opschoor, C. Schwab, and J. Zech. Exponential ReLU DNN expression of holomorphic maps in high
dimension. Constructive Apprexiimation, pages 1-46, 2021.
[65] G. Pang, L. Lu, and G. E. Karniadakis. fPINNs: Fractional
neural networks. SIAM
journal of Scientific computing, 41:A2203-A2666, 2019.
[66]] J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li,
K. Arizzadcnesheli, P. Hassanzadeh, K. Kashinath, and A. Anandkumar, Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202. /214, 2022.
[67] Raissi and E. Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, 357:125-144: 2018.
[68] M. Raissi, P. Perdikaris, and G. F. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlincar partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[69] M. Raissi, A. Yazdani, and G. E. Karniadakis. Hidden fluid mechanics: A Navier-Stokes informed deep learning framework for assimilating flow visualization data. arXiv preprint arXiv:1808.04533 2018.
[70] C. Schwah and 7ech. Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in uq. Analysis and Applications, 17(01):19-55 2019.
[71] Y. Shin, J. Darbon, and G. E. Karniadakis. On the convergence and generalization of physics informed neural networks. arXiv preprint arXiv:2004.00 2006000000000 2020.
[72] Y. Shin, Z. Zhang, and G. E. Karniadakis. Error estimates of residual minimization using neural networks for linear equations. arXiv preprint 000002000000019, 2020.
13

---

[73] S. Wang and P. Perdikaris, Long-time integration of parametric evolution equations with physics-informed DeepONets. arXiv preprint arXiv:2106.055 05384, 2021.
[74] S. Wang, H. Wang, and P. Perdikaris. Learning the solution operator of parametric partial differential equations with physics- informed DeepOnets. arXiv preprint arXiv:2103. 1at::21033.0994 10974, 2021.
[75] Yang. Q. Du, and Zhang. Uniform L"-bound of the Allen-Calie equation and its numerical discretization. International Journal of Numerical Analysis Modeling, 15, 2018.
[76] L. Yang. X. Meng, and G. E. Karniadakis, B-PINNs: Bayesian inysies-informed neural networks for forward and inverse pde problems with noisy data. Journal of Computational Physies, 425:109913, 2021.
[77] D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103- 111103111,, 2017.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section B and Section 4
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section [
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] All assumptions are either stated in the theorem statement or described in the text right above the theorem statement.
(b) Did you include complete proofs of all theoretical results? [Yes] For each result mention where the proof can be found.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "iVxn_a-HeTC", "reviewer": "Reviewer_Bnku", "summary": "This paper proposes a framework to derive some approximation error bounds for neural networks to approximate the solution operators of initial value problems.", "questions": "Please refer to the previous section.", "limitations": "I suggest the authors to clarify the function classes of L_a, G, and U.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nLiterature overview on recent operator learning such as DeepONets and Fourier Neural Operators is well written.\n\nOn the expressive power of neural networks for operators, I would suggest the following paper as an important previous work:\n\n[GS19] Guss and Salakhutdinov, On Universal Approximation by Neural Networks with Uniform Guarantees on Approximation of Infinite Dimensional Maps, arxiv preprint:1910.01545\n\nWeakness:\nI found that the theoretical part is sloppy. First, the function classes are often lacking. For example, what is the differential operator L_a in Equation 2.1? Is G a bounded operator? In what norm? Secondly, many assumptions are incompletely stated without mentioning sufficient conditions when those are strictly satisfied. For example, in Assumption 3.1, both U^\\epsilon and G are not defined. Dependence on a is unclear. Consequently, the main theorems and arguments are hard to read for mathematicians.\n\n=== After Discussion with Authors\nTheoretical details are clarified, and now I recognize the contributions clearer. So I raised my rating.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Wl-lJ7RakfF", "reviewer": "Reviewer_jfmr", "summary": "This paper proposes a framework for analyzing the approximation  error of PINNs and operator learning. The framework was applied to obtain  error bounds for physics-informed operator learning methods for several class of PDEs. The error bounds show that dimension-independent convergence rate can be obtained if the solution is smooth enough. ", "questions": "- The current results only prove the existence of some neural works achieving certain desired error accuracy. Can the authors comment on how the neural networks can be obtained? Especially, if one consider the empirical risk minimization approach, how does the generalization error scales?\n\n- Theorem 4.6 concerns the approximation error of physical informed operator networks. The result is stated for the loss function $\\|\\mathcal{L}(\\mathcal{G}_\\theta)\\|_{L^2}$. Can the author elaborate on whether this result implies an error bound for the operator itself, e.g. a bound for $\\|\\mathcal{G} -\\mathcal{G}_{\\theta}\\|$? In addition, the Gaussian measure $\\mu$ of the input functions requires subexponential decay in the KL expansion coefficients. Can this be relaxed?\n\n- Is $p^\\ast$ used in the statement of Assumption 3.5? If not, can the authors define it outside the Assumption?\n    \n- In Assumption 3.1, `` ...  there exist ... \" there exist should be `` ... there exists ... \"\n\n- In Theorem B.5 of Appendix, are the indices $i$ and $j$ the same in equation (B.15)?\n", "limitations": "The authors adequately addressed the limitations in the discussion section. I am not ware of any potential negative societal impact of their work.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " - Strengths. This is a nice work on the error analysis of PINNs and operator learning, and to the best of my knowledge the first theoretical work on physics-informed operator networks. The main results are interesting, nontrivial and important for understanding the performance of operator learning in PDEs. The paper is very well written and easy to read. \n      \n- Weaknesses. The paper is strong enough to be accepted. I do not have much to comment on the weakness, except minor comments on the few main results and typos. See the section of Questions for more details. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "TdOjw3L8zIR", "reviewer": "Reviewer_hG4A", "summary": "In this paper, the authors provide each bound and its connetions on PINN, DeepONet, FNO, physics-informed DeepONet, and physics-informed FNO. These general connections are also applied to show that the neural network representation can overcome the curse of dimensionality.", "questions": "## Questions\n* The discussion says that assumption 3.3 is a natural assumption for various PDEs, but I wonder if this is correct. For example, I wonder if it is an assumption that can be easily made for the examples used in Burgers' and other operator learning papers.\n* There is confusion in the notation of $u_\\theta$, $\\mathcal{G}_\\theta(v)$ and $\\mathcal{u}^\\varepsilon$. Assumption3.1 and Assumption3.3 are assumptions related to $\\mathcal{u}^\\varepsilon$ and $\\mathcal{G}_\\theta(v)$, but Theorem 3.4 uses these assumptions to create $u_\\theta$ and $\\mathcal{L}(u_\\theta)$. In case of PINN, $u_\\theta$ is learned while $v$ is fixed.\n* For the term $\\|u_\\theta-u\\|_{L^2(\\partial([0,T]\\times D)}$ on the left side from the trace inequality in (3.4),  I wonder if this term affects the fact that there exists a nn that can sufficiently reduce the PINN loss. \n\n## Minor\n* Assumption 3.5. there is no $p^*$ is not in inequality (3.5)\n* p.5 194 What is $L^2_N$?\n", "limitations": "The authors noted their limitations and future works well in the conclusion section.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* While various studies measured the approximation error for a specific PDE for one method among PINN, FNO, and DeepONet, this paper measured the error by connecting all methods at once for the first time.\n\n## Weaknesses\n* Theorem 3.4, Theorem 3.8, and Theorem 3.9 talked about the existence of nn that reduces the PDE residual. I can't guarantee it's a good match. These theorems only talk about the possibility of sufficiently reducing the loss. Furthermore, I wonder if the boundary condition or initial condition is not considered. Without these conditions, I am not sure if reducing $\\|\\mathcal{L}(\\mathcal{G}_\\theta)\\|$ enough will lead to the convergence to the exact solution.\n* [9] as well as [*] is a study showing that the curse of dimension can be solved when the solution of PDE is approximated by nn. Regarding these papers, it is necessary to explain the differences between Theorem 4.2 and Theorem 4.4 described in Section 4.1.\n\n\n\n\n[*] Marwah, Tanya, Zachary Lipton, and Andrej Risteski. \"Parametric complexity bounds for approximating PDEs with neural networks.\" Advances in Neural Information Processing Systems 34 (2021): 15044-15055.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept