On the Representation of Solutions to Elliptic PDEs in Barron Spaces
Authors: Ziang Chen
Jianfeng Lu
Yulong Lu
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: partial differential equations
neural networks
Barron norms
high dimension
approximation

Abstract: Numerical solutions to high-dimensional partial differential equations (PDEs) based on neural networks have seen exciting developments. This paper derives complexity estimates of the solutions of $d$-dimensional second-order elliptic PDEs in the Barron space, that is a set of functions admitting the integral of certain parametric ridge function against a probability measure on the parameters. We prove under some appropriate assumptions that if the coefficients and the source term of the elliptic PDE lie in Barron spaces, then the solution of the PDE is $\epsilon$-close with respect to the $H^1$ norm to a Barron function. Moreover, we prove dimension-explicit bounds for the Barron norm of this approximate solution, depending at most polynomially on the dimension $d$ of the PDE. As a direct consequence of the complexity estimates, the solution of the PDE can be approximated on any bounded domain by a two-layer neural network with respect to the $H^1$ norm with a dimension-explicit convergence rate.

On the Representation of Solutions to Elliptic PDEs in Barron Spaces
Ziang Chen Department of Mathematics Duke University Box 90320, Durham, NC 27708 zi ziang@math. duke. edu
Jianfeng Lu Departments of Mathematics, Physies, and Chemistry Duke University Box 90320, Durham, NC 27708 jianf fanfeng@math. duke. edu
Yulong Lu
Department of Mathematics and Statisties Lederle Graduate Research Tower University of Massachusetts
710 N. Pleasant Street, Amherst, MA 01003 lu@math. umass. edu
Abstract
Numerical solutions high-dimensional partial differential equations (PDEs) based on neural networks have seen exciting developments, This paper derives complexity estimates of the solutions of d-dimensional second-order elliptic PDEs in the Barron space, that set of functions admitting the integral certain parametric ridge function against probability measure the parameters. We prove under some appropriate assumptions that the coefficients and the source term of the elliptic PDE lie in Barron spaces, then the solution of the PDE is e-close with respect to the H norm to a Barron function. Moreover, we prove dimension- explicit bounds for the Barron most polynomially on the dimension of the PDE. As direct consequence of the complexity estimates, the solution of the PDE be approximated on any bounded domain by a two-layer neural network with respect to the H! norm with dimension-explicit convergence rate.
of this approximate solution, depending
Introduction
Inspired by the tremendous success of deep learning in diverse machine learning tasks including image classification, natural language processing, and artificial intelligence, there has been growing interest in exploring scientific and engineering applications of deep learning 36 As partial differential equations (PDEs) play a fundamental role in almost all branches of sciences and engineering, numerical solutions to PDE problems based on neural networks have become
32
26
47].
important research direction in scientific machine learning [25 6 23 17
7 10 (2). Among the
various directions, numerical solutions high-dimensional PDEs the unknown function depending
on many variables are perhaps the most exciting possibility, as solving such PDEs has been long-
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

standing challenge and breakthrough would lead to tremendous progress in fields such as many-body physics 4 [8) multiple agent control 335 just to name a few.
Numerical solutions to (ow-dimensional PDEs, such as Navier-Stokes equation in fluid dynamics, has become a standard practice after decades of work. However, the computational cost of the conventional numerical methods for PDEs grows exponentially with the dimension, as a manifestation of the curse of dimensionality (CoD). Given a target accuracy e, conventional methods, such as finite element or finite difference, would need a mesh size of (ee), and thus degree of freedom on the order of O(e where d is the dimension of the problem. Such complexity severely limits the numerical solutions to PDEs in high dimension, such the many-body Schrödinger equations from quantum mechanics and the high-dimensional Hamitton-Jacobi-Bele BeieoobbBBllllann equations from control theory. Neural networks, in particular deep neural networks, provide promising way overcome the CoD in representing functions in high dimension. It is thus a natural idea to parametrize the solution ansatz to PDE as neural networks and to employ variational search for the optimal parameters. Various neural network methods [25 6 7
16, |] for PDEs have been proposed recently and some of them have demonstrated great empirical success in solving PDEs of hundreds and thousands of dimensions [7 10 17,, much beyond the capability of conventional approaches. Question remains though on theoretical analysis of such neural networkk based methods for solving high- nigh-dimensional PDEs. While there have been some recent progress on approaches including thysics-informed neural
TO 33 77 46
networks [37 31 38 and the deep Ritz. method 28 27 many questions still remain open. Among
them, a fundamental question is
Whether the solution of a high-dimensional PDE can be efficiently approximated by neural network, and if so, how to quantify the complexity of the neural network representation with respect to the increasing dimension?
The focus of the current study takes a functional-analytio approach to this question. Namely, identify function class suitable for neural network approximations and prove that the solutions to a class of PDEs can be well approximated by functions in this class. More specifically, the PDE we consider is a family of second-order elliptic PDEs of the form
Our contributions
Lu = v (AVu) + cu = f on Rd.
(1.1)
We choose to work with the Barron class of functions defined in [ (see also (.) which is class of functions admitting the integral of certain parametric ridge function against a probability measure on the parametens; see Definition 2.2 for precise description. This Barron space is inspired by the pioneering work by Barron 22 where he proved that a class of functions whose Fourier transform has the first order moment can be approximated by two-layer networks without CoD. The main result of our work, stated informally, is the following; a more precise statement can be found in Section 2.3
Main Theorem (informal version) If the coefficients A, c and the source term f of the second- order elliptic PDE (I.I) are all Barron functions, then the solution u" can be approximated by another Barron function such that ||u w" || S 6, where the Barron norm of is upper bounded by ((d/c)C Moreover, if the Barron space is defined by the cosine activation function, then the upper bound on the Barron norm can be improved to OldClog (/dClog///) /r).
We note that while the better rate only obtained for the cosine activation function, such periodic activation function has indeed been found effective in certain PDE related tasks, see e.g., 42
Since the Barron functions can be approximated on a finite domain Ω w.r.t. H norm by two- layer neural networks with a rate O(1/V)) where k is the network width (see Theorem 2.5 the theorem above directly implies that there exists a two-layer network uh with the number of widths &- = ((dfe) ((dd/))Cog((/))), or k = O(t log 1/e) if the activation function is cosine, such that llue n= (()) S Therefore in our setting the solution can be approximated by two-layer neural network without CoD, namely the complexity depends at most polynomially on the dimension d for fixed c. Alternatively, we can rewrite the rates as ((1/e) d+log 1/e)) and ((1/e) Clog d) to contrast with that of conventional grid-based numerical methods for PDEs, which scales as ((1/c))). We observe that the dependence on d is replaced with log d in the complexity bound for neural network approximations.
We emphasize that such approximation result does not follow directly from the universal approxima- tion property of neural networks for Barron functions since it is not priori known that the solution to

---

the PDE is a Barron function. In fact, directly imposing regularity or complexity assumption on the solution itself unreasonable since the solution is unknown and its fine properties are generally inac- cessible. Our main contribution is to establish the fact that the solution can be indeed approximated by a Barron function, under the assumption that coefficients and the right hand term of the PDE are Barron. From a mathematical point of view, our main theorem is in the same spirit regularity estimates of PDEs, which are of crucial importance in the study of PDEs. While such regularity estimates are well developed in low dimension, the extension to results in high dimension is highly non-trivial and is the main focus of our work.
Related works Several theoretical work have been devoted to the above representation question. It has been established in 15 20 14] that deep neural networks can approximate solutions to certain class of parabolic equations and Poisson equation without CoD. The major limitation of those work lies in that the PDEs considered in those work must admit certain stochastic representation such as the Feymann- Kac formula and it seems difficult to generalize the proof techniques to broader classes of PDEs with no probabilistic interpretation. The work [28] 27 analyzed a priori generalization error of two-layer networks for solving elliptic PDEs and the Schrödinger eigenvalue problem on a bounded domain with Neumann boundary condition by assuming that the exact solutions lie in certain spectral Barron space, where the later was rigorously justified with a new regularity theory carried out in [29] for second-order PDEs and in [9] for general even-order clliptic PDEs, but without the solution. Compared to those work, work focuses on deriving complexity estimates of the solution in the reeegraalrppeeeettttionn--- based Barron space, which is more flexible and arguably more suitable for high-dimensional settings, see e.g., discussion in [8. The work (9] established such estimates in the Barron space for certain specific PDEs that essentially admit explicit solution, whereas we aim prove such estimates for general elliptic PDEs for which the analytical ansatz is not available. The work (30) is closest to ours where the authors proved that the solution of the same type of elliptic PDE with Dirichlet boundary condition can be approximated by (deep) neural networks with at most O (ppoly(d)N) parameters if the coefficients of the PDE are approximable by neural networks with at most N parameters, While our overall approach based on iterative scheme borrows idea from 330. our result differs and improves theirs in many aspects: (1) Our result shows that the solution can be well approximated without CoD by a two-layer neural network with a single activation whereas the result in [30 requires deep network which uses mixure of least two activation functions; (2) Our PDE is set up on the whole space rather than a compact domain, so our setting covers some important PDEs in physics, such as the stationary Schrödinger equation; (3) The result in 30 relies on another key assumption that the source term lies with the span of finitely many eigenfunctions of the elliptic operator whereas our result completely removes such assumption. This is achieved by utilizing a novel preconditioning technique to uniformly control the condition number of the iterative scheme that underpins the proof of our main theorem.
of the PDE solutions in the spectral Barron space. Similar generalization analysis
justifying the Barron assumption
Organization The rest of this paper will be organized as follows. In Section 2.1 we set up the PDE problem on the whole space and in Section 2.2 we introduce the dcfinition of Barron functions and diseuss their H -aapproximation by two-layer networks (see Theorem 2.5p. Our main theorems are stated in Section 2.3 We present the sketch proofs of the main theorems in Section 3 and defer the complete proof to Appendix. The paper concluded with discussions on some future directions.
2 Problem setup and main results
2.1 Problem description
Notations Throughout this paper, we use ||a||| to denote the Euclidean norm of vector E Rd. For a matrix A E 1ddxd, we denote its operator norm by ||A||| = supoER (fo}
||All
|o] For R > 0, we
denote by By, the closed ball in Rd centered 0 with radius R, i.e., B {a E |||||| < R).
Recall that we consider the d-dimensional second-order elliptic PDE [.Ip. To guarantee the existence and uniqueness of the weak solution in H (R=)., we make the following minimum assumptions on coefficients A, e and right-hand side f; this assumption will be strengthened in our main representation theorem.
3

---

Assumption 2.1. A(x) = (Aig(z)) is symmetric with |A(x)] S amax < 00 and uniformly
elliptic, that is for some amin > 0, satisfies
A(x) 2 amin IE Vz,E E R
We also assume that 0 < Cmin S c()) S Cmax < <0 and f E (R(d)..
Under Assumption 2.1 standard argument using the Lax-Milgram theorem implies that there exists unique weak solution u" E H (R*), such that Lu = f H (T-((R)) which is the dual space of H- (BB)), i.e.,
f cu' cdd'dr= = J Suda, E (R.).
AVu" Vedr + Jyc
Our ultimate goal is to show that the solution can be approximated by a two-layer neural network on any bounded subset of R with respect to the H norm with rate scaling at most polynomially in the dimension. Notice that in general one cannot hope to obtain an approximation result on the whole space I because the asymptotic behavior of neural network function (determined by the activation) at infinity may mismatch that of the target function u. On the other hand, it is well-known that the convergence rate of neural networks for approximating functions in standard Sobolev or Hölder spaces still suffers from the CoD 44 45). Therefore to obtain a rate without CoD for the neural networks approximation function space which has low complexity compared to Sobolev or Hölder spaces. We will work with the Barron space and show that u" is arbitrarily close to Barron function which can be approximated by a two-layer neural network without CoD.
the solution u²", we need to argue that w' lies in suitable smaller
2.2 Barron spaces
The definition of Barron space is strongly motivated by the two-layer neural networks. Recall that two-layer neural network with i hidden neurons a function of the form
1
(z)
(o + bi),
E ERR..
(2.1)
Here R - IR is some activation function and (a;, wy, by) E R x x 1,2, A are the network parameters. If the parameters are randomly chosen accordingly to some probability distribution, then in the infinite width limit the averaged sum in (2.1) formally converges to the following probability integral
i=
up(x)= = f ao x + b)p(da, dw, db), x Rdd
(2.2)
where is a probability measure on the parameter space IR x Rd x R. Observe that 2. is a special instance of (2.2 if we take p(a, w, b, E 5(a a, wi b b.).
The Barron norms and Barron spaces are then defined as follows, where we require the marginal measure in w to have compact support. This is because that the (formal) first-order and second-order partial derivatives of (x) would involve with components of w by chain rule. By adding some uniform bounds on w, can control the Barron norms after taking derivatives. In the subsequent discussion, we may also need to restrict our attention on functions defined on a bounded set. Therefore we present below the formal definition of a Barron function defined any domain Ω C Rd.
Definition 2.2. Fix Ω C. I&d and R. E [0, +oo]. For a function = -up with some probability measure p. define the Barron norm of g on Ω with index p [1, +oo] and support radius R by
-inf (ar(()
(J (/aaP IalPo(da, dw, db)
4/P
1991
9
aowww T x x++ b)p(da, du, db) on Z2
p is supported on B
x
where Bir fx Rd |||||| & R). The corresponding Barron space is then defined as (7) = {y= |11 < x

---

It is worth making some comments on the definition above. Our definition of Barron space adapts similar definition in (see also [T] with several important modifications for the purpose of PDE analysis. First we require that the w-marginal of the probability measure has compact support in order to control the derivatives of a Barron function defined in (2.2); in fact differentiating the integral of (2.2) leads to an integral of the product of the ridge function with w (or its powers) and enforcing p has a compact w w-maaginal thus controls the Barron norm of the derivatives of up. In addition, our definition of Barron norm only involves the p-th moment of p with respect to parameter whereas the Barron norm in [ takes the moments all parameters into account. This is because [8] uses the unbounded ReLU activation function, which requires the moment condition in all parameters to make the integral in (.2) well-defined; whereas we will only consider bounded a (see Assumption 2.3 and the integral is guaranteed to be finite under such assumption.
Both our notion of Barron space and the one are motivated by the seminal work of Barron where he proved that if the Fourier transform F(f) of a function f satisfies that
[ [FNEEEEE < 00,
then there exists a two-layer network ug with & hidden neurons such that |l - ux [ I CCk- Since Barron's s original function class is defined via the Fourier transform, call such function class the spectral Barron space to distinguish it from our Barron space based on the probability integral. We refer
24
40 28
for recent developments the spectral Barron space.
As we investigate the solution theory of the second-order PDE the Barron space, we expect to differentiate the integral representation (2.2) up to the second order. Therefore, assume that the activation function a as well as its first-order and second-order derivatives are all bounded R.
Assumption 2.3. 0 IR IR is smooth with Co suPyER (a()) oo, C1 == SuPyER (a'(v) co, and SuPyER Jo" [v)) < co.
Thanks to the Hölder inequality, it clear that (9)) ( when p S q. The following useful proposition (see also S8 Proposition 1J) shows that the reverse is also true and that the Barron norms and the Barron spaces in fact independent of p.
Proposition 2.4. For any function g E B [ , it holds that ||g| | (0) | = 119ll (1)) for any S S co. As consequence, BR (S2) = (92) = (22) for s
The proof of Proposition 2.4 can be found in Appendix B
The most important property that makes Barron functions distinct from Sobolev or Hölder functions is that they can be approximated by two-layer neural networks with inneension-indppendde approximation rate in H' norm as shown Theorem 2.5
Theorem 2.5 (Approximation theorem in H norm). Suppose that Assumption 2.3 holds and that (92). Then for any open bounded subset Q4 C Ω and any E N4, there exists {(a,, wi, b satisfying
2(C3 + RRC))m((O)) (1
Σ (w) x by)
g(x)
(2.3)
(10
where Co and C1 are the constants in Assumption
2.3 and m mQ2)) is the Lebesgue measure of Po-
Theorem 2.5 provides an H' aappoxximation rate for Barron functions defined by the integral rep- resentation 2.2). The proof is deferred Appendix [B] Similar approximation results in the sense of L for Barron functions (including formulations based on spectrum and integral representation) have been proved in 2 24
39 8. H approxxiiiatooonnn results for spectral Barron functions were
previously obtained in and 28
2.3 Main theorems
To state our main theorems, we need to make some additional complexity assumption on the coefficients A, e and the source term f the PDE (L.). which reasonable as otherwise there no hope that the solution would lie in smaller function class.
5

---

Assumption 2.6. For some RA, Re, Ry E (0, +00), we have la = maxigi,sa (iiy B/y (24) < le |cllB] (B4) < 00, and E == [[]lB,,,())) (B4) < co.
We remark that Assumption 2.6 is compatible with earlier Assumption 2.1 on the coefficients A, e and the source f. In fact, it is easy to see that constant coefficients A, c satisfy both assumptions if im)o) + {0]. i.e., a is not constantly zero. As for provide in Proposition of Appendix A a concrete class of f that satisfies both assumptions.
We also need two additional technical assumptions the activation function.
Assumption 2.7. The function h : R.² - IR (U1, + (11)0(U2)) satisfies that lm [1bl (B2) 3c. for some Rm, E +00).
Assumption 2.8. It holds that == || llsh/, (R) < o0 and la, == ||o" 118k (R) < for some
Rd,1 Ra,2 E (0, +o0).
Assumption 2.7 and Assumption 2.8 guarantee that Barron spaces are closed under multiplication and differentiations (up to the second order) respectively; see Lemma 3.3 (iii)-(iv) for precise statement. These operations and the associated closeness will useful for constructing approximation to the exact solution u" of the PDE (1.I) in Barron spaces. Proposition A.2 shows that Assumption 2.7 2.7and and Assumption hold for a relatively large class of activation functions including cosine.
With the preparations above, we are ready to state our main theorems below. The first main theorem concerns the complexity estimate of the exact solution u" in the Barron space.
Theorem 2.9. Suppose that Assumption 2.7 2.3
2.6
2.7 and 2.8 hold. For any e E 10,) there
exists u E B) (IR)) with R S (+) and |u]] (RR)) S B1 (2) De Palnel succh that ||u u² H (R S
e. Here ni Y2, 31, and B2 only depend on I| 1(4)) and constants in Assumptions 2.1 2.6 2.7 and 2.8
Furthermore, if a cos, then 1/u w² H (B.) S e can be achieved with R < V Ind and ||]]s,, (R2) S 3 dilllll,,, where 7h1 B1, and B3g only depend on I|. and constants in Assump-
tion 2. and 2.6
Theorem 2.9 shows that the exact solution u is e-close (in the sense of H') to a Barron function B ( In addition, the Barron norm of u grows at most polynomially in d, indicating that the complexity of u dose not suffer from the CoD. Also the complexity estimate gets substantially improved when the activation function is cosine. In fact, advantages of periodic activation functions
have been empirically observed in some earlier works, see e.g., [42]. J. It remains an open question whether results similar Theorem 2.9 can be established for activation functions that do not satisfy Assumption 2.7 and Assumption 2.8 This will be investigated in future works.
Thanks to Theorem 2.5 and Theorem 2.9 it is easy to conclude that the PDE solution u" can be approximated on any bounded subset 12 Rd using two-layer neural networks with the number of hidden neurons & scaling at most polynomially in d.
Theorem 2.10. Under the same assumptions as in Theorem 2.9 given any c (0,1/2) and any open Blinel
bounded subset Q C RRd, there exists a two-layer neural network ux(x) with i S 7mm(5)) (4) (
such that ||u - w²1 (2) < where y and 3 only depend on [ and constants in Assumptions 2.7
2.7 and
Furthermore, ifa cos, then I|u u" (()) S c can be achieved with ls < //mmm/mm)))))) (2)dd" [n where
n' and B' only depend on [| 1(24) and constants in Assumptions' 2.7 / 2.3 and 2.6
3 Proofs of the main results
We sketch the proof ideas in this section and present the full details in the Appendix.

---

3.1 Preconditioned functional iterative scheme
The key ingredient of our proof of Theorem 2.9 is functional iterative scheme for solving the elliptic PDE, which can be viewed an infinite dimensional analog of the preconditioned steepest descent algorithm to solve linear algebra equations. Recall when solving the linear equation Ar = b with A xx and z, b IR, the preconditioned steepest descent algorithm [33 runs the iteration
31+1 = 2t - aP - b),
where is preconditioning matrix, a is the step size, and 0, 0, indicates the iteration index. The purpose of the preconditioned iteration is to reduce the condition number of the iteration ((PA) by choosing a suitable and hence accelerate the convergence of the iterative algorithm.
In the case of solving the elliptic PDE 1.1 we generalize the preconditioned steepest descent iteration the functional setting by considering the following iteration scheme in H (IR"))
U++1 4 a (Cu ff).
(3.1)
where the inverse operator (1 -) plays the role of preconditioner. As a matter of fact, we will show that the condition number of (I- - bounded and this directly implies that the iterative scheme (3.T) converges exponentially to the exact solution w". Indeed, we have the following contraction estimate for the iteration (3.1 whose proof can be found in Appendix C
Proposition 3.1. Recall the constants Omin: Gmax; Crin; Cmax defined Assumption 0 and any E H (R.).
For any
- a(I- -)-) L)) S Maa)llllaa
(3.2)
where the contraction factor A(a) = (ampeemmmnnaaee )max)] |1 - ad1 with Amin min(amin Cuin } and Amax max Omnxx, Cmax). )
In particular, minimizing A(a) with respect to the step size a yields an optimal choice of step size
2
Amin + Amax
With a. in (322, we obtain that
(I- Hi(Rd) Amax + Amin
Amax Amin [[l]|
(3.3)
Amia + Amax
As direct consequence, obtain the following estimate for the number of iterations required to achieve given error tolerance.
Corollary 3.2. Let u² be the exact solution of the PDE (.I). Under Assumption 2.7 iteration scheme (.1 with a a. A Then for any
consider the Iuu u² (M(R))
T2 In Amax + Amia
In
Amax Amii
the iterate up satisfies ||u u² (1 S.c.
Let us remark that the idea of using iterative scheme to establish neural network representation results of solutions to PDEs is new, see e.g., [23 330, similar ideas have been also used construct neural network architectures inspired from iterative schemes, see e.g., 12). Closely related to our setting, the work (30) uses steepest descent iteration with the right hand side of the equation assumed to be in the span of first several eigenfunctions of the elliptic operator, while [23] considered general right hand side, but only after discretization which also effectively truncates the problem onto finite dimensional subspace. These restrictions were made to limit the condition number of the iteration. Unlike those works using standard steepest descent iterations, by using the preconditioning technique, can deal with general right hand side without restricting to miteediimensional subspace.

---

3.2 Algebra of Barron functions and representation of the solution
Corollary 3.2 in the previous subsection shows that we can obtain an approximate solution by running the iteration (3.1). To complete the proof of Theorem 2.9 we show in this subsection that the iteration (3.1) can be carried out in the Barron space B (IR ", Lee. each iteration u B (IR)) (with the support radius R potentially depending on 1). To this end, we first need to establish the closeness of
Barron space under function operations involved in the iteration. fact, by decomposing each of the
iteration step in (3.1) into two steps, we can write
Up f - U4+1 U. u4 a, A)-
+ Ac,Oeju)) + cut - S,
(3.4)
Thus, to show that the iterate u2 remains in Barron space, it suffices to establish that addition, scalar multiplication, product, differentiation, and action of (I A)-1 are closed in the Barron space. The closedness of Barron functions under those operations are not only useful for proving our main results, but also of its own interest. The next two lemmas summarize the algebras and the stability estimate of the inverse (I - A) A in the Barron space. Their proof' can be found in Appendix D
Lemma 3.3 (Algebras in Barron spaces). The followings hold:
(i) (Addition) Suppose that llgille,, (RY) < 00, i= 1,2, ..... k. Then 11 +.... + 9& 0ullB,,,R S Eigish l9ills,, (aty where R = maxigisk Ri.
(ii) (Scalar multiplication) Suppose that [| (R()) < oo and that E R. Then [1glls)
A 1l
(iii) (Product) Suppose that Assumption 2.3 and Assumption 2.7 hold and that 1|9: 19:1l1s,, (R4) < 00
for = 1,2. Then [191921|B), S fm |19| l8, (R-)
l19 B) (Bay where R Rm Ri1 + R2)
with Rm and m being constants in Assumption 2.7
(iv) (Derivatives) Suppose that Assumption 2.3 and Assumption with R < cc. Then [|0:g]ls,
2.8 hold and that (1llB,,,
S La l1 (lglss, (R and [10.gg [10,ggll (
ng, (R))
R2 |99l (Ra)
(R for any i, E ( d}, where Ra,1- Ra,2- la,1, and la. are constants in Assumption 2.8
Lemma 3.4 (Applying (I A)-1 on Barron functions). Suppose that ||øl|a, < co. Then
-)- | olls), (1)) I |1 (Rd))
we include a proof for Lemma 3.4/in Appendix D that uses similar arguments from (. though the analysis in [9] is for d 2 3. The lemmas above lead to the following recursive estimate on the Barron norm of ug-
Lemma 3.5. Suppose that Assumption 2.3
Assumption 2.7 and Assumption 2.8 hold. If [|ulll
oo with Ra, < cc. then u(+1 defined in (.. or (3.4) satisfies that
|| By (Rd) I (alm Aa ( R Ry Ra + almlo + 1) [lelloo, (Ra) + all,
(3.5)
for any
Ra,t+1 2 max Rmm Ra,1 (Ru, + RA), Ry (Ra,2 Ru,t + RA), Rm (Ru,t + Re), Ru,, Ry). (3.6)
The proof of Lemma 3.5 is deferred Appendix D One observation is that the amplification factor of the Barron norm in Lemma 3.5 increases as the support radius R increases. The reason is that differentiating the function would introduce components of w and hence the amplification depends on how large [|]l can be and thus the support of the measure.
One possible direction to improve the estimate is to realize that the preconditioner (I A) can counteract the action of taking derivatives. It is indeed possible to to remove the R dependence from the amplification factor, at least for some specific activation functions, through more careful analysis. In particular, we have the following lemma for the cosine activation function, the proof of which can also be found in Appendix D
8

---

Lemma 3.6. Suppose that Assumption 2.6 holds. If cos and ||
| Bi. (Rt) 00 with Ru, < co.
then 4441 defined in B.1b or (3.4) satishes
| B8 (R4) S (6ala max max 1dd + ale 1) llue 11s 1yy,,, (R4) + alyy
(3.7)
for any
Rus+1 2 Ru, + max (RA, Re: Ry)-
(3.8)
Lemma 3.5 and Lemma 3.6 estimate the amplification of the Barron norm in each iteration of (3.I) Combining them with the control of number of iterations, Corollary 3.2, are ready finish the proof of Theorem 2.9
Proof of Theorem 2.9 Fix uo and Amin Amaax
According to Corollary 3.2 it holds that
|ur u²" (R.) I for any
Amax + Amin Amin
|| (1a
22 In
In
Amax
Moreover, thanks to the estimate
Amin || (R.) S AVu" Vu dr + cuu'dxx / I dr S [I ilfll=- (x))))
have ||u" (R() I Amin
|1 11 Therefore, it suffices to take
T ( Amax + Amin
1 - In
[In Amax + Amin
|| 1(20))
> (
In
Xmax
Aminn
Amnin
Amii
13 Rus 2 Ru,t- Then (3.6) is satisfied for any L. Let us define sequence (Xi)120 via Xo and X
Set Ru.0 = max Feax(Ren Ra, Ro, RG, f1.1 and Ruge+1 = mnax(2Rm. Ra,1, Ra,2, 28m:
telmeallr k4,2) + alfntetes)41) d X, By (355), we have || lll (B4) X, for any
t. Therefore, it holds that
RU, max{RA RA. Res R2, 17 m2x Ra, 2m Ra,2, 28m, 1)"
and that
||urlles Bo R (34) I Xr
Calmealla la,2) (( alemle +lg) +1 +1)
d (Ru,o ) Ru,
&
Cs + la,2) Allmletly +
d2T
&
(max (RA, Re: Ry, 1))T m2x Ra,1 2pm Ra.2. 2nm 1)72
The first part of Theorem 2.9
established by setting u ur and Ru,T.
a cos, 3.8 satisfied by setting
Ru, = max ( (RA Re: R L.
Define Yo = 0 and Y++1 = ( max ( R2, 1e + al, + 1) Y, + aly. By 3.7
we obtain that
| llgg R... (R4) < Y2 for any t, and in particular that
||r|l By (Rd) < Yy aly ((aalaaax(R3) 1)d2 + al + 1) - 6all max It2 4: c + al
which finishes the proof by setting u = ur and R R.,T.
Theorem 2.10 is then corollary of Theorem 2.9 and Theorem 2.5 (the approximation theorem).
Proof of Theorem 2.10 Theorem 2. 10 follows directly from applying Theorem 2.5 with error toler- ance 9/2 and applying Theorem 2.9 with error tolerance s/2.

---

4 Conclusion
In this work, we establish the approximation rate for the solution second-order elliptic PDE by Barron function and by two-layer neural network. Under the assumption that the coefficients and the source the PDE are all the Barron spaces with some compact support property on the underlying probability measure, the approximation rate is shown to depend at most polynomially the dimension. Therefore, our results indicate that even a neural network as simple as a two-layer network with single activation function can have adequate representation ability to encode the solution of an elliptic PDE, without incurring the CoD. Our result provides theoretical guarantee for numerical methods for solving nigh-dimensional PDEs using neural networks.
For future directions, it is of interest to extend the functional analysis framework to more general activation functions (such as unbounded ones) and more general neural network architectures. One interesting direction is to establish depth separation result for representing PDE solutions. Our analysis also indicates some potential benefit of using periodic activation function such cosine in terms of approximation, further studies and understanding of the choice of activation function and architecture crucial. Morcover, while focus approximation error, generalization error and analysis of training should also be considered in future works.
It is possible to extend the approximation results to a wider range of high-dimensional PDEs such as parabolic PDEs, PDE eigenvalue problems, and nonlinear equations such as those arise from control theory. The analysis tools and characterization of Barron space we establish in this work would be useful for these future studies.
Acknowledgments and Disclosure of Funding
The work of Z.C. and J.L. is supported in part by the National Science Foundation via grants DMS- 2012286 and award DMS-2107934.
Y.L. thanks the National Science foundation for its support through the
References
[1] F. Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1):629-68)) 681, 2017.
[2] A. R. Barron. Universal approximation bounds for superpositions of sigmoidal function. IEEE Transactions on Information theory, 39(3):930-945, 1993.
[3] G. Bresler and D. Nagaraj. Sharp representation theorems for ReLU networks with precise dependence on depth. arXiv preprint arXiv:2006.0400 2020.
[4] G. Carlco and M. Troyer. Solving the quantum many-body problem with artificial neural networks. Science, 355(6325):602--6 6066 2017.
[5] F. Chen, J. Huang, C. Wang, and H. Yang. Friedrichs learning: Weak solutions of partial differential equations via deep learning, arXiv preprint arXiv:2012.08022 2020.
[6] M. W. M. G. Dissanayake and N. Phan-Thien. Neural-network- based approximations for solving partial differential equations. Communications in Numerical Methods in Engineering, 10(3): 191-201, 1994.
[7] W. E, J. Han, and A. Jentzen. Deep learning- based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Commut nications in Mathematics and Statistics, 5(4):349-380, 2017.
[8] W. E, C. Ma, and Wu. The Barron space and the flow-induced function spaces for neural network models. Constructive Approximation, pages 1-38, 2021.
[9] W. and S. Wojtowytsch. Some observations on high-dimensional partial differential equations with Barron data. arXiv preprint arXiv:2012.01488 2020.
10

---

[10] W. E and B. Yu. The deep Ritz method: a deep learning based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
[11] X. Gao and L.-M. Duan. Efficient representation of quantum many-body states with deep neural networks. Nature Communications, 8(1):1-6, 2017.
[12] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In International Conference Machine Learning, pages 1263-1272. PMLR, 2017.
[13] G. H. Golub and C. F. Van Loan. Matrix Computations. JHU Press, 4th edition, 2013.
[14] Grohs and L. Herrmann. Deep neural network approximation for high-dimensional elliptic PDEs with boundary conditions. arXiv preprint arXiv:2007.05384, 2020.
[15] Grohs, F. Hornung, A. Jentzen, and P. Von Wurstemberger. A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of black-scholes partial differential equations. arXiv preprint arXiv: 000622, 2018.
[16] Y. Gu, H. Yang. and C. Zhou. Selectnet: Self-paced learning for high- nigh-dimensional partial differential equations. Journal of Compulational Physics, 441: 10444, 2021.
[17] J. Han, A. Jentzen, and W. E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505- -515505558810,,,- 2018.
[18] J. Hermann, Z. Schätzle, and F. Noé. Deep-neural-networrk solution of the electronic Schrödinger equation. Nature Chemistry, 12(10)891-897, 2020.
[19] Q. Hong, J. W. Siegel, and J. Xu. A priori analysis of stable neural network solutions to numerical PDEs. arXiv preprint 1xXxx22004002000 2021.
[20] M. Hutzenthaler, A. Jentzen, T. Kruse, and T. A. Nguyen. A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations. SN Partial Differential Equations and Applications, 1(2): 2020.
[21] V. I. Istratescu. Introduction to Linear Operator Theory. CRC Press, 2020.
[22] Y. Khoo, Lu, I.d Ying. Solving for high-dimensional committor functions using artificial neural networks. Research in the Mathematical Sciences, 6(1):1-13, 2019.
[23] Y. Khoo, Lu, and Ying. Solving parametric PDE problems with artificial neural networks. European Journal of Applied Mathematics, 32(3):421 4355, 2021.
[24] M. Klusowski and A. Barron. Approximation by combinations of ReLU and squared ReLU ridge functions with ll and e controls. IEEE Transactions on Information Theory, 64(12):7649-7656 2018.
[25] I. E. Lagaris, A. Likas, and D. I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 95):9877-1000 1998.
[26] Z. Li, N. Kovachki, K. Arizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
[27] J. Lu and Y. Lu. A priori generalization error analysis of two-layer neural networks for solving high dimensional Schrödinger eigenvalue problems. arXiv preprint 1rXiv:2105.01222 2021.
[28] Y. Lu, J. Lu, and M. Wang. A priori generalization analysis of the deep Ritz method for solving high dimensional elliptic partial differential equations. In Conference on Learning Theory, pages 3196-3241. PMLR, 2021.
[29] Luo and Yang. Two-layer neural networks for partial differential equations: Optimization and generalization theory. arXiv preprint 20X1:2006.11733, 2020.

---

[30] T. Marwah, Z. C. Lipton, and A. Risteski. Parametric complexity bounds for approximating PDEs with neural networks. In Advances in Neural Information Processing Systems, volume 34, 2021.
[31] S. Mishra and R. Molinaro. Estimates on the generalization error of physics informed neural networks (PINNs) for approximating PDEs. arXiv preprint arXiv:2006.16144 2020.
[32] Noé, S. Olsson, J. Köhler, and H. Wu. Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning. Science, 365(6457), 2019.
[33] M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informen neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[34] M. Raissi, A. Yazdani, and G. E. Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, (67( (1:644)):1020) 10:66-6- 2020.
[35] L. Ruthotto, S. J. Osher, W. Li, L. Nurbekyan. and S. W. Fung. A machine learning framework for solving high digediimensional mean field game and mean field control problems. Proceedings of the National Academy of Sciences, 1 117(17):9183-99) 9193, 2020.
[36] A. W. Senior, R. Evans, J. Jumper, J. Kirkpatrick, L. Sifre, T. Green, C. Qin, A. Zidek, A. W. Nelson, A. Bridgland, H. Penedones, S. Petersen, K. Simonyan, S. Crossan, P. Kohli, D. T. Jones, D. Silver, K. Kavukcuoglu, and D. Hassabis. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792):706-711 2020.
[37] Y. Shin, J. Darbon, and G. E. Karniadakis. On the convergence and generalization of physics informed neural networks. arXiv preprint arXiv:2004.0/8066 2020.
[38] Y. Shin, Z. Zhang, and G. E. Karniadakis. Error estimates of residual minimization using neural networks for linear PDEs. arXiv preprint arXiv:2010.08019, 2020.
[39] W. Siegel and J. Xu. Approximation rates for neural networks with general activation functions. Neural Networks, 128:313-321, 2020.
[40] J. W. Siegel and J. Xu. High-order approximation rates for shallow neural networks with cosine and ReLU* activation functions. arXiv preprint arXi::2012.07200 2020.
[41] J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339-1364 2018.
[42] V. Sitzmann. J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems, volume 33, pages 7462-7473, 2020.
[43] Y. Yang, J. Sun, H. Li, and Z. Xu. Deep ADMM-net for compressive sensing MRI. In Advances in Neural Information Processing Systems, volume 29, 2016.
[44] D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103-114, 2017.
[45] D. Yarotsky. Optimal approximation of continuous functions by very deep ReLU networks, In Conference on Learning Theory, pages 639-649. PMLR, 2018.
[46] Y. Zang, G. Bao, X. Ye, and H. Zhou. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics, 411: 2020.
[47] L. Zhang, J. Han, H. Wang, R. Car, and W. E. Deep potential molecular dynamics: scalable model with the accuracy of quantum mechanics. Physical Review Letters, 123010) 2018.
12


Reviews:
[{"review_id": "mH8KnyI-ZMa", "reviewer": "Reviewer_b2vX", "summary": "This paper studies the neural network approximation of high-dimensional elliptic PDEs. Specifically, it shows that when the source term and the coefficient functions lie in a Barron space, the solution can be approximated by two-layer neural networks without the curse of dimensionality.  The Barron norm characterizes the approximability of functions by two-layer neural networks.  The primary technique is to represent the PDE solution as the fixed point of the preconditioned steepest descent iteration.  Then,  it shows that the iteration satisfies two properties: (1) The iteration converges to the fixed point exponentially fast; (2) Each iteration only amplifies the Barron norm by a small fact.  To guarantee the first property, the PDE must be strongly elliptic. To guarantee the second property, the Barron space must be closed under multiplication, which excludes the most commonly-used activation functions. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Quality and clarity \nThis paper is well-written, and I can follow the authors' ideas very smoothly.  I went through the proof and they seem correct to me, though there are many tiny (no essential) errors.\n\n## Originality\nThe result of this paper is obtained from a novel combination of the Barron space theory of two-layer neural networks and the property of elliptic PDEs, in particular representing the PDE solution as the fixed point of the preconditioned steepest iteration.  The authors also develop some interesting extensions of Barron space to bounded smooth activation functions, which may be of independent interest. \nGiven the previous work, the main result (Theorem 2.10) is not surprising to me.  However,  I believe that the rigorous proof is not straightforward, which can also be seen from this paper, which extensively exploits the structures of the two-layer neural nets and the strongly elliptic PDE.  Therefore, in my opinion, this work makes a decent contribution to understanding high-dimensional PDEs and the neural net approximation.  \n\n## Limitations and questions\n- Assumption 2.7 excludes all the commonly-used activation functions.  I understand that this assumption is needed to establish the closeness of Barron space under multiplication. But, can authors comment more on this point? For instance, is it essential or just a technical issue?\n\n- The main result 2.10 seems to suggest that cosine activation gives rise to a much better rate. Does the cosine activation function work better in practice under the setting of the theorem? \n\n\n# Minor issues\n- In Theorem 2.10, the upper bounds should depend on $m(\\Omega)$  instead of $\\sqrt{m(\\Omega)}$.\n- In line 232, the upper bound of $k$ at least has a factor $1/\\varepsilon^2$, which comes from the Monte-Carlo discretization. \n- In Proposition A.1 and the proof, $R_f$ is not defined. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "XiPBxIoAFRt", "reviewer": "Reviewer_M9Yp", "summary": "The objective of this paper is to prove that the solution $u$ of a certain class of PDEs, of the form \n\\begin{equation}\\label{eq:PDE}\n- \\nabla\\cdot (A \\cdot \\nabla u) + cu = f \\mbox{ on } \\mathbb{R}^d  \n\\end{equation}\nfor some matrices $A$, coefficient $c$ and right-hand side $f$ can be well-approximated by functions belonging to some Barron spaces, provided that $A$, $c$ and $f$ belong to some Barron spaces as well. The precise approximation results are stated \nin Theorems~2.9 and~2.10 of the manuscript. \n\nSuch kinds of approximation result are very interesting since they can give some insight on how well the solutions of some high-dimensional PDEs can be approximated by neural networks-based numerical methods. Indeed, \nit is proved in earlier works that elements of these Barron spaces can be approximated in the $H^1$ norm by two-layer neural networks.  \n\nThe article is organized as follows: notation and assumptions on the data of the problem are introduced in Section~2.1. The definition of Barron spaces is recalled in Section~2.2, together with approximation results in the $H^1$ norm of elements of Barron spaces \nby two-layer neural networks (Theorem~2.5). The proofs of these results is postponed in Appendix~B. The main results of the paper concerning the approximation of the solution of PDE (\\ref{eq:PDE}) by elements of a Barron space are stated in Theorem~2.9 and Theorem~2.10. \nThe main idea of the proofs of these results is sketched in Section~3, together with the statement of useful auxiliary lemmas. The detailed proofs of these results are given in the Appendix. ", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The article is very well written and the presented results are very interesting with regards to the theoretical questions that neural-networks based numerical methods for the resolution of high-dimensional PDEs are rising. The proofs of the results all seem \ncorrect to me although I did not check all the details. Hence, I am in favor of the publication of the present manuscript in NeurIPS, up to a few minor modifications highlighted below. \n\n\\begin{itemize}\n \\item p.4: Def.~2.2: precise the definition of $\\overline{B}_R^d$ l. 153\n \\item Theorem~2.5, p. 5, l. 184: In this theorem $g$ is assumed to belong to the space $\\mathcal B_R^2(\\Omega)$. However, in Proposition~2.4, it is stated that $\\mathcal B_R^\\infty(\\Omega) = \\mathcal B_R^p(\\Omega) = \\mathcal B_R^1(\\Omega)$ for $1\\leq p \\leq \\infty$. \n The use of the exponent $2$ in Theorem~2.5 is then confusing. I suspect a typo and that the author(s) actually meant to assume that $g$ belongs to $\\mathcal B_R^1(\\Omega)$. Correct? Similar remark/question at l.205 of the same page (Assumptions~2.8) where \n $l_{d,2}$ is defined as $\\|\\sigma''\\|_{\\mathcal B^2_{R_{d,2}}}$. Do the authors mean $l_{d,2}:= \\|\\sigma''\\|_{\\mathcal B^1_{R_{d,2}}}$?  Please check in the manuscript if other similar typos are present. \n \\item p.5, l. 200: Do the authors simply mean by ${\\rm im}\\sigma \\neq \\{0 \\}$ the fact that $\\sigma$ has to be non identically equal to $0$? I am unsure of the meaning of the ${\\rm im}$ notation there. \n \\item Theorem~2.9, l.214, p. 6: Could the authors comment on the fact that the dimension $d$ has to be assumed greater than $3$ for the result to hold? Could the authors add a remark on the reason why wuch a result could not hold in lower dimension? \n\\end{itemize}\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact identified", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Jpxq4kULZUH", "reviewer": "Reviewer_oNuF", "summary": "The paper derives complexity estimates of the solutions of second-order linear elliptic PDEs in the Barron space. It proves the solution of the PDE is ε-close with respect to the H1 norm to a Barron function and the complexity depends polynomially on the dimension d. The theorem implies physics-informed neural networks can efficiently approximate the solution of second-order linear elliptic PDEs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Compared to previous work, the paper focuses on deriving complexity estimates of the solution in the integral-representation-based Barron space. \n\nPros:\n(1) the paper uses the Barron spaces techniques to improve the previous theorem.\n(2) the writing is clear.\n(3) clean related works.\n\nCons:\n(1) the improvement looks subtle compared to previous work, especially [30], which also proves complexity estimates for elliptic equations. (2) The relief of assumption from 2 layer activations to 1 layer seems not very significant.\n\nI have to admit that I am not very confident to assess the theoretical contribution and proof techniques. But overall, I like this work. I would vote for the border of acceptance. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The second-order linear elliptic equation are in general easy for conventional solvers. The higher dimension cases are not very common to my opinion. It will be very interesting if the work can be extended to nonlinear PDE so that it can be applied to equations such as Navier-Stokes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "3wAi-Wh-9gW", "reviewer": "Reviewer_1yLq", "summary": "This paper studies the representation of solutions of second order elliptic PDEs using two layer neural networks,\nand gives sufficient conditions so that this can be done without curse of dimension (the number of units grows at \nmost polynomially in the dimension).  Precise theorems to this effect are stated and proven. \nThe mathematical framework includes the definition of a Barron space of functions (essentially, finite norm of the\nfinal layer weights), many structural results about this function space, and an iterative solution scheme with its analysis.\n", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In recent years there has been much interest in developing numerical methods which take advantage of machine\nlearning concepts and technology, including the use of feedforward neural networks (FFNs) to represent solutions of PDEs.\nThere is a fair amount of numerical evidence that FFNs mitigate the curse of dimension for high dimensional PDEs of\npractical interest, such as the HJB equation and equations of mathematical finance.\nThe present work studies the representation of solutions of second order elliptic PDEs using two layer neural networks,\nand gives sufficient conditions so that this can be done without curse of dimension, here meaning the number of units grows at \nmost polynomially in the dimension.  \nThe arguments do not use special properties of the PDEs and thus are more general than most previous work.\nTheir main conceptual tool is the definition of a Barron space of functions -- essentially, functions with a two layer FNN representation of arbitrary width such that the final layer weights have finite norm.\nThis definition is inspired by 1993 work of Barron (ref. [2]) as developed in ref. [8].\nThe authors develop the idea systematically in order to implement an iterative solution scheme and its analysis,\nthus proving their theorems 2.9 and 2.10 which are precise forms of the claims.\n\nThe authors adequately cite the main works in this area that I know of.\nThe previous work closest to theirs is Marwah et al ref. [30], which uses a very similar iterative scheme to get similar results. \nHowever ref. [30] does not define and systematize the function spaces involved as clearly.\nI found the arguments in the present paper somewhat simpler and clearer, also they have some advantages (listed in \"related works\").   Also ref. [30] is not yet published and the preprint only appeared in March 2021, so the citation to it here seems adequate.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No apparent direct societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agree that it is a very interesting paper in its field. I share the same opinion. 

The main contribution to this paper is to show that solutions of elliptic PDE of the form
\begin{equation}
\- \nabla \cdot (A \nabla u) + cu = f
\end{equation}
can be approximated by a two-layer neural network in some Sobolev norm. This kind of results is really important due to the increase popularity of PDE solvers based on deep neural networks. 

The main assumption considered by the authors is that the coefficients $A,c$ and $f$ all belong to some Baron spaces which have to be closed by multiplication. To obtain such a results, the authors builds upon results on Baron spaces recently obtained in the literature and on the representation of solutions of elliptic PDE as fixed-points of a steepest descent scheme. 

While, this result and the strategy of the proof may seem not really surprising, the paper is very well-written and succeed in gathering all the ingredients of the proofs smoothly. 

That being said, some points have been raised by the reviewers. I encourage the authors to address these in the final version of their paper.
Decision: Accept (Spotlight)