On the Representation of Solutions to Elliptic PDEs in Barron Spaces

1 Introduction . Inspired by the tremendous success of deep learning in diverse machine learning tasks including image classification , natural language processing , and artificial intelligence , there has been growing interest in exploring scientific and engineering applications of deep learning [ 36 , 32 , 34 , 26 , 47 ] . As partial differential equations ( PDEs ) play a fundamental role in almost all branches of sciences and engineering , numerical solutions to PDE problems based on neural networks have become an important research direction in scientific machine learning [ 25 , 6 , 23 , 17 , 7 , 10 , 22 ] . Among the various directions , numerical solutions to high-dimensional PDEs – the unknown function depending on many variables – are perhaps the most exciting possibility , as solving such PDEs has been a long- 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . standing challenge and breakthrough would lead to tremendous progress in fields such as many-body physics [ 4 , 11 , 18 ] , multiple agent control [ 35 , 17 ] , just to name a few . Numerical solutions to low-dimensional PDEs , such as Navier-Stokes equation in fluid dynamics , has become a standard practice after decades of work . However , the computational cost of the conventional numerical methods for PDEs grows exponentially with the dimension , as a manifestation of the curse of dimensionality ( CoD ) . Given a target accuracy , conventional methods , such as finite element or finite difference , would need a mesh size ofO ( ) , and thus degree of freedom on the order of O ( −d ) , where d is the dimension of the problem . Such complexity severely limits the numerical solutions to PDEs in high dimension , such as the many-body Schrödinger equations from quantum mechanics and the high-dimensional Hamilton-Jacobi-Bellman equations from control theory . Neural networks , in particular deep neural networks , provide a promising way to overcome the CoD in representing functions in high dimension . It is thus a natural idea to parametrize the solution ansatz to a PDE as neural networks and to employ variational search for the optimal parameters . Various neural network methods [ 25 , 6 , 7 , 10 , 33 , 17 , 41 , 46 , 16 , 5 ] for PDEs have been proposed recently and some of them have demonstrated great empirical success in solving PDEs of hundreds and thousands of dimensions [ 7 , 10 , 17 ] , much beyond the capability of conventional approaches . Question remains though on theoretical analysis of such neural-network based methods for solving high-dimensional PDEs . While there have been some recent progress on approaches including physics-informed neural networks [ 37 , 31 , 38 ] and the deep Ritz method [ 28 , 27 ] , many questions still remain open . Among them , a fundamental question is Whether the solution of a high-dimensional PDE can be efficiently approximated by a neural network , and if so , how to quantify the complexity of the neural network representation with respect to the increasing dimension ? Our contributions The focus of the current study takes a functional-analytic approach to this question . Namely , we identify a function class suitable for neural network approximations and prove that the solutions to a class of PDEs can be well approximated by functions in this class . More specifically , the PDE we consider is a family of second-order elliptic PDEs of the form Lu = −∇ · ( A∇u ) + cu = f on Rd . ( 1.1 ) We choose to work with the Barron class of functions defined in [ 8 ] ( see also [ 1 ] ) , which is a class of functions admitting the integral of certain parametric ridge function against a probability measure on the parameters ; see Definition 2.2 for a precise description . This Barron space is inspired by the pioneering work by Barron [ 2 ] , where he proved that a class of functions whose Fourier transform has the first order moment can be approximated by two-layer networks without CoD . The main result of our work , stated informally , is the following ; a more precise statement can be found in Section 2.3 . Main Theorem ( informal version ) If the coefficients A , c and the source term f of the secondorder elliptic PDE ( 1.1 ) are all Barron functions , then the solution u∗ can be approximated by another Barron function u such that ‖u − u∗‖H1 ≤ , where the Barron norm of u is upper bounded by O ( ( d/ ) C log ( 1/ ) ) . Moreover , if the Barron space is defined by the cosine activation function , then the upper bound on the Barron norm can be improved to O ( dC log 1/ ) . We note that while the better rate is only obtained for the cosine activation function , such periodic activation function has indeed been found effective in certain PDE related tasks , see e.g. , [ 42 ] . Since the Barron functions can be approximated on a finite domain Ω w.r.t . H1 norm by twolayer neural networks with a rate O ( 1/ √ k ) where k is the network width ( see Theorem 2.5 ) , the theorem above directly implies that there exists a two-layer network uk with the number of widths k = O ( ( d/ ) C log ( 1/ ) ) , or k = O ( dC log 1/ ) if the activation function is cosine , such that ‖uk−u∗‖H1 ( Ω ) ≤ . Therefore in our setting the solution can be approximated by a two-layer neural network without CoD , namely the complexity depends at most polynomially on the dimension d for fixed . Alternatively , we can rewrite the rates as O ( ( 1/ ) C ( log d+log 1/ ) ) and O ( ( 1/ ) C log d ) to contrast with that of conventional grid-based numerical methods for PDEs , which scales as O ( ( 1/ ) d ) . We observe that the dependence on d is replaced with log d in the complexity bound for neural network approximations . We emphasize that such approximation result does not follow directly from the universal approximation property of neural networks for Barron functions since it is not a priori known that the solution to the PDE is a Barron function . In fact , directly imposing regularity or complexity assumption on the solution itself is unreasonable since the solution is unknown and its fine properties are generally inaccessible . Our main contribution is to establish the fact that the solution can be indeed approximated by a Barron function , under the assumption that coefficients and the right hand term of the PDE are Barron . From a mathematical point of view , our main theorem is in the same spirit as regularity estimates of PDEs , which are of crucial importance in the study of PDEs . While such regularity estimates are well developed in low dimension , the extension to results in high dimension is highly non-trivial and is the main focus of our work . Related works Several theoretical work have been devoted to the above representation question . It has been established in [ 15 , 20 , 14 ] that deep neural networks can approximate solutions to certain class of parabolic equations and Poisson equation without CoD . The major limitation of those work lies in that the PDEs considered in those work must admit certain stochastic representation such as the Feymann-Kac formula and it seems difficult to generalize the proof techniques to broader classes of PDEs with no probabilistic interpretation . The work [ 28 , 27 ] analyzed a priori generalization error of two-layer networks for solving elliptic PDEs and the Schrödinger eigenvalue problem on a bounded domain with Neumann boundary condition by assuming that the exact solutions lie in certain spectral Barron space , where the later was rigorously justified with a new regularity theory of the PDE solutions in the spectral Barron space . Similar generalization analysis was carried out in [ 29 ] for second-order PDEs and in [ 19 ] for general even-order elliptic PDEs , but without justifying the Barron assumption on the solution . Compared to those work , our work focuses on deriving complexity estimates of the solution in the integral-representation-based Barron space , which is more flexible and arguably more suitable for high-dimensional settings , see e.g. , discussion in [ 8 ] . The work [ 9 ] established such estimates in the Barron space for certain specific PDEs that essentially admit explicit solution , whereas we aim to prove such estimates for general elliptic PDEs for which the analytical ansatz is not available . The work [ 30 ] is closest to ours where the authors proved that the solution of the same type of elliptic PDE with a Dirichlet boundary condition can be approximated by a ( deep ) neural networks with at most O ( poly ( d ) N ) parameters if the coefficients of the PDE are approximable by neural networks with at most N parameters . While our overall approach based on iterative scheme borrows idea from [ 30 ] , our result differs and improves theirs in many aspects : ( 1 ) Our result shows that the solution can be well approximated without CoD by a two-layer neural network with a single activation whereas the result in [ 30 ] requires a deep network which uses a mixure of at least two activation functions ; ( 2 ) Our PDE is set up on the whole space rather than a compact domain , so our setting covers some important PDEs in physics , such as the stationary Schrödinger equation ; ( 3 ) The result in [ 30 ] relies on another key assumption that the source term lies within the span of finitely many eigenfunctions of the elliptic operator whereas our result completely removes such assumption . This is achieved by utilizing a novel preconditioning technique to uniformly control the condition number of the iterative scheme that underpins the proof of our main theorem . Organization The rest of this paper will be organized as follows . In Section 2.1 we set up the PDE problem on the whole space and in Section 2.2 we introduce the definition of Barron functions and discuss their H1-approximation by two-layer networks ( see Theorem 2.5 ) . Our main theorems are stated in Section 2.3 . We present the sketch proofs of the main theorems in Section 3 and defer the complete proof to Appendix . The paper is concluded with discussions on some future directions . 2 Problem setup and main results . 2.1 Problem description . Notations Throughout this paper , we use ‖v‖ to denote the Euclidean norm of a vector v ∈ Rd . For a matrix A ∈ Rd×d , we denote its operator norm by ‖A‖ = supv∈Rd\ { 0 } ‖Av‖ ‖v‖ . For R > 0 , we denote by B d R the closed ball in Rd centered at 0 with radius R , i.e. , B d R = { x ∈ Rd : ‖x‖ ≤ R } . Recall that we consider the d-dimensional second-order elliptic PDE ( 1.1 ) . To guarantee the existence and uniqueness of the weak solution in H1 ( Rd ) , we make the following minimum assumptions on coefficientsA , c and right-hand side f ; this assumption will be strengthened in our main representation theorem . Assumption 2.1 . A ( x ) = ( Aij ( x ) ) 1≤i , j≤d is symmetric with ‖A ( x ) ‖ ≤ amax < ∞ and uniformly elliptic , that is for some amin > 0 , it satisfies ξ > A ( x ) ξ ≥ amin ‖ξ‖2 , ∀ x , ξ ∈ Rd . We also assume that 0 < cmin ≤ c ( x ) ≤ cmax < ∞ and f ∈ L2 ( Rd ) . Under Assumption 2.1 , a standard argument using the Lax-Milgram theorem implies that there exists a unique weak solution u∗ ∈ H1 ( Rd ) , such that Lu∗ = f in H−1 ( Rd ) which is the dual space of H1 ( Rd ) , i.e. , ∫ Rd A∇u∗ · ∇vdx+ ∫ Rd cu∗vdx = ∫ Rd fvdx , ∀v ∈ H1 ( Rd ) . Our ultimate goal is to show that the solution can be approximated by a two-layer neural network on any bounded subset of Rd with respect to the H1 norm with a rate scaling at most polynomially in the dimension . Notice that in general one can not hope to obtain an approximation result on the whole space Rd because the asymptotic behavior of a neural network function ( determined by the activation ) at infinity may mismatch that of the target function u∗ . On the other hand , it is well-known that the convergence rate of neural networks for approximating functions in standard Sobolev or Hölder spaces still suffers from the CoD [ 44 , 45 ] . Therefore to obtain a rate without CoD for the neural networks approximation to the solution u∗ , we need to argue that u∗ lies in a suitable smaller function space which has low complexity compared to Sobolev or Hölder spaces . We will work with the Barron space and show that u∗ is arbitrarily close to a Barron function which can be approximated by a two-layer neural network without CoD .

Summary: The objective of this paper is to prove that the solution $u$ of a certain class of PDEs, of the form  \begin{equation}\label{eq:PDE} - \nabla\cdot (A \cdot \nabla u) + cu = f \mbox{ on } \mathbb{R}^d   \end{equation} for some matrices $A$, coefficient $c$ and right-hand side $f$ can be well-approximated by functions belonging to some Barron spaces, provided that $A$, $c$ and $f$ belong to some Barron spaces as well. The precise approximation results are stated  in Theorems~2.9 and~2.10 of the manuscript.   Such kinds of approximation result are very interesting since they can give some insight on how well the solutions of some high-dimensional PDEs can be approximated by neural networks-based numerical methods. Indeed,  it is proved in earlier works that elements of these Barron spaces can be approximated in the $H^1$ norm by two-layer neural networks.    The article is organized as follows: notation and assumptions on the data of the problem are introduced in Section~2.1. The definition of Barron spaces is recalled in Section~2.2, together with approximation results in the $H^1$ norm of elements of Barron spaces  by two-layer neural networks (Theorem~2.5). The proofs of these results is postponed in Appendix~B. The main results of the paper concerning the approximation of the solution of PDE (\ref{eq:PDE}) by elements of a Barron space are stated in Theorem~2.9 and Theorem~2.10.  The main idea of the proofs of these results is sketched in Section~3, together with the statement of useful auxiliary lemmas. The detailed proofs of these results are given in the Appendix.