On the Representation of Solutions to Elliptic PDEs in Barron Spaces

1 Introduction . Inspired by the tremendous success of deep learning in diverse machine learning tasks including image classification , natural language processing , and artificial intelligence , there has been growing interest in exploring scientific and engineering applications of deep learning [ 36 , 32 , 34 , 26 , 47 ] . As partial differential equations ( PDEs ) play a fundamental role in almost all branches of sciences and engineering , numerical solutions to PDE problems based on neural networks have become an important research direction in scientific machine learning [ 25 , 6 , 23 , 17 , 7 , 10 , 22 ] . Among the various directions , numerical solutions to high-dimensional PDEs – the unknown function depending on many variables – are perhaps the most exciting possibility , as solving such PDEs has been a long- 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . standing challenge and breakthrough would lead to tremendous progress in fields such as many-body physics [ 4 , 11 , 18 ] , multiple agent control [ 35 , 17 ] , just to name a few . Numerical solutions to low-dimensional PDEs , such as Navier-Stokes equation in fluid dynamics , has become a standard practice after decades of work . However , the computational cost of the conventional numerical methods for PDEs grows exponentially with the dimension , as a manifestation of the curse of dimensionality ( CoD ) . Given a target accuracy , conventional methods , such as finite element or finite difference , would need a mesh size ofO ( ) , and thus degree of freedom on the order of O ( −d ) , where d is the dimension of the problem . Such complexity severely limits the numerical solutions to PDEs in high dimension , such as the many-body Schrödinger equations from quantum mechanics and the high-dimensional Hamilton-Jacobi-Bellman equations from control theory . Neural networks , in particular deep neural networks , provide a promising way to overcome the CoD in representing functions in high dimension . It is thus a natural idea to parametrize the solution ansatz to a PDE as neural networks and to employ variational search for the optimal parameters . Various neural network methods [ 25 , 6 , 7 , 10 , 33 , 17 , 41 , 46 , 16 , 5 ] for PDEs have been proposed recently and some of them have demonstrated great empirical success in solving PDEs of hundreds and thousands of dimensions [ 7 , 10 , 17 ] , much beyond the capability of conventional approaches . Question remains though on theoretical analysis of such neural-network based methods for solving high-dimensional PDEs . While there have been some recent progress on approaches including physics-informed neural networks [ 37 , 31 , 38 ] and the deep Ritz method [ 28 , 27 ] , many questions still remain open . Among them , a fundamental question is Whether the solution of a high-dimensional PDE can be efficiently approximated by a neural network , and if so , how to quantify the complexity of the neural network representation with respect to the increasing dimension ? Our contributions The focus of the current study takes a functional-analytic approach to this question . Namely , we identify a function class suitable for neural network approximations and prove that the solutions to a class of PDEs can be well approximated by functions in this class . More specifically , the PDE we consider is a family of second-order elliptic PDEs of the form Lu = −∇ · ( A∇u ) + cu = f on Rd . ( 1.1 ) We choose to work with the Barron class of functions defined in [ 8 ] ( see also [ 1 ] ) , which is a class of functions admitting the integral of certain parametric ridge function against a probability measure on the parameters ; see Definition 2.2 for a precise description . This Barron space is inspired by the pioneering work by Barron [ 2 ] , where he proved that a class of functions whose Fourier transform has the first order moment can be approximated by two-layer networks without CoD . The main result of our work , stated informally , is the following ; a more precise statement can be found in Section 2.3 . Main Theorem ( informal version ) If the coefficients A , c and the source term f of the secondorder elliptic PDE ( 1.1 ) are all Barron functions , then the solution u∗ can be approximated by another Barron function u such that ‖u − u∗‖H1 ≤ , where the Barron norm of u is upper bounded by O ( ( d/ ) C log ( 1/ ) ) . Moreover , if the Barron space is defined by the cosine activation function , then the upper bound on the Barron norm can be improved to O ( dC log 1/ ) . We note that while the better rate is only obtained for the cosine activation function , such periodic activation function has indeed been found effective in certain PDE related tasks , see e.g. , [ 42 ] . Since the Barron functions can be approximated on a finite domain Ω w.r.t . H1 norm by twolayer neural networks with a rate O ( 1/ √ k ) where k is the network width ( see Theorem 2.5 ) , the theorem above directly implies that there exists a two-layer network uk with the number of widths k = O ( ( d/ ) C log ( 1/ ) ) , or k = O ( dC log 1/ ) if the activation function is cosine , such that ‖uk−u∗‖H1 ( Ω ) ≤ . Therefore in our setting the solution can be approximated by a two-layer neural network without CoD , namely the complexity depends at most polynomially on the dimension d for fixed . Alternatively , we can rewrite the rates as O ( ( 1/ ) C ( log d+log 1/ ) ) and O ( ( 1/ ) C log d ) to contrast with that of conventional grid-based numerical methods for PDEs , which scales as O ( ( 1/ ) d ) . We observe that the dependence on d is replaced with log d in the complexity bound for neural network approximations . We emphasize that such approximation result does not follow directly from the universal approximation property of neural networks for Barron functions since it is not a priori known that the solution to the PDE is a Barron function . In fact , directly imposing regularity or complexity assumption on the solution itself is unreasonable since the solution is unknown and its fine properties are generally inaccessible . Our main contribution is to establish the fact that the solution can be indeed approximated by a Barron function , under the assumption that coefficients and the right hand term of the PDE are Barron . From a mathematical point of view , our main theorem is in the same spirit as regularity estimates of PDEs , which are of crucial importance in the study of PDEs . While such regularity estimates are well developed in low dimension , the extension to results in high dimension is highly non-trivial and is the main focus of our work . Related works Several theoretical work have been devoted to the above representation question . It has been established in [ 15 , 20 , 14 ] that deep neural networks can approximate solutions to certain class of parabolic equations and Poisson equation without CoD . The major limitation of those work lies in that the PDEs considered in those work must admit certain stochastic representation such as the Feymann-Kac formula and it seems difficult to generalize the proof techniques to broader classes of PDEs with no probabilistic interpretation . The work [ 28 , 27 ] analyzed a priori generalization error of two-layer networks for solving elliptic PDEs and the Schrödinger eigenvalue problem on a bounded domain with Neumann boundary condition by assuming that the exact solutions lie in certain spectral Barron space , where the later was rigorously justified with a new regularity theory of the PDE solutions in the spectral Barron space . Similar generalization analysis was carried out in [ 29 ] for second-order PDEs and in [ 19 ] for general even-order elliptic PDEs , but without justifying the Barron assumption on the solution . Compared to those work , our work focuses on deriving complexity estimates of the solution in the integral-representation-based Barron space , which is more flexible and arguably more suitable for high-dimensional settings , see e.g. , discussion in [ 8 ] . The work [ 9 ] established such estimates in the Barron space for certain specific PDEs that essentially admit explicit solution , whereas we aim to prove such estimates for general elliptic PDEs for which the analytical ansatz is not available . The work [ 30 ] is closest to ours where the authors proved that the solution of the same type of elliptic PDE with a Dirichlet boundary condition can be approximated by a ( deep ) neural networks with at most O ( poly ( d ) N ) parameters if the coefficients of the PDE are approximable by neural networks with at most N parameters . While our overall approach based on iterative scheme borrows idea from [ 30 ] , our result differs and improves theirs in many aspects : ( 1 ) Our result shows that the solution can be well approximated without CoD by a two-layer neural network with a single activation whereas the result in [ 30 ] requires a deep network which uses a mixure of at least two activation functions ; ( 2 ) Our PDE is set up on the whole space rather than a compact domain , so our setting covers some important PDEs in physics , such as the stationary Schrödinger equation ; ( 3 ) The result in [ 30 ] relies on another key assumption that the source term lies within the span of finitely many eigenfunctions of the elliptic operator whereas our result completely removes such assumption . This is achieved by utilizing a novel preconditioning technique to uniformly control the condition number of the iterative scheme that underpins the proof of our main theorem . Organization The rest of this paper will be organized as follows . In Section 2.1 we set up the PDE problem on the whole space and in Section 2.2 we introduce the definition of Barron functions and discuss their H1-approximation by two-layer networks ( see Theorem 2.5 ) . Our main theorems are stated in Section 2.3 . We present the sketch proofs of the main theorems in Section 3 and defer the complete proof to Appendix . The paper is concluded with discussions on some future directions . 2 Problem setup and main results . 2.1 Problem description . Notations Throughout this paper , we use ‖v‖ to denote the Euclidean norm of a vector v ∈ Rd . For a matrix A ∈ Rd×d , we denote its operator norm by ‖A‖ = supv∈Rd\ { 0 } ‖Av‖ ‖v‖ . For R > 0 , we denote by B d R the closed ball in Rd centered at 0 with radius R , i.e. , B d R = { x ∈ Rd : ‖x‖ ≤ R } . Recall that we consider the d-dimensional second-order elliptic PDE ( 1.1 ) . To guarantee the existence and uniqueness of the weak solution in H1 ( Rd ) , we make the following minimum assumptions on coefficientsA , c and right-hand side f ; this assumption will be strengthened in our main representation theorem . Assumption 2.1 . A ( x ) = ( Aij ( x ) ) 1≤i , j≤d is symmetric with ‖A ( x ) ‖ ≤ amax < ∞ and uniformly elliptic , that is for some amin > 0 , it satisfies ξ > A ( x ) ξ ≥ amin ‖ξ‖2 , ∀ x , ξ ∈ Rd . We also assume that 0 < cmin ≤ c ( x ) ≤ cmax < ∞ and f ∈ L2 ( Rd ) . Under Assumption 2.1 , a standard argument using the Lax-Milgram theorem implies that there exists a unique weak solution u∗ ∈ H1 ( Rd ) , such that Lu∗ = f in H−1 ( Rd ) which is the dual space of H1 ( Rd ) , i.e. , ∫ Rd A∇u∗ · ∇vdx+ ∫ Rd cu∗vdx = ∫ Rd fvdx , ∀v ∈ H1 ( Rd ) . Our ultimate goal is to show that the solution can be approximated by a two-layer neural network on any bounded subset of Rd with respect to the H1 norm with a rate scaling at most polynomially in the dimension . Notice that in general one can not hope to obtain an approximation result on the whole space Rd because the asymptotic behavior of a neural network function ( determined by the activation ) at infinity may mismatch that of the target function u∗ . On the other hand , it is well-known that the convergence rate of neural networks for approximating functions in standard Sobolev or Hölder spaces still suffers from the CoD [ 44 , 45 ] . Therefore to obtain a rate without CoD for the neural networks approximation to the solution u∗ , we need to argue that u∗ lies in a suitable smaller function space which has low complexity compared to Sobolev or Hölder spaces . We will work with the Barron space and show that u∗ is arbitrarily close to a Barron function which can be approximated by a two-layer neural network without CoD .

Summary: This paper studies the neural network approximation of high-dimensional elliptic PDEs. Specifically, it shows that when the source term and the coefficient functions lie in a Barron space, the solution can be approximated by two-layer neural networks without the curse of dimensionality.  The Barron norm characterizes the approximability of functions by two-layer neural networks.  The primary technique is to represent the PDE solution as the fixed point of the preconditioned steepest descent iteration.  Then,  it shows that the iteration satisfies two properties: (1) The iteration converges to the fixed point exponentially fast; (2) Each iteration only amplifies the Barron norm by a small fact.  To guarantee the first property, the PDE must be strongly elliptic. To guarantee the second property, the Barron space must be closed under multiplication, which excludes the most commonly-used activation functions.