Parametric Complexity Bounds for Approximating PDEs with Neural Networks

1 Introduction . A partial differential equation ( PDE ) relates a multivariate function defined over some domain to its partial derivatives . Typically , one ’ s goal is to solve for the ( unknown ) function , often subject to additional constraints , such as the function ’ s value on the boundary of the domain . PDEs are ubiquitous in both the natural and social sciences , where they model such diverse processes as heat diffusion [ 7 , 32 ] , fluid dynamics [ 2 , 39 ] , and financial markets [ 6 , 10 ] . Because most PDEs of interest lack closed-form solutions , computational approximation methods remain a vital and an active field of research [ 1 ] . For low-dimensional functions , dominant approaches include the finite differences and finite element methods [ 26 ] , which discretize the domain . After partitioning the domain into a mesh , these methods solve for the function value at the vertices of the mesh . However , these techniques scale exponentially with the input dimension , rendering them unsuitable for high-dimensional problems . Following breakthroughs in deep learning for approximating high-dimensional functions in such diverse domains as computer vision [ 21 , 33 ] and natural language processing [ 3 , 9 , 40 ] , a burgeoning line of research leverages neural networks to approximate solutions to PDEs . This line of work has produced promising empirical results for common PDEs such as the Hamilton-Jacobi-Bellman and Black-Scholes equations [ 17 , 16 , 37 ] . Because they do not explicitly discretize the domain , and given their empirical success on high-dimensional problems , these methods appear not to suffer the curse of dimensionality . However , these methods are not well understood theoretically , leaving open questions about when they are applicable , what their performance depends on , and just how many parameters are required to approximate the solution to a given PDE . Over the past three years , several theoretical works have investigated questions of representational power under various assumptions . Exploring a variety of settings , [ 22 ] , [ 16 ] , and [ 19 ] , proved that the number of parameters required to approximate a solution to a PDE exhibits a less than exponential 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . dependence on the input dimension for some special parabolic PDEs that admit straightforward analysis . [ 15 ] consider elliptic PDEs with Dirichlet boundary conditions . However , their rate depends on the volume of the domain , and thus can have an implicit exponential dependence on dimension ( e.g. , consider a hypercube with side length greater than one ) . In this paper , we focus on linear elliptic PDEs with Dirichlet boundary conditions , which are prevalent in science and engineering ( e.g. , the Laplace and Poisson equations ) . Notably , linear elliptic PDEs define the steady state of processes like heat diffusion and fluid dynamics . Our work asks : Question . How many parameters suffice to approximate solutions to a linear elliptic PDE up to a specified level of precision using a neural network ? We show that when the coefficients of the PDE are expressible as small neural networks ( note that PDE coefficients are functions ) , the number of parameters required to approximate the PDE ’ s solution is proportional to the number of parameters required to express the coefficients . Furthermore , we show that the number of parameters depends polynomially on the dimension and does not depend upon the volume of the domain . 2 Overview of Results . To begin , we formally define linear elliptic PDEs . Definition 1 ( Linear Elliptic PDE [ 11 ] ) . Linear elliptic PDEs with Dirichlet boundary condition can be expressed in the following form : { ( Lu ) ( x ) ≡ ( −div ( A∇u ) + cu ) ( x ) = f ( x ) , ∀x ∈ Ω , u ( x ) = 0 , ∀x ∈ ∂Ω , where Ω ⊂ Rd is a bounded open set with a boundary ∂Ω . Further , for all x ∈ Ω , A : Ω→ Rd×d is a matrix-valued function , s.t . A ( x ) 0 , and c : Ω→ R , s.t . c ( x ) > 0 . 1 We refer to A and c as the coefficients of the PDE . The divergence form in Definition 1 is one of two canonical ways to define a linear elliptic PDE [ 11 ] and is convenient for several technical reasons ( see Section 4 ) . The Dirichlet boundary condition states that the solution takes a constant value ( here 0 ) on the boundary ∂Ω . Our goal is to express the number of parameters required to approximate the solution of a PDE in terms of those required to approximate its coefficients A and c. Our key result shows : Theorem ( Informal ) . If the coefficients A , c and the function f are approximable by neural networks with at most N parameters , the solution u ? to the PDE in Definition 1 is approximable by a neural network with O ( dlog ( 1 ) N ) parameters . This result , formally expressed in Section 5 , may help to explain the practical efficacy of neural networks in approximating solutions to high-dimensional PDEs with boundary conditions [ 37 , 27 ] . To establish this result , we develop a constructive proof technique that simulates gradient descent ( in an appropriate Hilbert space ) through the very architecture of a neural network . Each iterate , given by a neural network , is subsumed into the ( slightly larger ) network representing the subsequent iterate . The key to our analysis is to bound both ( i ) the growth in network size across consecutive iterates ; and ( ii ) the total number of iterates required . Organization of the paper We introduce the required notation along with some mathematical preliminaries on PDEs in Section 4 . The problem setting and formal statement of the main result are provided in Section 5 . Finally , we provide the proof of the main result in Section 6 . 3 Prior Work . Among the first papers to leverage neural networks to approximate solutions to PDEs with boundary conditions are [ 23 ] , [ 24 ] , and [ 29 ] . However , these methods discretize the input space and thus are not 1Here , div denotes the divergence operator . Given a vector field F : Rd → Rd , div ( F ) = ∇·F = ∑d i=1 ∂Fi ∂xi suitable for high-dimensional input spaces . More recently , mesh-free neural network approaches have been proposed for high-dimensional PDEs [ 17 , 34 , 35 ] , achieving impressive empirical results in various applications . [ 37 ] design a loss function that penalizes failure to satisfy the PDE , training their network on minibatches sampled uniformly from the input domain . They also provide a universal approximation result , showing that for sufficiently regularized PDEs , there exists a multilayer network that approximates its solution . However , they do not comment on the complexity of the neural network or how it scales with the input dimension . [ 20 ] also prove universal approximation power , albeit with networks of size exponential in the input dimension . Recently , [ 16 , 19 ] provided a better-thanexponential dependence on the input dimension for some special parabolic PDEs , for which the simulating a PDE solver by a neural network is straightforward . Several recent works [ 5 , 22 , 28 , 27 ] show ( experimentally ) that a single neural network can solve for an entire family of PDEs . They approximate the map from a PDE ’ s parameters to its solution , potentially avoiding the trouble of retraining for every set of coefficients . Among these , only [ 22 ] provides theoretical grounding . However , they assume the existence of a finite low-dimensional space with basis functions that can approximate this parametric map—and it is unclear when this would obtain . Our work proves the existence of such maps , under the assumption that the family of PDEs has coefficients described by neural networks with a fixed architecture ( Section 7 ) . In the work most closely related to ours , [ 15 ] provides approximation rates polynomial in the input dimension d for the Poisson equation ( a special kind of linear elliptic PDE ) with Dirichlet boundary conditions . They introduce a walk-on-the-sphere algorithm , which simulates a stochastic differential equation that can be used to solve a Poisson equation with Dirichlet boundary conditions ( see , e.g . [ 31 ] ’ s Theorem 9.13 ) . The rates provided in [ 15 ] depend on the volume of the domain , and thus depend , implicitly , exponentially on the input dimension d. Our result considers the boundary condition for the PDE and is independent of the volume of the domain . Further , we note that our results are defined for a more general linear elliptic PDE , of which the Poisson equation is a special case . 4 Notation and Definitions . We now introduce several key concepts from PDEs and some notation . For any open set Ω ⊂ Rd , we denote its boundary by ∂Ω and denote its closure by Ω̄ : = Ω ∪ ∂Ω . By C0 ( Ω ) , we denote the space of real-valued continuous functions defined over the domain Ω . Furthermore , for k ∈ N , a function g belongs to Ck ( Ω ) if all partial derivatives ∂αg exist and are continuous for any multi-index α , such that |α| ≤ k. Finally , a function g ∈ C∞ ( Ω ) if g ∈ Ck ( Ω ) for all k ∈ N. Next , we define several relevant function spaces : Definition 2 . For any k ∈ N ∪ { ∞ } , Ck0 ( Ω ) : = { g : g ∈ Ck ( Ω ) , supp ( g ) ⊂ Ω } . Definition 3 . For a domain Ω , the function space L2 ( Ω ) consists of all functions g : Ω → R , s.t . ‖g‖L2 ( Ω ) < ∞ where ‖g‖L2 ( Ω ) = ( ∫ Ω |g ( x ) |2dx ) 1 2 . This function space is equipped with the inner product 〈g , h〉L2 ( Ω ) = ∫ Ω g ( x ) h ( x ) dx . Definition 4 . For a domain Ω and a function g : Ω → R , the function space L∞ ( Ω ) is defined analogously , where ‖g‖L∞ ( Ω ) = inf { c ≥ 0 : |g ( x ) | ≤ c for almost all x ∈ Ω } . Definition 5 . For a domain Ω and m ∈ N , we define the Hilbert space Hm ( Ω ) as Hm ( Ω ) : = { g : Ω→ R : ∂αg ∈ L2 ( Ω ) , ∀α s.t . |α| ≤ m } Furthermore , Hm ( Ω ) is equipped with the inner product , 〈g , h〉Hm ( Ω ) = ∑ |α|≤m ∫ Ω ( ∂αg ) ( ∂αh ) dx and the corresponding norm ‖g‖Hm ( Ω ) = ∑ |α|≤m ‖∂αg‖2L2 ( Ω ) 12 . Definition 6 . The closure of C∞0 ( Ω ) in Hm ( Ω ) is denoted by Hm0 ( Ω ) . Informally , Hm0 ( Ω ) is the set of functions belonging to H m ( Ω ) that can be approximated by a sequence of functions φn ∈ C∞0 ( Ω ) . This also implies that if a function g ∈ Hm0 ( Ω ) , then g ( x ) = 0 for all x ∈ ∂Ω . This space ( particularly with m = 1 ) is often useful when analyzing elliptic PDEs with Dirichlet boundary conditions . Definition 7 ( Weak Solution ) . Given the PDE in Definition 1 , if f ∈ L2 ( Ω ) , then a function u : Ω→ R solves the PDE in a weak sense if u ∈ H10 ( Ω ) and for all v ∈ H10 ( Ω ) , we have∫ Ω ( A∇u · ∇v + cuv ) dx = ∫ Ω fvdx ( 1 ) The left hand side of ( 1 ) is also equal to 〈Lu , v〉L2 ( Ω ) for all u , v ∈ H10 ( Ω ) ( see Lemma 6 ) , whereas , following the definition of the L2 ( Ω ) norm , the right side is simply 〈f , v〉L2 ( Ω ) . Having introduced these preliminaries , we now introduce some important facts about linear PDEs that feature prominently in our analysis . Proposition 1 . For the PDE in Definition 1 , if f ∈ L2 ( Ω ) the following hold : 1 . The solution to Equation ( 1 ) exists and is unique . 2 . The weak solution is also the unique solution of the following minimization problem : u ? = argmin v∈H10 ( Ω ) J ( v ) : = argmin v∈H10 ( Ω ) { 1 2 〈Lv , v〉L2 ( Ω ) − 〈f , v〉L2 ( Ω ) } . ( 2 ) This proposition is standard ( we include a proof in the Appendix , Section A.1 for completeness ) and states that there exists a unique solution to the PDE ( referred to as u ? ) , which is also the solution we get from the variational formulation in ( 2 ) . In this work , we introduce a sequence of functions that minimizes the loss in the variational formulation . Definition 8 ( Eigenvalues and Eigenfunctions , [ 11 ] ) . Given an operator L , the tuples ( λ , ϕ ) ∞i=1 , where λi ∈ R and ϕi ∈ H10 ( Ω ) are ( eigenvalue , eigenfunction ) pairs that satisfy Lϕ = λϕ , for all x ∈ Ω . Since ϕ ∈ H10 ( Ω ) , we know that ϕ|∂Ω = 0 . The eigenvalue can be written as λi = inf u∈Xi 〈Lu , u〉L2 ( Ω ) ‖u‖2L2 ( Ω ) , ( 3 ) where Xi : = span { ϕ1 , . . . , ϕi } ⊥ = { u ∈ H10 ( Ω ) : 〈u , ϕj〉L2 ( Ω ) = 0 ∀j ∈ { 1 , · · · , i } } and 0 < λ1 ≤ λ2 ≤ · · · . Furthermore , we define by Φk the span of the first k eigenfunctions of L , i.e. , Φk : = span { ϕ1 , · · · , ϕk } . We note that since the operator L is self-adjoint and elliptic ( in particular , L−1 is compact ) , the eigenvalues are real and countable . Moreover , the eigenfunctions form an orthonormal basis of H10 ( Ω ) ( see [ 11 ] , Section 6.5 ) .

Summary: The authors attempt to bound a neural network's complexity that approximates a solution to a self-adjoint elliptic PDE. The paper lacks mathematical rigor on the PDE theory side: the most important one is that their gradient descent step doesn't make sense from a functional analysis point-of-view (see main review). The authors need to compare the complexity of their neural network to the bound obtained by combining standard PDE regularity theory (see Evan's Section 6.3) with the complexity of neural networks for approximating smooth functions. Since the main theorem doesn't take into account the smoothness of the boundary of \Omega, I suspect that the latter gives much stronger bounds for many practical domains.