Neural Network Approximations of PDEs Beyond Linearity: Representational Perspective
Authors: Tanya Marwah
Zachary Chase Lipton
Jianfeng Lu
Andrej Risteski
Venue: Submitted to ICLR 2023 (2023)
Keywords: PDE
Partial Differential Equations
Deep Learning Theory
Universal Approximation

Abstract: A burgeoning line of research has developed deep neural networks capable of approximating the solutions to high dimensional PDEs, opening related lines of theoretical inquiry focused on explaining how it is that these models appear to evade the curse of dimensionality. However, most theoretical analyses thus far have been limited to simple linear PDEs. In this work, we take a step towards studying the representational power of neural networks for approximating solutions to nonlinear PDEs. We focus on a class of PDEs known as nonlinear elliptic variational PDEs, whose solutions minimize an Euler-Lagrange energy functional $\mathcal{E}(u) = \int_\Omega L(\nabla u) dx$. We show that if composing a function with Barron norm $b$ with $L$ produces a function of Barron norm at most $B_L b^p$, the solution to the PDE can be $\epsilon$-approximated in the $L^2$ sense by a function with Barron norm $O\left(\left(dB_L\right)^{p^{\log(1/\epsilon)}}\right)$. By a classical result due to \cite{barron1993universal}, this correspondingly bounds the size of a 2-layer neural network needed to approximate the solution. Treating $p, \epsilon, B_L$ as constants, this quantity is polynomial in dimension, thus showing neural networks can evade the curse of dimensionality. Our proof technique involves neurally simulating (preconditioned) gradient in an appropriate Hilbert space, which converges exponentially fast to the solution of the PDE, and such that we can bound the increase of the Barron norm at each iterate. Our results subsume and substantially generalize analogous prior results for linear elliptic PDEs. 

Under review as a conference paper at ICLR 2023
NEURAL NETWORK APPROXIMATIONS OF PDES BE- YOND LINEARITY: REPRESENTATIONAL P
Anonymous authors Paper under double-blind review
ABSTRACT
A burgeoning line of research has developed deep neural networks capable of approximating the solutions to high dimensional PDEs, opening related lines of theoretical inquiry focused on explaining how it is that these models appear to evade the curse of dimensionality. However, most theoretical analyses thus far have been limited to simple linear PDEs. In this work, we take a step towards studying the representational power of neural networks for approximating solu- tions to nonlinear PDEs. We focus on a class of PDEs known as nonlinear vari- ational elliptic PDEs, whose solutions minimize an Buler-Lagrange energy fune- tional {u) = Ja L(Vu)dz, We show that if composing a function with Barron norm b with produces a function of Barron norm most B,, the solution to the PDE can be e-approximated in the L sense by a function with Barron norm O ((dB))) ). By classical result due to Barron 1993), this cor- respondingly bounds the size of a 2-layer neural network needed approximate the solution. Treating p. e, By as constants, this quantity is polynomial in dimen- sion, thus showing that neural networks can evade the curse of dimensionality. Our proof technique involves "neurally simulating" (preconditioned) gradient in an appropriate Hilbert space, which converges exponentially fast to the solution of the PDE, and such that we can bound the increase of the Barron norm at each iterate. Our results subsume and substantially generalize analogous prior results for linear elliptic PDEs.
INTRODUCTION
Scientific applications have become of the frontiers for the application of deep learn- ing (Jumper et al.) 2021] Tuuyasuuvunakoll et al. 2021 (Snndrby et al. (0020). PDEs are one of the fundamental modeling techniques scientific domains, and designing neural network-aided solvers, particularly in high-dimensions, is of widespread usage in many domains (Hsieh et al. 2019 Brand- stetter al. 2022). One of the most common approaches for applying neural networks solve PDEs is to parameterize the solution as neural network and minimize a loss which characterizes the solution (Sirignano & Spiliopoulos 2018 & Yuj 2017). The hope doing so is have method which computationally avoids the "curse of cimennsinality"-.... that scales less than ex- ponentially with the ambient dimension.
To date, neither theoretical analysis nor empirical applications have yielded a precise characteriza- tion of the range of PDEs for which neural network-aided methods outperform classical methods. Active research on the empirical side (Han et al.] 2018 E et al.) 2017: Li et al. [220aa b) has ex- plored several families PDEs, c.g., networks have been demonstrated outperform classical griddbased methods. On the theory side, a recent line of works (Marwah et al. 2021 Chen et al. 2021 2022) has considered the following fundamental question:
and
where neural
For what families of PDEs can the solution be represented by a small neural network?
The motivation for this question is computational: since the computational complexity of fitting neural network (by minimizing some objective) will grow with its size. Specifically, these works focus understanding when the approximating neural network can be sub-exponential in size, thus

---

Under review as a conference paper at ICLR 2023
avoiding the curse of dimensionality. Unfortunately, the techniques introduced in this line of work have so far only been applicable to linear PDEs.
In this paper, take the first step beyond linear PDEs, with particular focus nonlinear vari- ational elliptic PDEs. These equations have the form div (V((Vu) = 0 and are instances of nonlinear Eider-Lagrange equations. Equivalently,
is the minimizer of the energy functional E(u) = In L(Vu)dr. This paradigm is very generic: its origins are in Lagrangian formulations of classical mechanics, and for different L, variety of variational problems can be modeled or learned (Schmidt & Lipson 2009 Cranmer et al.) 2020). These PDEs have a variety of applications in scien- tific domains, e.g.. -ewtonnnn))))) fluid dynamics (Koleva & Vulkov 2018), meteorology (Weller et al.
2016), and nonlinear diffusion equations (Burgers) 2013))
Our main result is to show that when the function has "low complexity", so does the solution. The notion of complexity we work with is the Barron norm of the function, similar to Chen et al. (2021): Lee et al. (2017). This frequently used notion of complexity, as function with small Barron norm can be represented by a small, two-layer neural network, due to a classical result (Barron 1993). Mathematically, our proof techniques are based on "neurally unfolding" an iterative preconditioned gradient descent in an appropriate function space: namely, we show that each of the iterates can be represented by neural network with Barron norm not much worse than the Barron norm of the previous iterate-alee with showing bound on the number of required steps.
Importantly, our results go beyond the typical non parametric bounds on the size of an approximator network that be easily shown by classical regularity results of the solution the nonlinear variational PDEs (De Giorgi 1957 Yarotsky 2017).
Nash 1957 1958) along with universal approximation results
2 OVERVIEW OF RESULTS
Let Ω C Idd be bounded open set with 0 E Ω and O denote the boundary of Ω. Furthermore, we assume that the domain such that the Poincare constant Cp is greater than (see Theorem 2 for the exact definition of the Poincare constant).
We first define the energy functional whose minimizers are represented by a nonlinear variational elliptic PDE-ie.. the Euler-Lagrange equation of the energy functional.
Definition 1 (Energy functional). For all u: Ω -> R such that ulan = 0, we consider an energy functional of the following form:
E(u) = I(Vu)dx, J2
(1)
where L: I.d
+ R is a smooth and uniformly convex functionn i.e., there exists constant 0 < <>< A such that for all I we have Ala < D2 D²L(x) < Ala. Further, without loss of generality we assume that X S 1/Cp-
Note that due to the convexity of the function L, the minimizer u² exists and is unique. The proof of existence and uniqueness is standard (e.g., Theorem 3.3 in Fernández-Reen Real & Ros- Cos-Oton| (2020).
Writing down the condition for stationarity, we can derive a (nonlinear) elliptic PDE for the mini- mizer of the energy functional in Definition!] D
Lemma 1. Let u² Ω - IK be the unique minimizer for the energy functional in Definition Then for all ip E Hg (Q) the minimizer u" satisfies the following condition.
DE(u](p) = [ VV(Vu)Vipdar =,
where dE dE(u)] (1) denotes the dirctional derivative of the energy functional calculated at u in the direction of 9. Thus, the minimizer u" of the energy functional satisfies the following PDE:
DE(u) == -ivvvVVL(Vui =0
Vx ExER St
and u(x) 0, Vx E N.. Here div denote the divergence operator.
'Since X is a lower bound the strong convexity constant. If we choose weaker lower bound, we always ensure < 1/Cp-
2

---

Under review as a conference paper at ICLR 2023
The proof for the Lemma can be found in Appendix Section A.1 Here, -div(VL(V.) is functional operator that acts on a function (in this case u).
Our goal is to determine if the solution the PDE in Equation can be expressed by neural network with a small number of parameters. In order do so, we utilize the concept of Barron norm, which measures the complexity of a function in terms of its Fourier representation. We show that if composing with the function L is such that it increases it has a bounded increase in the Barron norm of u, then the solution to the PDE in Equation 3 will have bounded Barron norm. The motivation for using this norm is a seminal paper (Barron 1993). which established that any function with Barron norm C' can be e-approximated in the L sense by a two-layer neural network with size O(C²/e), thus evading the curse of dimensionality if C' is substantially smaller than exponential in d. Informally, we will show the following result:
Theorem (Informal). Let be convex and smooth, such that composing a function with Barron norm b with L produces a function of Barron norm at most Byb.. Then, for all sufficiently small e > O, the minimizer of the energy functional in Definition
] can be e-approximated in the L2 sense
by a function with Barron norm O ((dBy)" ) yolo0//))
As consequence, when e. P are thought of as constants, we can represent the solution to the Fuler- Lagrange PDE Equation 3 by polynomially-size network, as opposed to an exponentially sized network, which is what we would get by standard universal approximation results and using regu- larity results for the solutions of the PDE.
We establish this by "neurally simulating" a preconditioned gradient descent (for a strongly-convex loss) in appropriate Hilbert space, and show that the Barron norm of each iterate- wwhich is function is finite, and at most polynomially bigger than the Barron norm of the previous iterate. We get the final bound by (i) bounding the growth of the Barron norm at every iteration; and (ii) bounding the number of iterations required to reach an e-approximation to the solution. The result in formally stated in Section
3 RELATED WORK
Over the past few years, growing line of work has focused on parameterizing the solutions to PDEs with neural networks. Works such as [E et al.] (2017): E & Yu (2017); [Sirignano & Spiliopoulos (0018) Raissi et al. (2017) achieved impressive results on variety of differrnt applications and demonstrated the empirical efficacy of neural networks in solving high-dimensional PDEs, even outperforming previously dominant numerical approaches like finite differences and finite element methods (Le 2007) that proceed by discretizing the input space, hence limiting their use to problems on dimennnnsi input spaces.
Several recent works look to theoretically analyze these neural network based approaches for solving PDEs. Mishra & Molinaro (2020) look at the generalization properties of physics informed neural networks. In Lu et al. (202T) show the generalization analysis for the Deep Ritz method for elliptic equations like the Poisson equation and Lu & Lu 2021 extends their analysis to the Schrödinger eigenvalue problem.
In addition to analyzing the generalization capabilities of the neural networks, theoretical analysis into their representational capabilities has also gained lot of attention. Khoo et al.) (2021 I) show
the existence of a network by discretizing the input space into a mesh and then using convolutional
NNs, where the size of the layers is exponential in the input dimension. (Sirignano & Spiliopoulos (018)) provide a universal approximation result, showing that for sufficiently regularized PDEs, there exists multilayer network that approximates its solution. In Jentzen et al. (2018): Grohs & Herrmann (2020); Hutzenthaler et al. (2020) provided a better-than--xponentii dependence on the input dimension for some special parabolic PDEs, based on stochastic representation using the Feynman-Kaa Lemma, thus limiting the applicability of their approach to PDEs that have such a probabilistic interpretation. Moreover, their results avoid the curse of dimensionality only over domains with unit volume.
For a vector valued function F: Rd + IRd, we will denote the divergence operator either by divF or by V F, where div F V. F
Ed 8.F
D2,
3

---

Under review as a conference paper at ICLR 2023
A recent line of work has focused families of PDEs for which neural networks evade the curse of dimensionality i..., the solution can be approximated by a neural network with a subexponen- tial size. Marwah et al. (202T) show that for elliptic PDE's whose coefficients neural networks with at most N parameters, a neural network exists that 6-approximates the solu- tion and has size ((d N). Chen et al. 2021 extends this analysis to elliptic PDEs with coef- ficients with small Barron norm, and shows that if the coefficients have Barron norm bounded by B, an e-approximate solution exists with Barron norm at most (dd B). The work by Chen et al. 2022) derives related results the Schrödinger equation.
approximable by
As mentioned, while most previous works show key regularity results for neural network approxi- mations of solution to PDEs, most their analysis is limited to simple linear PDEs. The focus of this paper is towards extending these results to family of PDEs referred to nonlinear variational PDEs. This particular family of PDEs consists of many famous PDEs, such as p-Laplacian (on a bounded domain), which is used to model phenomena like non-Newtonian fluid dynamics and non- linear diffusion processes. The regularity results for these family of PDEs was posed as Hilbert's XIX problem. We note that there are classical results like De Giorgi (1957) and Nash 19577 1958) that provide regularity estimates on the solutions of a nonlinear variational elliptic PDE of the form in Equation One can easily use these regularity estimates, along with standard univer- sal approximation results (Yarotsky] (017)) to show that the solutions can be approximated arbitrar- ily well. However, the size of the resulting networks will be exponentially large (i.e. suffer from the curse of dimensionality) Ss are of no use for our desired results.
4 NOTATION AND DEFINITION
In this section, we introduce some key concepts and notation that will be used throughout the paper. For a vector x E R4, we use |a||| to denote its E norm. Further, (*)) denotes the set functions Ω > E that infinitely differentiable. We also define some important function spaces and associated key results below.
Definition 2. For a vector valued faluedfunction g: + Rdd, we define the (P()) norm for p [1, 00) as
= ( Σ (xx())) dr
1/p
loalleo())
For 0o, we have
[ max Isi5d
119:ll ((0)
where (1)))(==(0) inf 0: [9(x))
| S for almost all 9).
Definition 3. For a domain 9t the space of functions H. (S2) is defined as
(2)) = {g Ω R: L2 (22), V E (2)) 0}. The corresponding norm for H (I2) is defined as (1lllng = ||
We will make use of the Poincaré inequality throughout several of our results.
Theorem (Poincaré inequality, Poincare (890)). For C Rd which open and bounded, there exists a constant C, such that for all u E Hg (S2)
Tu]l222(0) - Cpl Vull
This constant can be very benignly behaved with dimension for many natural domains even mension independent. Examples include convex domains (Payne & Weinberger 1960).
even di-
4.1 BARRON NORMS
For function Rdd R. the Fourier transform and the inverse Fourier transform defined as
f(w) = (2 [ f(a)e ((x)e-ix"u ix "x dx, and f(x) [ f(w)e dw.
1
(4)
4

---

Under review as a conference paper at ICLR 2023
The Barron nomn is an average of the norm of the frequency vector weighted by the Fourier magni- tude [f(w) A slight technical issue is that the the Fourier transform is defined only for f R - IR. Since we are interested in defining the Barron norm of functions defined over a bounded domain, we allow for arbitrary extensions of a function outside of their domain. (This the standard defini- tion, e.g. in (Barron 1993).)
Definition 4. We define F be the set of functions g E (2) such that the Fourier inversion formula g (2 (z) holds over the domain 92 i.e.,
F F=
9 Rd Vo 92 g(x))() = g(0) /
- (w)(
Definition 5 (Spectral Barron Norm, (Barron 1993). Let I' be a set of functions defined over Q such that their extension over Rd belong to F, that is,
T= (f Ω - 3g. gla = FF
Then define the spectral Barron norm ||
/
lf
inf glo=f.gEF JRa
(1 ( (w))
The Barron norm is an L1 relaxation of requiring sparsity in the Fourier basis-which is intuitively why it confers representational benefits terms of size neural network required. We refer to Barron 1993) for more exhaustive list of the Barron norms of some common function classes.
The main theorem from Barron (1993) formalizes this intuition, by bounding the size of 2-layer network approximating a function with small Barron norm:
Theorem (Theorem 1, Barron 1993). Let I such that [I/ll S and ir be probability
measure defined over Q. There exists a, E R&, b, E R and o R such that Σ |c/ $ 2C, there exists a function fa(x) = Ei= Go (af x + b). such that we have,
J (((()) fx(x))) p(dx) S
402
k
Here denotes sigmoidal activation function, i.e., lim, o (xx) and lim, (xx) = 0.
Note that while Theorem is stated for sigmoidal activations like sigmoid and tanh (after appropriate rescaling), the results are also valid for ReLUU activation functions, since ReLU((x) ReL.((z 1) in fact sigmoidal. We will also need work with functions that do not have Fourier coefficients beyond some size (i.e. are band limited), hence we introduce the following definition: Definition 6. Let Fw(()) be the set of functions whose Fourier coefficients vanish outside bounded ball, i.e..
Fw = {g Rd R s.t. Vw, |ol 2 W.ğ(w) = 03.
Similarly, we denote
Tw (f :f:SA Ω R Eg.9la f.g g Fw).
Since will work with vector valued function, we will also define the Barron norm of vector- valued function as the maximum of the Barron norms its coordinates:
Definition 7. For a vector valued function g Ω + Rd, we define [[gl] = max; llq: (m)))
5
MAIN RESULT
Before stating the main result we introduce the key assumption.
Assumption 1. The function Definition I can be approximated by a function L: Rd - such that there exists a constant Eg E [0, X) with supzeg [[V (i(x)]l2 S EL||z|12.
Furthermore, we assume that L is such that for any E (I)) we have Log E H3()), Log EF and
| | & Billola
(5)
for some constants B₁ 2 0, and p 0. Furthermore, if Tw then L. Tew for 0.
5

---

Under review as a conference paper at ICLR 2023
This assumption is fairly natural: it states that the function L can be approximated (up to e,, in the sense of the gradients of the functions) by function L that has the property that when applied to a function g with small Barron norm, the new Barron norm is not much bigger. The constant p specifies the order of this growth. The functions for which when the dependence of By, on is at most polynomial-so that the final size of the approximating network does not exhibit curse of dimensionality. For instance, we can take L be multivariate polynomial of degree up to P: we show in Lemma 7 the constant Br, is O(df) (intuitively, this dependence comes from the total number of monomials of this degree), whereas p and k are both ((P).
results
most interesting
With all the assumptions stated, we now state our main theorem:
Theorem 4 (Main Result). Consider the nonlinear variational elliptic PDE in Equation 3 which satisfies Assumption 1 and let u² (S2) denote the solution the PDE. If un E (S2) function such that ug Two then for all sufficiently small >> 0, and
2 (E(w)) E(("))) /log
T
log
(1+C,)A
there exists function WT E (S2) such that ur TRTW, and Barron norm bounded as
x²+1
A3
Uur (1s(()) - (1+ (Cp 1)A3
dk W
Furthermore, ur satisfies l|ut - u*1l (1H)(00) Set where
\ el (Iuu (1eg 2{((o)) ES (C, + 1)A3
(11+ A3 [I,(C,- (C,+ (A teu)) T -1).
A e
Remark The function ug can be seen as an initial estimate the solution, that can be refined to an estimate ur, which is progressively better at the expense of larger Barron norm. A trivial choice could be uh = 0, which has Barron norm 1, and which by Lemma would satisfy ((u)) E(u") S Al[u" 1|2 (
Remark 2: The final approximation error has two terms, T goes to oc as E tends 0 and a conse- quence of the way ur constructed-byy bby simulating functional (preconditioned) gradient descent which converges to the solution to the PDE. The error term ? stems from the approximation that we make between L and L, which grows as T increases it is consequence of the fact that the gradi- ent descent updates with L and progressively drift apart as tends to co.
Remark 3: As in the informal theorem, if we think of p, A, X, Cps k, [| as constants, the theorem implies u² can be e-approximated in the L sense by function with Barron norm O (dB) plo Therefore, combining results from Theorem 4 and Theorem 3 the total param-
eters required to e-approximate u* by 2-layer neural network is O (dB1) 2p 2p/x()/)
We further note that this result recovers (and generalizes) prior results which bound the Barron norm of linear elliptic PDEs like Chen al. (2021). In these results, the elliptic PDE takes the form -div(AVu) and A is assumed to have bounded Barron norm. Thus, || ul] S d² [[] (l]l hence satisfying Equation in Assumption I with p
Remark 4:
PROOF OF MAIN RESULT
The proof will proceed by "neurally unfolding" a preconditioned gradient descent on the objective E in the Hilbert space H. (f). This is inspired by previous works by Marwah et al. (2021) 1); Chen et al. (2021) where the authors show that for a linear elliptic PDE, an objective which is quadratic can be designed. In our case, we show that & is "strongly convex" in some suitable sense thus again, bounding the amount of steps needed.
More precisely, the result will proceed in parts:

---

Under review as a conference paper at ICLR 2023
1. First, we will show that the sequence of functions {u where u4+11
u 17(I
A)- (1,) can be interpreted as performing preconditioned gradient descent, with the We show that in some appropriate sense (Lemma 2, E is strongly convex in (h) the updates converge at rate of O(log(1/e))
(constant) preconditioner (I A)- P
2. We then show that the Barron norm of each iterate W4+1 can be bounded in terms of the Barron norm of the prior iterate Ue. We show this in Lemma where show that given Assumption
the | [[a) is Odd Il (()))) By unrolling this recursion we show that |
the Barron norm of the c-approximation of u* is of the order O(d"" || where T are the total steps required for e-approximation and || (00)) is the Barron norm of the first function in the iterative updates.
We now proceed to delineate the main technical ingredients for both of these parts.
6.1 CONVERGENCE RATE OF SEQUENCR
The proof to show the convergence to the solution u² is based on adapting the standard proof (in finite dimension) for convergence of gradient descent when minimizing strongly convex function f. Recall, the basic idea is to Taylor expand f(r + 8) s f(x)= + Vf(=)' + O(16)]]². Taking ő = 7 V (xx), we lower bound the progress term 7/ f=)| using the convexity of f. and upper bound the second-order term 17 [V using the smoothness of f.
We follow analogous steps, and prove that we can lower bound the progress term by using some appropriate sense of convexity of E, and upper bound some appropriate sense of smoothness of E, when considered as a function over H. (S2). Precisely, we show:
Lemma (Strong convexity of Hb). If E L are as in Definition 1 we have
1. Vu, E (9) (DE(u), 12))) Sn -i.vivvv (idivvV((Vu)vddeee Sa vL(Vu) Vedr.
2. Vu, U E H. (2): n||u - v [mg S (DE(u) DE 1)20(00) S Allu =1( + + v) Lo(0) s E(u+t))E(( < (DE(u), v) (200+
3. Vu, u E (1)
\ | v [2
4. Vu (9) - l I E((u) E(u²) < タ ||u u- 1 11 (0)
Part I is helpful way to rewrite an inner product of a "direction" with DE(u)-it is essentially consequence of integration by parts and the Dirichlet boundary condition. Part 2 and 3 are common proxies of convexity: they ways of formalizing the notion that E is strongly convex, when viewed as a function over H3 (?). Finally, part 4 is a consequence of strong convexity, capturing the fact that if the value of E()) is suboptimal, u must be (quantitatively) far from u". The proof of the Lemma can be found in the Appendix (Section B.1)
When analyzing gradient descent (in finite dimensions) to minimize a loss function E, the standard condition for progress is that the inner product of the gradient with the direction towards the optimum lower bounded (DE(u), u² - u) L2(() 2 ai|uu- u² I12 From Parts and 3, one can readily see that the above condition only satisfied "with the wrong norm": i.e. we only have (DE(u), u* u) (()) 2 alIu u= [22 (0)) We can fix this mismatch by instead doing preconditioned gradient, using the fixed preconditioner (T -)- Towards that, the main lemma about the preconditioning we require is the following one:
Lemma (Norms with preconditioning). For all u E (9),, we have
1. [1(1 -) -V Vulle = ||(1 11 A) Aull S ||
2. ((1 -)- 1,, 1) 2 1 ~A- 1--,,,) 1)
The first part of the lemma is a relatively simple consequence of the fact that the operators A and V "commute", and therefore can be re-ordered. The latter lemma can be understood intuitively as
7

---

Under review as a conference paper at ICLR 2023
(I- and A -! act similar operators eigenfunctions of A with large eigenvalues (the extra does not do much)- and are only different for cigenfunctions for small eigenvalues. However, since the smallest eigenvalues is lower bounded by 1/Cp. their gap be bounded.
Next we utilize the results in Lemma and Lemma to show preconditioned gradient descent exponentially converges the solution to the nonlinear variational elliptic PDE in
Lemma 4 (Preconditioned Gradient Descent Convergence). Let u² denote the unique solution to the PDE Definition 3 For all E N, we define the sequence of functions
A3
U++1 < ut (1+ Cp+Ca)A3 -)-1-111100 (
(1
(6)
uo (11) after t iterations we have,
A E(u++1) - -((u)) 1 (1- (1+ C₁)
(E(wo) - E(u²))
The complete proof for convergence can be found in Section Therefore, using the result from Lemma i.e., llue
B.3 of the Appendix.
2 ())
I A (E(ue) E(u))), we have
2 1
1 2(1 + Cp)A1
|/4- uu 1111110000
(E(up) E(u"))
and | uz u'I ((( I c after T steps, where,
T 2 log (((u)) - -E(u))
/log
Xr/2
(1+C,)A4
6.2 BOUNDING THE BARRON NORM
Having obtained a sequence of functions that converge to the solution u², we bound the Barron norms of the iterates. We draw inspiration from Marwah al. (202T): Lu et al. (202T) and show that the Barron norm of each iterate in the sequence has bounded increase on the Barron norm of the previous iterate. Note that in general, the Fourier spectrum a composition of functions can not casily be expressed in terms of the Fourier spectrum of the functions being composed. However, from Assumption I we know that the function L can be approximated by L such that L g has a bounded increase the Barron norm of g. Thus, instead of tracking the iterates Equation 6 we track the Barron norm of the functions in the following sequence,
ū++1 = ū, - 1 A) Dē.)))
(8)
We can derive the following result (the proof is deferred to Section D.I of the Appendix):
Lemma 5. Consider the updates in Equation g if ū, 1'w, then for all n E (0, [C,+1)A7] we have
x
ūt-1 1 F&w, and the Barron norm of u4+1 can be bounded as
| $ (1 nd(k)W,) [|
The proof consists of using the result in Equation 5 about the Barron norm of composition of function with ī., as well as counting the increase in the Barron norm of a function by any basic algebraic operation, as established Lemma Precisely we show:
Lemma (Barron norm algebra). If h, h1, he T, then the following set of results hold,
Addition: || + hall & |/ (()))) |/2||
Multiplication: || h2ll8 & ||A||| (12ll
Derivative: if E Tw for [d] have [|abbl)a()) W||M)BB)
8

---

Under review as a conference paper at ICLR 2023
Preconditioning: if h T, then [| -) >>h-lll(o))) [1)])
The proof for the above Lemma can be found in Appendix [33 It bears similarity to an analogous result in Chen et al. (2021), with the difference being that our bounds are defined in the spectral Barron space which is different from the definition of the Barron norm used in Chen et al. (2021).
Expanding on the recurrence in Lemma 6 we therefore have that after T we have, we have Wr s k" Wo and hence 4++1 E Ta and the Barron norm of UT can be bounded as follows,
| ((()) (1 + ndk W B; p"+ 2/2 (()))
(9)
Finally, we exhibit natural class of functions that satisfy the main Barron growth property in Equations 5 Precisely, we show (multivariate) polynomials of bounded degree have effective bound on p and By:
Lemma 7. Let f(x) = Eo.loss AA 2.a²)
where is
and x R. If
Rd - Rdd is such that g E Tw, then we have o g rww and the Barron norm can be bounded as, |If goll(()))) S d012 Ea.
1/2 [|))
Hence if 7. is polynomial of degree P the constants in Assumption | will take the following values 1/2
Bj= = dd/2 (Ea,alis Aal?)
and r P.
Finally, since are using an approximation of the function L we will incur error at each step of the iteration. The following Lemma shows that the error between the iterates u, and the approximate iterates ú, increases with The error calculated by recursively tracking the error between o, and de for each in terms of the error at I 1. Note that this error can be controlled by using smaller values of 17.
Lemma 8. Let L Rd + R be the function satisfying the properties in Assumption | such that supzER [[V vi(x)ll2 s <1||æ1|22 and we have E(u) Sn. L(Vu)dz and E(u)
Sq L(Vu)dæ. For E (0, (C,+1JAJ consider the sequences,
u4+1 4,- 17( A) DE(u), and ūtt1 = = -nnII A) D')D(())
then for all E N and R < | (2)( + {(ua) we huve,
[u ( t
+A))
CONCLUSION AND FUTURE WORK
In this work, take representational complexity perspective neural networks, they used to approximate solutions of nonlinear variational elliptic PDEs of the form -iv(vvL(Vau = 0. We prove that if L. is such that composing T. with function of bounded Barron Barron norm a bounded fashion, then we can bound the Barron norm of the solution u² to the PDE PDt-pootntyally evading the curse of dimensionality depending on the rate of this increase. Our results subsume and vastly generalize prior work on the linear case (Marwah et al. 202 Chen et al.) 2021 Our proof consists of neurally simulating preconditioned gradient descent on the energy function defining the PDE, which we prove is strongly convex in an appropriate sense.
increases the
CChnnChen&
There are many potential avenues for future work. Our techniques (and prior techniques) strongly rely on the existence of a variational principle characterizing the solution of the PDE. In classical PDE literature, these classes of PDEs are also considered better behaved: e.g., proving regularity bounds is much easier for such PDEs (Fermández-Redt & Ros-Oton] 2020). There are many non- linear PDEs that come without a variational formulation-c.22 the Monge-Ampert equation-for which regularity estimates are derived using non-constructive methods like comparison principles. is a wide-open question to construct representational bounds for any interesting family of PDEs of this kind. It is also a very interesting question to explore other notions of complexity-ea. eeg. number of parameters in a (potentially deep) network like in Marwah et al. (202T), Rademacher complexity, among others.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3):930-945, 1993.
Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural PDE solvers. arXiv preprint arXiv:200.003376, 2022.
Johannes Martinus Burgers. The nonlinear diffusion equation: asymptotic solutions and statistical problems. Springer Science & Business Media, 2013.
Ziang Chen, Jianfeng Lu, and Yulong Lu. On the representation of solutions to elliptic PDBs in Barron spaces. Advances in Neural Information Processing Systems, 34, 2021.
Ziang Chen, Jianfeng Lu, Yulong Lu, and Shengxuan Zhou. A regularity theory for static Schrödinger equations on Rd in spectral Barron spaces, arXiv preprint 1xXi::2211.1072, 2022.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho.
Lagrangian neural networks. arXiv preprint (xXi:20033044630 2020.
Memoria di Ennio De Giorgi. Sulla differenziabilitte l'analiticita delle estremali degli integrali
multipli regolari. Ennio De Giorgi, pp. 167, 1957.
Weinan E and Bing Yu. The deep Ritz method: deep learning-based numerical algorithm for solving variational problems. arXiv preprint urXiv: 2017.
Weinan E, Jiequn Han, and Arnulf Jentzen. Deep learning-based numerical methods for high- dimensional parabolic partial differential equations and backward stochastic differential equa- tions. Communications in Mathematics and Statistics, 5(4):349-380, 2017.
Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
Xavier Fernández-Real and Xavier Ros-Oton. Regularity theory for elliptic PDE. Forthcoming book, 2020.
Philipp Grohs and Lukas Herrmann. Deep neural network approximation for high-dimensional elliptic PDEs with boundary conditions. arXiv preprint arXiv:2007.053884 2020.
Jiequn Han, Arnulf Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505-8510) 2018.
Jun-Ting Hsieh, Shengjia Zhao, Stephan Eismann, Lucia Mirabella, and Stefano Ermon. Learning neural PDE solvers with convergence guarantees. arXiv preprint arXiv: 1oXivv/9060000 020000 2019.
Martin Hutzenthaler, Arnulf Jentzen, Thomas Kruse, and Tuan Anh Nguyen. A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations. SN partial differential equations and applications, 1(2):1-34, 2020.
Arnulf Jentzen, Diyora Salimova, and Timo Welti. A proof that deep artificial neural networks overcome the curse of dimensionality the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients. arXiv preprint 1rx1v:1809.07321, 2018
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, (907:69977777738555555
2021.
Yuehaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric PDE problems with artificial neural networks. European Journal of Applied Mathematics, 32(3):421-43: 2021.
Miglena N Koleva and Lubin G Vulkov. Numerical solution of the Monge-Ampère equation with an application to fluid dynamics. In AIP Conference Proceedings, volume 2048, pp. 030002. AIP Publishing I.C.C 2018
10

---

Under review as a conference paper at ICLR 2023
Holden Lee, Rong Ge, Tengyu Ma, Andrej Risteski, and Sanjeev Arora. On the ability of neural nets to express distributions. In Conference on Learning Theory, pp. 1271-1296. PMLR, 2017.
Randall J LeVeque. Finite difference methods for ordinary and partial differential equations: steady- state and time-dependent problems. SIAM, 2007.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differ- ential equations. arXiv preprint arXiv:2003.03485, 2020b.
Jianfeng Lu and Yulong Lu. A priori generalization error analysis of two-layer neural networks for solving high dimensional Schrödinger eigenvalue problems. arXiv preprint AXXi:22055001228. 2021.
Jianfeng Lu, Yulong Lu, and Min Wang. A priori generalization analysis of the deep Ritz method for solving high dimensional elliptic equations. arXiv preprint urXiv:2101.0/708, 2021.
Tanya Marwah, Zachary Lipton, and Andrej Risteski. Parametric complexity bounds for approx- imating PDEs with neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Siddhartha Mishra and Roberto Molinaro. Estimates on the generalization error of physics informed neural networks (PINNs) for approximating PDEs. arXiv preprint arXiv:2006. 16144, 2020.
John Nash. Parabolic equations. Proceedings of the National Academy of Sciences, 43(8):754-758, 1957.
John Nash. Continuity of solutions of parabolic and elliptic equations. American Journal of Mathe- matics, 80(4):931-95) 1958.
Lawrence E Payne and Hans F Weinberger. An optimal Poincaré inequality for convex domains. Archive for Rational Mechanics and Analysis, 5(1):286-292 1960.
Henri Poincaré. Sur les équations aux dérivées partielles de physique mathématique. American Journal of Mathematics, pp. 211-294, 1890.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learn- ing (Part I): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.1056/ 2017.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data, Science, 324(5923):81-88) 2009.
Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339-1364, 2018.
Casper Kaae Sønderby, Lasse Espeholt, Jonathan Heek, Mostafa Dehghani, Avital Oliver, Tim Sal- imans, Shreya Agrawal, Jason Hickey, and Nal Kalchbrenner. Metnet: A neural weather model for precipitation forecasting. arXiv preprint arXiv:2003. /2/40, 2020.
Kathryn Tunyasuvunakool, Jonas Adler, Zachary Wu, Tim Green, Michal Zielinski, Augustin Židek, Alex Bridgland, Andrew Cowie, Clemens Meyer, Agata Laydon, et al. Highly accurate protein structure prediction for the human proteome. Nature, 596(7873):590-554)) 2021.
Hilary Weller, Philip Browne, Chris Budd, and Mike Cullen. Mesh adaptation on the sphere using optimal transport and the numerical solution of a Monge-Ampère type equation. Journal of Computational Physics, 308:102-122 2016.
Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103-114, 2017. 2017.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1
PROOF FOR LEMMA 1
The proof follows form Fernández-Real & Ros-Oton 2020 Chapter completeness.
We provided it here for
Proof of Lemmal / for all E
If the function u' minimizes the energy functional in Definition) |
then we have
Eu)) s E(u + e4p)
where & (2))). That is, have minima at 0 and taking derivative w.r.t we get,
E(u + (e) E(u) 0
dE(u](p) = lim
Jg L (Vu (Vp) dr 11 L L(Vu)dr 0
lim
c+0
Jg VVV(VVV + fr(x) 0
lim
where for all Ω have [r(x)| S Vip suppeR DDD(
Since e - the final derivative is of the form,
dE(u / VL(VV) =
(10)
For functions r E Ho (22) note the following Green's formula,
(vr)sdo / Vr Vodr Lon On
Ou edT
(Vr)sdr =
Vr Vddx
(1I)
12
52
Using the identity Equation 11 in Equation 10 we get,
dE [E[u((pp= = / div (VL(Vu)) edx = 0
That is the minima for the energy functional is reached at which solves the following PDE,
dE(u) = div (VL(Vu)) =
B PROOFS FROM SECTION 6.1
B. PROOF FOR LEMMA
Proof. In order to prove part we will use the following integration by parts identity, for functions Ω IR such that and Ω + R. and r, E HQ)))
|
sdx sds
( Os
dr
rsndl
(12)
dr.
Or:
where is normal at the boundary and dT is infinitesimal element of the boundary 00.
12

---

Under review as a conference paper at ICLR 2023
Using the formula in Equation 12 for functions u, Ho (12, have
(DE( ( (-V (L(Vu).))200
VL(Vu)udz
0 (VV(VV)) vdx
Ox:
Ou dx + / (VL(Vu)), unddr
(VL(Vu));
Ooi
i=1
VL(Vu) Vedr
where in the last equality we use the fact that the function E (92), thus v(z) = 0, Vr E aR. To show the second part, first note since L Rd - strongly convex and smooth, we have
Vx, E R A|I= ulla & [V)() L s A|i - vll2.
(13)
This implies
Vx Ω. (VL(Vu(>)) VL(Vo(>))]l Al)V((>) Vu(2)]2
(14)
Taking square on each side and itegrating l we get
VV(Vv(()) Vl( 34 ^² /g
/ Vu(x)- vo(>)]
IVV(VV) - VL(V & A -
Vullzz())
On the other hand, from part 1, we have
(DE(u) DE(v), u 1) (VL(V - VL(Ve), Vu Vo)
Hence, by Cauchy-Sehwartz, get
(DE(u) DE((), u v) 22(0))
(VL(Vu) VL(VV) Vu Vo) LP())
VL(Vu)) VVV Volle
A/Vu -
which proves the right hand side of the inequality in part 2.
For the left size of the inequality, by convexity of L we have Vz, E R -VL())-V(W)>T 8) 2 A|Iz - v1/ is convex,
L (Vu(=)) - VL (V)(() - -v(x))))(())) 2 > - ve(x)]1
Integrating over Ω we get,
/ (V(()())) VL( VV(())-Vo(a)dd
Vo(a))de 2 A / |Vu(x) -
(V - VV(VE), Vu Ve) 2 A/ Vollia0
(15)
Using part again, this implies Equation
(DE(u) (E u LP = (VI(V)) VI(Ve), Vu Vo)
2 Vl|Vo - Vulli -
wanted.
13

---

Under review as a conference paper at ICLR 2023
To show part 3, we first Taylor expand L to rewrite the energy function
/ I(Vu(x) + Vu(z))dr
E(u+ 1)
1 +
dr
(16)
where 5 Idd (and is potentially different for every x E M... Since the function is strongly convex we have
Ala S D²L(0) S Ala-
Plugging in these bounds in Equation 16 we have:
E(u + v) < / VLVV)))))VVVVV())VV)))
A ( (Vo do
A Vv, Vo) 2
E(u+ & E(u) (DE(4) +
(17)
as well
E(u+ 2 / - A \o(())1P)) dr
2
v) 8(u) + (DE(4) 2タ (v, Vo)
(18)
Combining Equation 17 and Equation
18 we get,
Volliao + (DE(u), s E(u+r) - E(u) (DE(u), o) A
Voll
Finally, part 4 follows by plugging in 2² and = =" in part 3 and using the fact that DE("") =0
B.2 PROOF FOR LEMMA
Proof. Let {Ai, Φ; 1 denote the (eigenvalue, eigenfunction) pairs of the operator - where 0 <
A1 S A2 S. ..., which are real and countable. (Evans) (2010), Theorem Section 6.5)
Using the definition of eigenvalues and eigenfunctions, we have
(-Av,))aa00
A1 = inf (EH)(R)
|
(Ve, Vo) |0lli200
inf (E
C
where the last equality we use Theorem 2
Let us write the functions v in the eigenbasis = E; 140;- Notice that eigenfunction of -A is also eigenfunction for (T A)-1, with correspondinding eigenvalue I+X,
14

---

Under review as a conference paper at ICLR 2023
Thus, to show part 1, we have,
(I - - Vr Vol (2)) = [| -
"Ar"
(2
8
+
X
12(0))
P:0:
1=1
(2
Σ
p || (1))(())
where in the last equality we use the fact that Φ1 are orthogonal.
Now, note that (T - = Ei= Mi0; now, note that since a S A2 ... and is monotonically increasing, have for all E N
(19)
1+ A
C1+
and note that + are the eigenvalues for (-A)-1 for all i N.
Now, bounding ((1 - -)- L)(0))
2
F4
Φi
P:::
Xi
12
14
|| |0
(()
(20)
X
where the orthogonality of d/s to get Equation 20 further lower bound ((1 -
Using the inequality in Equation 19
-|---))-1 A)-1o 0, (2))) as follows,
1 ((-A)
((1 - -)- L3())
C,) 1 (
2, L2()))
1 C
where we use the following set of equalities in the last step,
0 M | [()
((-A)-'o,
A
L2(R)
B.3 PROOF FOR CONVERGENCE: PROOF FOR LEMMA 4
Proof. For the analysis we consider = (I+C,)A3
Taylor expanding as in Equation 17 we have
El E(u+1) < (wu) (VL(Vu)) V(I - -A) -( (E(u))
Temm
?? + 2
V(T A) DE ( (1
(21)
Term
where we have in Equation 17 plugged in u4+1 = -n (1 - --) DE()..
First we lower bound Term 1. Since ut is the solution to the PDE in Equation we have DE(u*)
0. Therefore we have
(VL(V V(I A) DE- (A(0) (VL(Vu)) v(I -) (WE(u)) - DE L2
(22)
15

---

Under review as a conference paper at ICLR 2023
Similarly, since u² is the solution to the PDE in Equation 3 Equation 2 we have for all t HQ)))
(VL(Vu"), V = 0. Using this Equation 22 we get,
(VL(Vu)), V - - (E (2)
(VL((Vu), ( - A -1 (DE(wr) DE((²))) (2(3))
[VII(Vu), (T - -)- (DE(w)) (E(("))) 12(02) + (VI(VV") V(r ADE((DE(w))) - DE(("))) 12
-(VL(Vu) VVLVV), V(I (DE(U)) 13(0)
(23)
Using Equation 23 we can rewrite Term as
(VL(Vu)), - (E) (2(2)
(VI(Vu)) VI(VV'), V(T -) A)- (DE(14) - DE(u") 1)) 12((0)
(VL(Vu)) VL(VV'))) V(I A) (-V (VL(Vu)) Vl("")) dr
(VL(VL)) - VL(V V(I - A) -1 (- - L(u"))) da
(VL(Vu) VL(VE (I - (--) (VL(v Vl(u")) dx
(24)
- Vo= - Au and the fact that V commutes with (7 - A--1(-A). where in step (i) we use
Plugging part 2 of Lemma 3 Equation 24 we get
(VL(Vu), V(I -) A)
(E (A(0))
1
/ (VL(V..)
VL(("")) dx
1+ Cp
(VL(Vu)) VL(VE'), VL(V)) (22V
C
X²
| Vu" (1:2(()0)))
Cp
2X2
(1 C,)A (E(w) E(u)))
(25)
where (i) follows by part of Lemma and we use part 4 of Lemma 2 for the last inequality.
We will proceed to upper bounding Term 2. Using part of Lemma 3 we have
| (E( = || - (DE DE
[ A) - (VL(Vu) VL(V 12))))
-V(I-A)-((-)) (-A)) - L(V=")) (2(
| - VL(Vu'))]
(18))
[VL(Vm) VI(V=')]
< A2 V Vu-1 (2())
2A2
< X (E(u.) E(u"))
(26)
Here, use the fact that -V V A in step (i) and the fact that V commutes with (I A) A) in step (fi), and finally we use the result from part 2 of Lemma to get the inequality
in (ii1).
16

---

Under review as a conference paper at ICLR 2023
Combining Equation 25 and Equation 26 in Equation 21 we get
2x²
12
E(u++1) - E(u")= S E(u)) E(("))
( C,) C,)
-xx
n (E(u)) E(u"))
Since 77 =9/((11 + Cp)AA) we have
A5
8(u441) E(u)) -E(4'))(u)))
(1 C,) (((u)) (8(u))-((u))
1t
8(441) - - 1
(1+ Cp)41) (E(uo) E(u")).
C ERROR ANALYSIS
First, we will need the following simple technical lemma showing that the (S2) norm is self-dual: Lemma 9. The dual norm of (larg(a) is || [lagco))
Proof. If |ull. denotes the dual norm of ||u| (g) by definition we have,
||]l+ =
sup (R)
(u, ()
||> Mggea =1
sup (E) ||e|| [eell =1
(Vu, (2)
sup (E) ugjan
where the inequality follows by Cauchy- Schwarz. On the other hand, equality can be achieved by taking v = - Thus, ||u]] = |uu]lz=(0) = [| as we wanted.
With this can prooceed to the proof of Lemma g
C.1 PROOF FOR LEMMA
Proof. We define for all (. = ü, uy and will iteratively bound ||re|]
Starting with up and 0, 0, we define the iterative sequences as,
uD ug
M1+1 Uz 77(1 - -))
ū = un
ū++1 = ūt n1(1 - -)-
where 7) E (0,
3 [T+C))A) Subtracting the two we get,
u+t1 Uc+1 = -g - -4 - 17(T - -)-1 (DE(U)) - DE(U))
T241 = - A)-1 - + DE((E))
(27)
Taking H. (2) norm on both sides we get,
(r4lllag00o S |Ieellggo) + [I A)-1 (DE(U) + - 119600
(28)
17

---

Under review as a conference paper at ICLR 2023
||I (1- -) + + DE(u))
Towards bounding
from Lemma we know that the
dual norm of | (1) is ||w0 [sg) thus,
(1 (1-
re) (E 173(022
- (DÉ(U + - [D) 12(
sup (0) |1 =1
( (DE()) + +r)) +1)) ri). vD) LP(0)
sup (EM)()) [1))
sup (E [0 x
(V(7-4) -)- (DE(u +r2) DE(())) V2) (2(())
((II-))->> (vI Vri)) v1)) 12(0)
= sup se (0)
+ sup PEH (9)
(V(I- - A (VL(VE - (9) (2
|P 1l 1 =1
Vi(Vu, ( V Vr.), v.) 120)
sup PE (2) [ =1
+ sup (0) [||| 11
(VL(Vu+ + -V(1)) (2))2
$ <L Vrellaay VVLVV, + - VVV
(29)
Using the Lipschitzness of VL, we have
[[VL(Vu(()) + Vr((z)) VL(Vu))ll < VV((V.(<)) (L())))))))
sup D2 (rr
DERa
< (+)))
(30)
Squaring and integrating over Ω on both sides we get
VVL(Vu(x)) + Vre(z)) VL(Vu)llo(0)) S V
(31)
Pluggin in Equation in Equation 29 we get,
(I A) - + r2) - DE(())
(A ( tel Vullea)))
(32)
Hz(00
Furthermore, from Lemma we have for all t E N,
A5
E(w) - E((u) < 1
E(uo)
C,A4
< E(wo)
and
2
1 - ==1l
5 (E(u:) E(uo))
7
2
$ A
E(wo)
Hence we have that for all t E N.
||ullagao & ||"2 (2)(2)) + A
E(uo) = R.
18

---

Under review as a conference paper at ICLR 2023
Putting this all together, we have
||T- ( - -E(ue +re) DE(WE)) (2(
S (A+)))) +
(33)
Hence using the result from Equation 33 in Equation 28 to get.
| (1) S (1 + n(A + EL)) II ( + FEL
112-11113000 RL
((1 + n(A +c2))
(34)
EL
where we use the fact that |eellezg = O.
2 Notice that ez << A. Further, we have E (0, (I that is 1) S 1, it implies that 17(A + eL) <
1.
Hence we can further bound Equation 34 as follows,
CL ((++n(++e))) 1)
|| I A+E
D PROOFS FOR BARRON NORM APPROXIMATION: SECTION 6.2
D.1 PROOF FOR LEMMA S
BARRON NORM RECURSION
Proof. Note that the update equation looks like,
4 ut nT[I - A) DE(w))
-
-t -n-n([=a== Σ LV()))
(35)
From Lemma 6 we have
V
B(0) max [1:.uel :E[d]
W. W.llwellBao
(36)
Note that since u, Iw, we have Vu, E Tw, and L(Vue) E Taw, (from Assumption
Therefore, we can bound the Barron norm as,
(1 ()- ( Σ L((()))
0 L( Vve
B(2)
8(9)
<d T V ((
$ <((aW))\]L((vu)))
d(kW,) B/ (l
where we use the fact that for a function h. have ||(I -|(I-A)) (((1-)--1 ->>->(ll)))) S [| from Lemma 6
in (i) and the fact that I(Vu.) E T&w, in (ii). Using the result of Addition from Lemma 6 we have
| I ((()) S [| + (nd(kW:)) By) (|
- (1 + n)(kkW)) By) || |
19

---

Under review as a conference paper at ICLR 2023
D.2 PROOF FOR BARRON NORM POLYNOMIAL
Lemma 10 (Lemma] restated). Let
f(z)
Σ
a.ja/SP
where is a multi-index and E Rd and Aa R is scalar. If g R - R is function such that g E D w. then we have f log g Pw and the Barron norm can be bounded as,
|f | S 1/2
Σ
AAal
|19l|
a.JoSP
Proof. Recall from Definition 7 we know that for vector valued function g R - Rdd we have B(0)-
||
||
iE|d
Then, using Lemma 6
we have
P
[ B(0)
Σ
Aa
[a/o|=0
0
1=1
B(9))
P
Aa
0./a/=0
2m
18(52)
P
A
9;
0./a/=0
i=1
B(R))
P
|
II 1=1
9:
0.|0/=0
B(A)
P
Σ
|Aal
II v=ll
1
(1(((0)
0,/a/=0
|
II
119:
(8(())
0./a/=0
1/2
1/2
P
P Σ a,/a/==
2
II
((S)
(37)
0./o/=0
where we have repeatedly used Lemma 6 and Cauchy- Mnuchy-Schwartzr in the last line. Using the fact that for multivariate function R + Rdd we have for all E [d]
[1al|aa || [()))
Therefore, from Equation 37 we get,
1/2
1/2
(g) x
Σ
|Aa?
Σ
[I
(())
Ed a,
2
0:/o/SP
P
1/2
1/2
S
Σ
|Aa
(1
(a//a/S
1/2
S S/12
Σ
Aa
|19l|
0s/a|SP
20

---

Under review as a conference paper at ICLR 2023
Since the maximum power of the polynomial can take is P from Corollary | we will have o gg Tpw.
D.3 PROOF FOR BARRON NORM ALGEBRA:LEMMA G
The proof of Lemma is fairly similar the proof of Lemma 3.3 in Chen et al. 2021 the change stemming from the difference of the Barron norm being considered
Proof. We first show the result for Addition and bound || + h2llBa))
[
[| + h2llB(0)=
inf en h ET Jud
(1 + [|wll2) 91 + (w
92 22le=h2.92E3
inf 9i 1a= eF go go Bd
(1 + [klll)) [w] + 92(w)) dw
/ + [|]]|2) lái (w)
/ (1 + (|l|l2) (w)(w) dw
<
inf
inf s2 la EF
91la=hi.0 EF ga
[11 + bell I [ ((())) + |
For Multiplication, first note that multiplication of functions is equal to convolution of the functions in the frequency domain, i.e., for functions 91 Rd -> d and Ø2 R -> d, we have,
91
(38)
Now, to bound the Barron norm for the multiplication of two functions,
|A1 he hellss0)
inf
(1 ||l]|2)911
h= 91
Rd
inf Mi
(1+ ||w||2)|11 * (2(w)l
EF
Rd
#2 n 1
inf 2= 91E 92 12
(1 + [w]ll2) 16n (z)02(w - 2) dwdz
WERR 2EER
/
inf
(1 + || 2||2 ||=||2 + || | =112) 10n (=) =)] dwdz
9i n= h.
WERR
2E3d
82 2
Where we use ||w||2 S || 2|12 + || and the fact that
/. | (1)2l 2)lddd2 >> 0.
Collecting the relevant terms together we get,
/ ZERR (1+ ||w =112) (1 + [|=||2) l (x) 122(w 2) dw
[/1 fo (la(a) I
inf in halzhhi EF WERR 92
inf s1hn OF 8a
((1 + ([w1l29] * ((1 + || )ğz (w))
Hence using Young's 's convolution identity from Lemma we have
( (1+ || (1))))) (S (1 + (12)))))2((
|/1 he (1)) I
inf 91 mo EF 5
Il/ halls I |141 (l
21

---

Under review as a conference paper at ICLR 2023
In order to show the bound for Derivative, since h Tw, there exists function R + R such that,
g(z) /
A ((w)dw
1w sW
Now taking derivative on both sides we get,
0,g(#) = 1 ie""""u;glu (Wjō(w)
(39)
1wllloss
This implies that upper bound [6g[(w)
(,g(w) = (w,ō(w)
[0g(w0] S Wl9(w))
(40)
Hence we can bound the Barron norm of 0,h as follows:
inf I glo=h.0EFw ||wll ow SW
||ā
(1 + -wll ( dw
inf / ewF SW
(1 |||| W (
I
W inf gla=h.gEFw SW
(1+ [|/l)))) ((w) do
In order to show the preconditioning, note that for function Rd RR f (T A)- then
have then we have (1 A)f = g. Using the result form Lemma 12 we have
g(w)
(1 [wwl3) g(w) = B(w) 1 1
Bounding |I(T -)-
II(I -)) inf gla=h.ge I L /
1 + |||||
(1 + || س(w)d
inf ggat weRd
(1+ + (
[I(I A S |[
Corollary 1. Let Rdd + R then for any E N we have l|g S Ilal Furthermore, if the function g E Fw then the function g² E Taw.
Proof. The result from I|s0)) follows from the multiplication result in Lemma and we can show this by induction. For = 2, we have from Lemma 6 we have,
Ilan) $ |1 (1)
(41)
Assuming that we have for all n till k we have
l9g (B $ |1
(42)
for n = 1s we get,
llg (1)) = 119g (1((0) S ||ll Ilg" " S |
(43)
To show that for any the function a Tew, write g* in the Fourier basis. We have:
22

---

Under review as a conference paper at ICLR 2023
g'(x) (Ar olws)e Se a deis j=1
& II
I Mulwsa (JEin wy= my - dwr kdw
In particular, the coefficients with ||wllo kW vanish, needed.
Lemma 11 (Young's convolution identity). For functions E LP(RR) and h E L and
1 r
1
+1
p
where 1 P.q, S have
Ilf g/l S 11g1lp11b114
Here denotes the convolution operator.
Lemma 12. For differentiable function
Rd R. such that f E (IRd) we have Vf(w) = fww
23


Reviews:
[{"review_id": "eCfTzwGlsT2", "reviewer": "Reviewer_N9yD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors consider the potential approximability of solutions to PDEs by neural networks. This topic has attracted significant interest in the past few years and a key question, as noted by the authors, is to understand which class of PDEs give rise to solutions that can be efficiently represented by a neural network without the curse of dimensionality, i.e. when the network size doesn't grow exponentially with the spatial dimension. The authors expand upon the previous works on linear PDEs by consider nonlinear variational PDEs of the form $-\\nabla\\cdot(\\nabla L(\\nabla u))=0$ and show that the solution has low complexity when $L$ does. Here, the complexity is formulated in terms of the Barron norm of the solution, which is a standard tool in the deep learning and PDE literature.", "summary_of_the_review": "The authors generalize existing results on the approximation of solutions to elliptic PDEs by neural networks to nonlinear variational PDEs. The paper is well-written and a good addition to the literature on this topic but would benefit from a re-organization of the structure and clarification of the novelty of the proofs compared to prior works. In addition, the authors show that solutions to the nonlinear PDEs have low-complexity but do not continue to derive a result on the size of the neural networks needed to approximate them. I would recommend the paper for publication after a revision addressing the major and minor comments listed above.", "strength_and_weaknesses": "### Major comments\n\n1. The paper is well-written and nice to read. However, its current organization makes Section 2 not really rigorous (essentially because some notions are introduced later in Section 4). Additionally, the literature review is splitted in two parts between the introduction in Section 1 and related works in Section 3. I suggest a re-organization of Section 1-4 to merge all the literature review into Section 1 by adding:\n  - Subsection 1.1 Related works, which includes all the literature review.\n  - Subsection 1.2 Contributions, summarizing the essential contributions informally like Theorem 1 but without the notations and definitions introduced later.\n2. The authors show the existence of solutions to variational PDEs of the form of (1) with small Barron norm, motivated by a result connecting Barron norm and approximability by a neural network. They also mention in the conclusion that an interesting question is the number of parameters by a neural network like Marwah et al. (2021). However, the main question considered in the introduction (and most interesting due to its connection with the practical works on solving PDEs with neural networks) concern the representability of solutions to PDEs by a small neural network. While the authors show a small complexity of the solution, it would be much more interesting (and impactful) to state a theorem in terms of neural network size after proving Theorem 4.\n3. The authors vaguely mention in Section 1 the type of problems modeled by the nonlinear PDEs they consider. It would be good for the clarity of the paper to expand on this by giving precise examples with explicit functions $L$.\n4. Would the approach considered by the authors naturally extend to time-dependent PDEs like nonlinear parabolic equations of the form $du/dt - div(grad L(grad u))=0$?\n5. The proof structure seems very similar to (Marwhah et al., 2021; Chen et al., 2021), could the authors add some clarifications on the novelty of the proof techniques with respect to these prior works due to the nonlinear class of PDEs considered?\n\n### Minor comments\n\n1. First line of Section 2, $\\Omega \\subset R^d$, the Poincare constant is not defined. I would write \"is such that the Poincare constant Cp satisfies $Cp\\geq 1$\" and add a footnote defining Cp. [I understand it is defined later in Theorem 2, which is why I suggest a re-organization of the paper].\n2. Footnote 2 requires a dot at the end of the sentence.\n3. Definition 1, the function space for u is not defined, what is meant by \"L is smooth\"? In which class of functions does L belong to?\n4. Definition 2, the max should be over $1\\leq i\\leq d$.\n5. Theorem 2, add a reference for Poincare inequality?\n6. Definition 4 should be \"Let F be the set...\" or \"We define F to be the set...\".\n7. Definition 5, \"Let Gamma be a set...\".\n8. Theorem 3, would this work for any non-polynomial activation function like the assumption of the universal approximation theorem?\n9. Could you comment on the difference with the paper by De Ryck, Jagtap, and Mishra \"Error estimates for physics informed neural networks approximating the Navier-Stokes equations\", which is analyzing solutions to the Navier-Stokes equations?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Xy3jBMwLbn", "reviewer": "Reviewer_AUeL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript studies the approximation properties of solutions of variational problems with Barron functions. In particular, this includes solutions of nonlinear PDEs like the p Laplace equation. It follows recent works on approximation properties for solutions of linear PDEs and extends this analysis to general variational problems. More precisely, it considers a preconditioned gradient descent of the variational energy in the Sobolev space $H^1_0$, which converges linearly, and bounds the Barron norm of the iterates, which relies on an assumption on the variational energy. Overall, this provides an estimate on the Barron norm required for the approximation of the solution of the variational problem up to a given accuracy. By standard arguments, this also implies upper bounds on the complexity of shallow networks with sigmoidal or ReLU activation for these target class and provides way of analysing the dependence of the convergence properties in dependence of the dimension of the problem.", "summary_of_the_review": "The paper addresses the timely and important question of the approximation properties of neural network related function classes of solutions of variational problems. I believe that the extension to general variational problems is a valuable contribution. That being said, I believe that the manuscript in its form is not ready for publication. My main concerns are the following:\n\n* *Implications of results:* The actual implications are not well developed. In particular, it would be important to elaborate the consequences of Theorem 4. In particular, it is not very clear, whether Theorem is able to establish estimates for $\\varepsilon\\to0$. Interesting implications include a precise formulation of the Barron norm required for $\\varepsilon$ approximation of a solution of a variational problem and complexity bounds for neural networks for $\\varepsilon$ approximation. This would could greatly improve the insights of the results and would make the results more applicable for future works. I know that to some extend this is done in Remarks 1 and 2, but it is not clear to me, whether this holds for arbitrary $\\epsilon$ and if so how to proof this.\n* *Inhomogeneous problems:* Can you also treat inhomogeneous variational problems? This seems to be crucial for the following reason: Consider the Laplace equation, then solution of the homogeneous problem are harmonic hence smooth and can be approximated at a dimension independent rate, where the constants might depend on the dimension (follows from standard arguments like Yarotsky).\n* *Clarity and quality of writing:* Unfortunately, the paper is currently not very well written and exhibits a number of spelling and grammatical mistakes as well as unintuitive formulations, which make reading less smooth and can lead to some confusion. I attached a list of specific observations below.\n\nSpecific questions:\n* Can you provide a theorem that bounds the Barron norm and complexity of a network required for $\\epsilon$ approximation? \n* Why do you state your results in the abstract and introduction for $L^2$ and not for $H^1$ approximation?\n* In Assumption 1: Is this supposed to hold for some $\\epsilon_L<\\lambda$? Then I would formulate it that the assumption is that $\\epsilon_L := \\sup \\dots < \\lambda$. \n* In Theorem 4: it is not clear to me what „considering a constant $R \\le \\dots$ “ means. Can this be chosen? If so, can it be set to 0? If it can not be chosen, then I would maybe say „$R = \\dots$“.\n* Can you treat non-homogeneous problems? \n* What is the benefit of working with $\\tilde L$ and not with $L$?\n\nSpecific comments:\n* The formulation „neurally simulating (preconditioned) gradient descent“ is confusing to me. Although, it sounds very fancy, I do not understand what it means, since form my understanding you use this for preconditioned gradient descent in a Sobolev space without having a direct connection to neural networks nor to simulation of something. When using such a terminology it should be better explained.\n* In the introduction, Sirignano&Spiliopoulos (2018) is given as a reference for a work minimizing the variational formulation of a PDE. However, they minimize the residual energy, which is fundamentally different to the variational energy.\n* It would make sense in Lemma 1 to say that (3) is to be understood in the weak sense.\n* In Theorem 4: I wonder whether $\\tilde\\epsilon$ is a good symbol, since epsilons are usually going to zero, which this quantity is not. \n* You claim to „subsume and substantially generalize“ prior works. In general, I would be very careful with judging the impact of any work, but this claim seems not entirely correct. From what I can see you extend their analysis to general variational problems, but only study homogeneous equations. Hence, prior results on linear but inhomogeneous problems are not special cases of your results.\n* Further references of interest: E&Wojtowytsch (2022), Grohs&Herrmann (2022)\n* Regarding Yarotsky: Yarotsky provides an approximation rate of $O(N^{-r/d})$, where $r$ is the regularity of the target function; if the solutions of the PDEs are smooth, which is for example the case for harmonic functions, then approximation with a power law with exponent independent of the dimension is possible; hence I disagree with your statement that the networks must by exponentially large; further, I do not know what you mean by Yarotsky's results being non-parametric\n* It could be a consideration to include the appendix in the normal PDF rather than the supplement.\n\nThroughts regarding formatting and writing:\n* In the list of references, the titles should be case sensitive, for example „PDEs“ instead of „pdes“.\n* Try to avoid double brackets.\n* On page 1: it is stated that \"... is at least as expensive as the neural network required to represent it.\". I think this is not very clearly written, maybe it would be better to say \"In general the complexity of fitting a NN grows with its size\"?\n* Section 2: \n    * Should be $\\Omega\\subset\\mathbb R^d$ and $\\partial\\Omega$\n    * Generally better to use $u\\colon\\Omega\\to\\mathbb R$ instead of $u:\\Omega\\to\\mathbb R$ (\"\\colon\" instead of \":\")\n    * Below Definition 1 it would be nice to add in what space the problem is well posed\n    * In Lemma 1 it should be \"the minimizer $u^\\star$\n* Section 3: \n    * In the first sentence it should b \"... been a growing line of works that utilize neural\"\n    * Regarding \"This is a great and promising direction...\" I would in general shy away from judging the quality of other research in a research paper\n    * It should be \"equation and Lu & Lu (2021) extend their analysis\"\n    * It should be \"Schrödinger\" (\\ \"{o})\n    * When using a bibtex style that gives the authors names, I would not write \"in [authors names]\" but simply \"[authors names]\" (e.g., \"In Marwah et al. (2021)\")\n* Section 4: \n    * I would try to not start a sentence with a symbol (in your case $C^\\infty$)\n    * In Definition 3 it should be $g\\in L^2$ and I would rather write $\\nabla g\\in L^2$\n    * In Theorem 3 I would introduce $\\sigma$ before the statement.\n* Section 6: \n    * \"neurally unfolding\": As said before, I do not understand what is meant with this; I think the description as preconditioned GD in a Sobolev space is perfectly clear.\n    * Lemma 4: there should be a full stop after \"Definition 3\"\n    * The reference to Equation 28: is this where it should refer to? If so, referencing to something burried in the appendix for a high level idea is not ideal.", "strength_and_weaknesses": "The paper addresses the timely and important question of the approximation properties of neural network related function classes of solutions of variational problems. I believe that the extension to general variational problems is a valuable contribution. Main weaknesses of the manuscript include:\n* The work follows existing works on linear problems very closely. \n* The consequences of the main theorem are not well elaborated. For example, no theorem bounding the Barron norm or the complexity of a shallow network required for eps approximation is given. Although these steps seem straight forward, they would be important to be carried out, especially in light of Theorem 4 the error does not decay to zero as discussed by the authors. \n* The paper is currently not very well written and exhibits a lot of spelling and grammatical mistakes, which and should be addressed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "RQCYZJTiE45", "reviewer": "Reviewer_4yVB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is fundamentally a PDE analysis result involving the regularity of solutions to (nonlinear) variational problems with respect to the Barron norm when the function involved in the variational problem (to within an epsilon approximation in the H^1 seminorm) satisfies a Holder-like condition (with respect to the Barron norm).  The proof is via analysis of an iterative approximation scheme (preconditioned gradient descent).  This is relevant for variational approximation schemes using neural networks as a trial space, because bounds on the Barron norm of the solution correspond to bounds on the size of a two-layer neural network needed to represent the solution to a given accuracy.", "summary_of_the_review": "This is a nice result, and I would like to see it published somewhere, though it would not be obvious to me that an ML conference is the venue where it would have the best reception.  Complexity of approximation is important for understanding numerical schemes for solving PDEs generally, and there are relatively few such results around approximation schemes that involve neural networks.", "strength_and_weaknesses": "The paper is a nice result, clearly written.  The proof tools are standard for this type of PDE analysis, but they are well used here.\n\nThe result does not tell us anything about deeper neural networks, nor does it seem to tell us how to find the optimal approximating networks that satisfy the bound -- we just know that such networks must exist.\n\nI would like to see this work out in the literature somewhere, and it is certainly relevant to people trying to build neural network schemes for the solution of PDEs arising in science.  But I admit that if I was looking for a result of this form, I would probably usually start in a journal on approximation theory or PDE analysis rather than in ICLR or similar conference proceedings.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "LpUs6x2eXl", "reviewer": "Reviewer_23dt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the representational power of neural networks for approximating solutions to nonlinear PDEs. They focus on a specific class of nonlinear elliptic variational PDEs. They show that a 2-layer neural network can be used to approximate the solution. Thus showing neural networks can evade the curse of dimensionality. This is a theoretical paper; perhaps the application aspect can be strengthened.", "summary_of_the_review": "This paper presents a theoretical work on neural networks for nonlinear elliptic equations. I am not a mathematics major, and the mathematical proof is difficult for me to understand. It would be more solid to show some applications.", "strength_and_weaknesses": "Strength\n\nWhile most previous approaches have studied linear equations, this paper considers a nonlinear family of PDEs and studies nonlinear variational PDEs. \n\nThe results go beyond the typical non-parametric bounds on the size of an approximator network that can be easily shown by classical regularity results of the solution to the nonlinear variational PDEs and universal approximation results.\n\nWeaknesses\n\nThere is no obvious application for this paper.\n\nThis paper considers only the elliptic equations, but the equations that evolve over time are also important.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject