Refined Generalization Analysis of the Deep Ritz Method and Physics-Informed Neural Networks

## Abstract

Abstract. In this paper, we present refined generalization bounds for the Deep Ritz Method (DRM) and Physics-Informed Neural Networks (PINNs). For the DRM, we focus on two prototype elliptic PDEs: Poisson equation and static Schrödinger equation on the d 𝑑 d italic_d -dimensional unit hypercube with the Neumann boundary condition. And sharper generalization bounds are derived based on the localization techniques under the assumptions that the exact solutions of the PDEs lie in the Barron spaces or the general Sobolev spaces. For the PINNs, we investigate the general linear second elliptic PDEs with Dirichlet boundary condition via the local Rademacher complexity in the multi-task learning.

## 1. Introduction

Partial Differential Equations (PDEs) play a pivotal role in modeling of phenomena across physics, biology and engineering. However, solving PDEs numerically has been a longstanding challenge in scientific computing. Classical numerical methods like finite difference, finite element, finite volume and spectral methods may suffer from the curse of dimensionality when dealing with high-dimensional PDEs. Recent years, the remarkable successes of deep learning in diverse fields like computer vision, natural language processing and reinforcement learning have sparked interest in applying machine learning techniques to solve various types of PDEs. In fact, the idea of using machine learning to solve PDEs dates back to the last century [], but it has recently gained renewed attention due to the remarkable progress in hardware technology and algorithm development.

There are numerous methods proposed to solve PDEs using neural networks. One popular method, known as PINNs [], utilizes neural networks to represent the solution and enforces the neural network to satisfy the PDE constraints, initial conditions and boundary conditions by encoding these conditions into the loss function. The flexibility and scalability of the PINNs make it a widely used framework for addressing PDE-related problems. The Deep Ritz Method [] proposes incorporating the variational formulation into training the neural networks due to the widespread use of the variational formulation in traditional methods. In comparison to PINNs, the form of DRM has a lower derivative order, but the fact that not all PDEs have variational forms limits its applications. Both methods hinge on the approximation ability of the deep neural networks.

The approximation power of feed-forward neural networks (FNNs) with diverse activation functions has been studied for different types of functions, including smooth functions [], continuous functions [], Sobolev functions [, , , ], Barron functions []. It was proven in the last century that a sufficiently large neural network can approximate a target function in a certain function class with any given tolerance. Specifically, it has been shown in []
that the two-layer neural network with ReLU activation function is a universal approximator for continuous functions. More recently, specific approximate rate of neural networks has been shown for different function classes in terms of depth and width. [] showed that a ReLU FNN with width $\mathcal{O}(N\log N)$ and depth $\mathcal{O}(L\log L)$ can achieve approximation rate $\mathcal{O}(N^{-2s/d}L^{-2s/d})$ for the function class $C^{s}([0,1]^{d})$ in the $L^{\infty}$ norm, which is nearly minimax. For the continuous functions, [] demonstrated that ReLU neural networks of width $\mathcal{O}(N)$ and depth $\mathcal{O}(L)$ can approximate an arbitrary continuous function $f$ on $[0,1]^{d}$ with an optimal approximation error $\mathcal{O}(\omega_{f}((N^{2}L^{2}\log N)^{-1/d}))$ in terms of $N$ and $L$, where $\omega_{f}$ is the modulus of continuity of $f\in C([0,1]^{d})$. In the context of applying neural networks to solve PDEs, the focus shifts to the approximation rates in Sobolev norms. [] utilized multivariate spline to derive the required depth, width, and sparsity of a $\text{ReLU}^{2}$ deep neural network to approximate any Hölder smooth function in Hölder norms with the given approximation error. And the weights of the neural network are also controlled. [] derived the nearly optimal approximation results of deep neural networks in Sobolev spaces with Sobolev norms. Specifically, deep ReLU neural networks with width $\mathcal{O}(N\log N)$ and depth $\mathcal{O}(L\log L)$ can achieve approximation rate $\mathcal{O}(N^{-2(n-1)/d}L^{-2(n-1)/d})$ for functions in $W^{n,\infty}((0,1)^{d})$ with $W^{1,\infty}$ norm. For higher order approximation in Sobolev spaces, [] introduced deep super ReLU networks for approximating functions in Sobolev spaces under Sobolev norms $W^{m,p}$ for $m\in\mathbb{N}$ with $m\geq 2$. The optimality was also established by estimating the VC-dimension of the function class consisting of higher-order derivatives of deep super ReLU networks.

In this work, we focus on the DRM and PINNs, aiming to derive sharper generalization bounds. Compared to [, ], the localized analysis utilized in this paper leads to improved generalization bounds. We believe that this study provides a unified framework for deriving generalization bounds for methods that solve PDEs using machine learning.

### 1.1. Related works

Deep learning based PDE solvers: Solving high-dimensional PDEs
has been a long-standing challenge in scientific computing due to the curse of dimensionality. Inspired by the ability and flexibility of neural networks for representing high dimensional functions, numerous studies have focused on developing efficient deep learning-based PDE solvers. In recent years, the PINNs have emerged as a flexible framework for addressing problems related to PDEs and has achieved impressive results in numerous tasks. Despite their success, there are areas where further improvements can be made, such as developing better optimization targets [] and neural network architectures [, ]. Inspired by the use of weak formulation in traditional solvers, [] proposed to solve the weak formulations of PDEs via an adversarial
network and the DRM [] trains a neural network to minimize the variational formulations of PDEs.

Fast rates in machine learning: In statistical learning, the excess risk is expressed as the form $(\frac{\text{COMP}_{n}(\mathcal{F})}{n})^{\alpha}$, where $n$ is the sample size, $\text{COMP}_{n}(\mathcal{F})$ measures the complexity of the function class $\mathcal{F}$ and $\alpha\in[\frac{1}{2},1]$ represents the learning rate. The slow learning rate $\frac{1}{\sqrt{n}}\ (\alpha=\frac{1}{2})$ can be easily derived by invoking Rademacher complexity [], but achieving the fast rate $\frac{1}{n}\ (\alpha=1)$ is much more challenging. Based on localization techniques, the local Rademacher complexity [, ] was introduced to statistical learning and has become a popular tool to derive fast rates. It has been successfully applied across a variety of tasks, like clustering [], learning kernels [], multi-task learning [], empirical variance minimization [], among others. Variants of Rademacher complexity, such as shifted Rademacher complexity [] and offset Rademacher complexity [], also offer a potential direction for achieving the fast rates [, , ]. In this paper, our results are based on the localized analysis in [, , ].

Generalization bounds for machine learning based PDE solvers:
Based on the probabilistic space filling arguments [], [] demonstrated the consistency of PINNs for the linear second-order elliptic and parabolic type PDEs. An abstract framework was introduced in [] and stability properties of the underlying PDEs were leveraged to derive upper bounds on the generalization error of PINNs. Following similar methods widely used in machine learning for deriving generalization bounds, the convergence rate of PINNs was derived in [] by decomposing the error and estimating related Rademacher complexity. For the DRM, [] showed the generalization error bounds of two-layer neural networks for solving the Poisson equation and static Schrödinger equation on the $d$-dimensional unit hypercube, but in expectation and with the slow rates. When solutions of the PDEs fall in general sobolev spaces, [] established nonasymptotic convergence rate for DRM using a method similar to []. The most relevant work to ours is [], which used peeling methods to derive sharper generalization bounds of the DRM and PINNs for the Schrödinger equation on a
hypercube with zero Dirichlet boundary condition. However, [] assumed that the function class of neural networks is a subset of $H_{0}^{1}$, which is challenging to achieve. For the DRM, the peeling method in [] cannot be applied to derive the generalization error of the Poisson equation, as in this scenario, the population loss isn’t the expectation of the empirical loss. For the PINNs, [] required the strongly convex structure, but our approach does not need this condition.

### 1.2. Contributions

- •
For the aspect of approximation via neural networks, we show that the functions in $\mathcal{B}^{2}(\Omega)$ can be well approximated in the $H^{1}$ norm by two-layer ReLU neural networks with controlled weights, and similar results are also presented for functions in $\mathcal{B}^{3}(\Omega)$. When compared to the results in [], our approximation rate is faster and the Barron space in our setting is larger than the spectral Barron space in []. As comparing with other approximation results for Barron functions [, ], the constants in our results are independent of the dimension.
- •
For the DRM, we derive sharper generalization bounds for the Poisson equation and Schrödinger equation with Neumann boundary condition, regardless of whether the solutions fall in Barron spaces or Sobolev spaces. Our methods rely on strongly convex property of the variational form and the localized analysis of [, ]. However, these methods cannot be applied directly, as for the Poisson euqation, the expectation of empirical loss is not equal to the variational formulation. Additionally, for the static Schrödinger equation, the strongly convex property cannot be simply regarded as the Bernstein condition in [], as the solutions of the PDEs often do not belong to the function class of neural networks in our setting.
- •
For the PINNs, we can regard this framework as a scenario within multi-task learning (MTL). At this time, there are two key points: one is that the loss functions are non-negative and the other one is that a non-exact oracle inequality suffices. To achieve our goal, we extend the entropy method to derive a Talagrand-type concentration ineuqality for MTL, which offers better constants than those provided by Theorem 1 in []. Consequently, similar results to those in single-task setting can be established, yielding a non-exact oracle inequality tailored for PINNs. Unlike [], which required the strongly convex property, our approach does not impose this requirement. Although we have only presented results for the linear second-order elliptic equations with Dirichlet boundary conditions, our method can serve as a framework for PINNs for a wide range of PDEs, as well as other methods that share similar forms with PINNs.

### 1.3. Notation

For $x\in\mathbb{R}^{d}$, $|x|_{p}$ denotes its $p$-norm and we use $|x|$ as shorthand for $|x|_{2}$. We denote the inner product of vectors $x,y\in\mathbb{R}^{d}$ by $x\cdot y$. For the $d$-dimensional ball with radius $r$ in the $p$-norm and the boundary of this ball, we denote them by $B_{p}^{d}(r)$ and $\partial B_{p}^{d}(r)$ respectively. For a set $\mathcal{F}$ that is a subset of a metric space with metric $d$, we use $\mathcal{N}(\mathcal{F},d,\epsilon)$ to denote its covering number with given radius $\epsilon$ and the metric $d$. For given probability measure $P$ and a sequence of random variables $\{X_{i}\}_{i=1}^{n}$ distributed according to $P$, we denote the empirical measure of $P$ by $P_{n}$, i.e. $P_{n}=\frac{1}{n}\sum\limits_{i=1}^{n}\delta_{X_{i}}$. For the activation functions, we write $\sigma_{k}(x)$ for the $\text{ReLU}^{k}$ activation function, i.e., $\sigma_{k}(x):=[\max(0,x)]^{k}$. And we use $\sigma$ for $\sigma_{1}$ for simplicity. Given a domain $\Omega\subset\mathbb{R}^{d}$, we denote $|\Omega|$ and $|\partial\Omega|$ the measure of $\Omega$ and its boundary $\partial\Omega$, respectively.

## 2. Deep Ritz Method

### 2.1. Set Up

Let $\Omega=(0,1)^{d}$ be the unit hypercube on $\mathbb{R}^{d}$ and $\partial\Omega$ be the boundary of $\Omega$. We consider the Poisson equation and static Schrödinger equation on $\Omega$ with Neumann boundary condition.

Poisson equation:

$$ $\displaystyle-\Delta u=f\ in\ \Omega,\ \ \frac{\partial u}{\partial\nu}=0\ on \ \partial\Omega.$ $$

Static Schrödinger equation:

$$ $\displaystyle-\Delta u+Vu=f\ in\ \Omega,\ \ \frac{\partial u}{\partial\nu}=0\ on\ \partial\Omega.$ $$

In this section, we follow the framework established in [], which characterizes the solutions through variational formulations. For completeness, the detailed results are presented as follows.

###### Proposition 1 (Proposition 1 in [ 34 ] ) .

(1) Assume that $f\in L^{2}(\Omega)$ with $\int_{\Omega}fdx=0$. Then there exists a unique weak solution $u_{P}^{*}\in H_{*}^{1}(\Omega):=\{u\in H^{1}(\Omega):\int_{\Omega}udx=0\}$ to the Poisson equation. Moreover, we have that

$$ $u_{P}^{*}=\mathop{\arg\min}\limits_{u\in H^{1}(\Omega)}\mathcal{E}_{P}(u):= \int_{\Omega}|\nabla u|^{2}dx+(\int_{\Omega}udx)^{2}-2\int_{\Omega}fudx,$ $$

and that for any $u\in H^{1}(\Omega)$,

$$ $\mathcal{E}_{P}(u)-\mathcal{E}_{P}(u_{P}^{*})\leq\|u-u_{P}^{*}\|_{H^{1}(\Omega )}^{2}\leq\max\{2c_{P}+1,2\}(\mathcal{E}_{P}(u)-\mathcal{E}_{P}(u_{P}^{*})),$ $$

where $c_{P}$ is the Poincaré constant on the domain $\Omega$.

(2) Assume that $f,V\in L^{\infty}(\Omega)$ and that $0<V_{min}\leq V(x)\leq V_{max}<\infty$ for all $x\in\Omega$ and some constants $V_{min}$ and $V_{max}$. Then there exists a unique weak solution $u_{S}^{*}\in H^{1}(\Omega)$ to the static Schrödinger equation. Moreover, we have that

$$ $u_{S}^{*}=\mathop{\arg\min}\limits_{u\in H^{1}(\Omega)}\mathcal{E}_{S}(u):= \int_{\Omega}|\nabla u|^{2}+V|u|^{2}dx-2\int_{\Omega}fudx,$ $$

and that for any $u\in H^{1}(\Omega)$

$$ $\frac{1}{\max(1,V_{max})}(\mathcal{E}_{S}(u)-\mathcal{E}_{S}(u_{S}^{*}))\leq\| u-u_{S}^{*}\|_{H^{1}(\Omega)}^{2}\leq\frac{1}{\min(1,V_{min})}(\mathcal{E}_{S} (u)-\mathcal{E}_{S}(u_{S}^{*})).$ $$

Throughout the paper, we assume that $f\in L^{\infty}(\Omega)$ and $V\in L^{\infty}(\Omega)$ with $0<V_{min}\leq V(x)\leq V_{max}<\infty$. The boundedness is essential in our methods for deriving fast rates and it also leads to the strongly convex property in Proposition 1(2). There are also some methods for deriving generalization error beyond boundedness, as discussed in [, , ]. However, these approaches often require additional assumptions, such as specific properties of the data distributions or function classes, which can be difficult to verify in practice.

The core concept of DRM involves substituting the function class of neural networks for Sobolev spaces and then training the neural networks to minimize the variational formulations. Subsequently, we can employ Monte-Carlo method to compute the high-dimensional integrals, as traditional quadrature methods are constrained by the curse of dimensionality in this context.

Let $\{X_{i}\}_{i=1}^{n}$ be an i.i.d. sequence of random variables distributed uniformly in $\Omega$. Note that in our setting, the volume of $\Omega$ is 1, thus the empirical losses can be written directly as

$$ $\mathcal{E}_{n,P}(u)=\frac{1}{n}\sum\limits_{i=1}^{n}(|\nabla u(X_{i})|^{2}-2f (X_{i})u(X_{i}))+(\frac{1}{n}\sum\limits_{i=1}^{n}u(X_{i}))^{2}$ $$

and

$$ $\mathcal{E}_{n,S}(u)=\frac{1}{n}\sum\limits_{i=1}^{n}(|\nabla u(X_{i})|^{2}+V( X_{i})|u(X_{i})|^{2}-2f(X_{i})u(X_{i})),$ $$

where we write $\mathcal{E}_{n,P}$ and $\mathcal{E}_{n,S}$ for the empirical losses of the Poisson equation and static Schrödinger equation respectively.

Note that the expectation of $\mathcal{E}_{n,P}(u)$ is not equal to $\mathcal{E}_{P}(u)$, which limits most methods for deriving a fast rate for the Poisson equation.

###### Proposition 1 (Proposition 1 in [ 34 ] ) .

### 2.2. main results

The aim of this section is to establish a framework for deriving improved generalization bounds for the DRM. In the setting where the solutions lie in the Barron space $\mathcal{B}^{2}(\Omega)$, we demonstrate that the generalization error between the empirical solutions from minimizing the empirical losses and the exact solutions grows polynomially with the underlying dimension, enabling the DRM to overcome the curse of dimensionality in this context. Furthermore, when the solutions fall in the general Sobolev spaces, we provide tight generalization bounds through the localization analysis.

We begin by presenting the definition of the Barron space, as introduced in [].

$$ $\mathcal{B}^{s}(\Omega):=\{f:\Omega\rightarrow\mathbb{C}:\|f\|_{\mathcal{B}^{s }(\Omega)}:=\inf\limits_{f_{e}|\Omega=f}\int_{\mathbb{R}^{d}}(1+|\omega|_{1})^ {s}|\hat{f}_{e}(\omega)|d\omega<\infty\},$ $$

where the infimum is over extensions $f_{e}\in L^{1}(\mathbb{R}^{d})$ and $\hat{f}_{e}$ is the Fourier transform of ${f}_{e}$. Note that we choose $1$-norm in the definition just for simplicity.

There are also several different definitions of Barron space [] and the relationships between them have been studied in []. The most important property of functions in the Barron space is that those functions can be efficiently approximated by two-layer neural networks without the curse of dimensionality. It has been shown in [] that two-layer neural networks with sigmoidal activation functions can achieve approximation rate $\mathcal{O}(1/\sqrt{n})$ under the $L^{2}$ norm, where $n$ is the number of neurons. And the results have been extended to the Sobolev norms [, ]. However, some constants in these extensions implicitly depend on the dimension and there is a possibility that the weights may be unbounded. To address these concerns, we demonstrate the approximation results for functions in the Barron space under the $H^{1}$ norm, ensuring that the constants are independent of the dimension and the weights are controlled. Additionally, for completeness, the approximation result in $W^{k,\infty}(\Omega)$ with $W^{1,\infty}$ norm is also presented, which was originally derived in [].

###### Proposition 2 (Approximation results in the $H^{1}$ norm) .

(1)Barron sapce:
For any $f\in\mathcal{B}^{2}(\Omega)$, there exists a two-layer neural network $f_{m}\in\mathcal{F}_{m,1}(5\|f\|_{\mathcal{B}^{2}(\Omega)})$ such that

$$ ${\|f-f_{m}\|}_{H^{1}({\Omega})}\leq c\|f\|_{\mathcal{B}(\Omega)}m^{-(\frac{1}{ 2}+\frac{1}{3d})},$ $$

where $\mathcal{F}_{m,1}(B):=\{\sum\limits_{i=1}^{m}\gamma_{i}\sigma(\omega_{i}\cdot x
+t_{i}):\ |\omega_{i}|_{1}=1,t_{i}\in[-1,1],\sum\limits_{i=1}^{m}|\gamma_{i}|
\leq B\}$ for any positive constant $B$ and $c$ is a universal constant.

(2)Sobolev space: For any $f\in\mathcal{W}^{k,\infty}(\Omega)$ with $k\in\mathbb{N},\ k\geq 2$ and $\|f\|_{\mathcal{W}^{k,\infty}(\Omega)}\leq 1$, any $N,L\in\mathbb{N}_{+}$, there exists a ReLU neural network $\phi$ with the width $(34+d)2^{d}k^{d+1}(N+1)\log_{2}(8N)$ and depth $56d^{2}k^{2}(L+1)\log_{2}(4L)$ such that

$$ $\|f(x)-\phi(x)\|_{\mathcal{W}^{1,\infty}(\Omega)}\leq C(k,d)N^{-2(k-1)/d}L^{-2 (k-1)/d},$ $$

where $C(k,d)$ is the constant independent with $N,L$.

For the convenience of expression, we write $\Phi(N,L,B)$ for the function class of ReLU neural networks in Proposition 2(2) with the width $(34+d)2^{d}k^{d+1}(N+1)\log_{2}(8N)$, depth $56d^{2}k^{2}(L+1)\log_{2}(4L)$ and $W^{1,\infty}$ norm bounded by $B$ such that the approximation result in Proposition 2(2) holds for any $f\in\mathcal{W}^{k,\infty}(\Omega)$ with $\|f\|_{\mathcal{W}^{k,\infty}(\Omega)}\leq 1$.

The approximation result for $\mathcal{B}^{2}(\Omega)$ can also be extended to approximate functions in $\mathcal{B}^{k}(\Omega)$ with two-layer ${\text{ReLU}}^{k-1}$ neural networks and the approximation rate should be same as Proposition 2(1). However, the approximation rate cannot reflect the smoothness of the Barron space and the activation function. That is, as the smoothness index $s$ of the Barron space $\mathcal{B}^{s}(\Omega)$ and $k$ of the $\text{ReLU}^{k}$ function increase, we can’t obtain a improved approximation rate. [] has addressed this problem, but the constants in Theorem 3 in [] depend on the dimension implicitly and may depend exponentially on the dimension. Moreover, the weights of the two-layer neural networks cannot be controlled.

With the approximation results, we can derive the generalization error for the Poisson equation and the static Schrödinger equation through the localized analysis.

###### Theorem 1 (Generalization error for the Poisson equation) .

Let $u_{P}^{*}$ solve the Poisson equation and $u_{n,P}$ be the minimizer of the empirical loss $\mathcal{E}_{n,P}$ in the function class
$\mathcal{F}$.

(1) For $u_{P}^{*}\in\mathcal{B}^{2}(\Omega)$, taking $\mathcal{F}=\mathcal{F}_{m,1}(5\|u_{P}^{*}\|_{\mathcal{B}^{2}(\Omega)})$, then with probability as least $1-e^{-t}$

$$ $\mathcal{E}_{P}(u_{n,P})-\mathcal{E}_{P}(u_{P}^{*})\leq CM^{2}\log M(\frac{md \log n}{n}+(\frac{1}{m})^{1+\frac{2}{3d}}+\frac{t}{n}),$ $$

where $C$ is a universal constant and $M$ is the upper bound for $\|f\|_{L^{\infty}},\|u_{P}^{*}\|_{\mathcal{B}^{2}(\Omega)}.$

By taking $m=(\frac{n}{d})^{\frac{3d}{2(3d+1)}}$, we have

$$ $\mathcal{E}_{P}(u_{n,P})-\mathcal{E}_{P}(u_{P}^{*})\leq CM^{2}\log M((\frac{d} {n})^{\frac{3d+2}{2(3d+1)}}\log n+\frac{t}{n}).$ $$

(2) For $u_{P}^{*}\in\mathcal{W}^{k,\infty}(\Omega)$, taking $\mathcal{F}=\Phi(N,L,B\|u_{P}^{*}\|_{\mathcal{W}^{k,\infty}(\Omega)})$, then with probability at least $1-e^{-t}$

$$ $\mathcal{E}_{P}(u_{n,P})-\mathcal{E}_{P}(u_{P}^{*})\leq C(\frac{(NL)^{2}(\log N \log L)^{3}}{n}+(NL)^{-4(k-1)/d}+\frac{t}{n}),$ $$

where $n\geq C(NL)^{2}(\log N\log L)^{3}$ and $C$ is a constant independent of $N,L,n$.

By taking $N=L=n^{\frac{1}{4(d+2(k-1))}}$, we have

$$ $\mathcal{E}_{P}(u_{n,P}^{m})-\mathcal{E}_{P}(u_{P}^{*})\leq C(n^{-\frac{2k-2}{ d+2k-2}}(\log n)^{6}+\frac{t}{n}).$ $$

###### Theorem 2 (Generalization error for the static Schrödinger equation) .

Let $u_{P}^{*}$ solve the static Schrödinger and $u_{n,S}$ be the minimizer of the empirical loss $\mathcal{E}_{n,S}$ in the function class $\mathcal{F}$.

(1) For $u_{S}^{*}\in\mathcal{B}^{2}(\Omega)$, taking $\mathcal{F}=\mathcal{F}_{m,1}(5\|u_{S}^{*}\|_{\mathcal{B}^{2}(\Omega)})$, then with probability as least $1-e^{-t}$

$$ $\mathcal{E}_{S}(u_{n,S})-\mathcal{E}_{S}(u_{S}^{*})\leq CM^{2}(\frac{md\log n} {n}+(\frac{1}{m})^{1+\frac{2}{3d}}+\frac{t}{n}),$ $$

where $C$ is a universal constant and $M$ is the upper bound for $\|f\|_{L^{\infty}},\|u_{S}^{*}\|_{\mathcal{B}^{2}(\Omega)},\|V\|_{L^{\infty}}.$

By taking $m=(\frac{n}{d})^{\frac{3d}{2(3d+1)}}$, we have

$$ $\mathcal{E}_{S}(u_{n,S})-\mathcal{E}_{S}(u_{S}^{*})\leq CM^{2}((\frac{d}{n})^{ \frac{3d+2}{2(3d+1)}}\log n+\frac{t}{n}).$ $$

(2) For $u_{S}^{*}\in\mathcal{W}^{k,\infty}(\Omega)$, taking $\mathcal{F}=\Phi(N,L,B\|u_{S}^{*}\|_{\mathcal{W}^{k,\infty}(\Omega)})$, then with probability at least $1-e^{-t}$

$$ $\mathcal{E}_{S}(u_{n,S})-\mathcal{E}_{S}(u_{S}^{*})\leq C(\frac{(NL)^{2}(\log N \log L)^{3}}{n}+(NL)^{-4(k-1)/d}+\frac{t}{n}),$ $$

where $n\geq C(NL)^{2}(\log N\log L)^{3}$ and $C$ is a constant independent of $N,L,n$.

By taking $N=L=n^{\frac{1}{4(d+2(k-1))}}$, we have

$$ $\mathcal{E}_{S}(u_{n,S})-\mathcal{E}_{S}(u_{S}^{*})\leq C(n^{-\frac{2k-2}{d+2k -2}}(\log n)^{6}+\frac{t}{n}).$ $$

The generalization error presented in Theorem 1 and Theorem 2 shares a similar form with the excess risk in machine learning. However, our goal is to measure the differences between the empirical solutions and the true solutions using an appropriate Sobolev norm. This can be achieved directly through Proposition 1. Under the conditions outlined in Theorem 1 and Theorem 2, we can derive the generalization error for both the Poisson equation and the static Schrödinger equation under the $H^{1}$ norm.

###### Corollary 1 (Generalization error under the $H^{1}$ norm) .

(1) For the Poisson equation, if $u_{P}^{*}\in\mathcal{B}^{2}(\Omega)$, then

$$ $\|u_{n,P}-u_{P}^{*}\|_{H^{1}(\Omega)}^{2}\leq CM^{2}\log M((\frac{d}{n})^{ \frac{3d+2}{2(3d+1)}}\log n+\frac{t}{n});$ $$

if $u_{P}^{*}\in\mathcal{W}^{k,\infty}(\Omega)$, then

$$ $\|u_{n,P}-u_{P}^{*}\|_{H^{1}(\Omega)}^{2}\leq C(n^{-\frac{2k-2}{d+2k-2}}(\log n )^{6}+\frac{t}{n}).$ $$

(2) For the static Schrödinger equation, if $u_{S}^{*}\in\mathcal{B}^{2}(\Omega)$, then

$$ $\|u_{n,S}-u_{S}^{*}\|_{H^{1}(\Omega)}^{2}\leq CM^{2}((\frac{d}{n})^{\frac{3d+2 }{2(3d+1)}}\log n+\frac{t}{n});$ $$

if $u_{S}^{*}\in\mathcal{W}^{k,\infty}(\Omega)$, then

$$ $\|u_{n,S}-u_{S}^{*}\|_{H^{1}(\Omega)}^{2}\leq C(n^{-\frac{2k-2}{d+2k-2}}(\log n )^{6}+\frac{t}{n}).$ $$

Although the generalization bounds for the Poisson equation and the static Schrödinger equation appear to be identical in our setting when omitting the logarithmic term involving $M$, there is a key distinction between them. As pointed out in the proof, the generalization error for the Poisson equation also depends on the Poincaré constant associated with the domain
$\Omega$. In our specific setting, $\Omega=(0,1)^{d}$, the Poincaré constant is independent of the dimension and is a universal constant. However, in other scenarios, it may vary with the dimension.

###### Proposition 2 (Approximation results in the H 1 superscript 𝐻 1 H^{1} italic_H start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT norm) .

###### Theorem 1 (Generalization error for the Poisson equation) .

###### Theorem 2 (Generalization error for the static Schrödinger equation) .

###### Corollary 1 (Generalization error under the H 1 superscript 𝐻 1 H^{1} italic_H start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT norm) .

## 3. Physics-Informed Neural Networks

### 3.1. Set Up

In this section, we will consider the following linear second order elliptic equation with Dirichlet boundary condition.

$$ $\left\{\begin{aligned} -\sum\limits_{i,j=1}^{d}a_{ij}\partial_{ij}u+\sum \limits_{i=1}^{d}b_{i}\partial_{i}u+cu=f&,&in\ \Omega,\\ u=g&,&on\ \partial\Omega,\end{aligned}\right.$ $$

where $a_{ij}\in C(\bar{\Omega})$, $b_{i},c,f\in L^{\infty}(\Omega)$, $g\in L^{\infty}(\partial\Omega)$ and $\Omega\subset(0,1)^{d}$ is a open bounded domain with properly smooth boundary. Additionally, we assume that the strictly elliptic condition holds, i.e., there exists a constant $\lambda>0$ such that $\sum\limits_{i,j=1}^{d}a_{ij}\xi_{i}\xi_{j}\geq\lambda|\xi|^{2}$ for $\forall\ x\in\Omega,\xi\in\mathbb{R}^{d}$.

In the framework of PINNs, we train the neural network $u$ with the following loss function.

$$ $\mathcal{L}(u):=\int_{\Omega}(-\sum\limits_{i,j=1}^{d}a_{ij}(x)\partial_{ij}u( x)+\sum\limits_{i=1}^{d}b_{i}(x)\partial_{i}u(x)+c(x)u(x)-f(x))^{2}dx+\int_{ \partial\Omega}(u(y)-g(y))^{2}dy.$ $$

By employing the Monte Carlo method, the empirical version of $\mathcal{L}$ can be written as

$$ $\displaystyle\mathcal{L}_{N}(u)$ $\displaystyle:=\frac{|\Omega|}{N_{1}}\sum\limits_{k=1}^{N_{1}}(-\sum\limits_{i ,j=1}^{d}a_{ij}(X_{k})\partial_{ij}u(X_{k})+\sum\limits_{i=1}^{d}b_{i}(X_{k}) \partial_{i}u(X_{k})+c(X_{k})u(X_{k})-f(X_{k}))^{2}$ $\displaystyle\qquad+\frac{|\partial\Omega|}{N_{2}}\sum\limits_{k=1}^{N_{2}}(u( Y_{k})-g(Y_{k}))^{2},$ $$

where $N=(N_{1},N_{2})$, $\{X_{k}\}_{k=1}^{N_{1}}$ and $\{Y_{k}\}_{k=1}^{N_{2}}$ are i.i.d. random variables distributed according to the uniform distribution $U(\Omega)$ on $\Omega$ and $U(\partial\Omega)$ on $\partial\Omega$, respectively.

Given the empirical loss $\mathcal{L}_{N}$, the empirical minimization algorithm aims to seek $u_{N}$ which minimizes $\mathcal{L}_{N}$, that is:

$$ $u_{N}\in\mathop{\arg\min}\limits_{u\in\mathcal{F}}\mathcal{L}_{N}(u),$ $$

where $\mathcal{F}$ is a parameterized hypothesis function class.

### 3.2. Main Results

We begin by presenting the approximation results in the $H^{2}$ norm.

###### Proposition 3 (Approximation results in the $H^{2}$ norm) .

(1)Barron space: For any $f\in\mathcal{B}^{3}(\Omega)$, there exists a two-layer neural network $f_{m}\in\mathcal{F}_{m,2}(c\|f\|_{\mathcal{B}^{3}(\Omega)})$ such that

$$ ${\|f-f_{m}\|}_{H^{2}({\Omega})}\leq c\|f\|_{\mathcal{B}^{3}(\Omega)}m^{-(\frac {1}{2}+\frac{1}{3d})},$ $$

where $\mathcal{F}_{m,2}(B):=\{\sum\limits_{i=1}^{m}\gamma_{i}\sigma_{2}(\omega_{i}
\cdot x+t_{i}):\ |\omega_{i}|_{1}=1,t_{i}\in[-1,1],\sum\limits_{i=1}^{m}|
\gamma_{i}|\leq B.\}$ for any positive constant $B$ and $c$ is a universal constant.

(2)Sobolev space: For any $f\in\mathcal{W}^{k,\infty}(\Omega)$ with $k>3$ and any integer $K\geq 2$, there exists some sparse $\text{ReLU}^{3}$ neural network $\phi\in\Phi(L,W,S,B;H)$ with $L=\mathcal{O}(1),W=\mathcal{O}(K^{d}),S=\mathcal{O}(K^{d}),B=1,H=\mathcal{O}(1)$, such that

$$ $\|f(x)-\phi(x)\|_{H^{2}(\Omega)}\leq\frac{C}{K^{k-2}},$ $$

where $C$ is a constant independent of $N$, $\Phi(L,W,S,B;H)$ denote the function class of $ReLU^{2}$ neural networks with depth $L$, width $W$ and at most $S$ non-zero weights taking their values in $[-B,B]$. Moreover, the $W^{2,\infty}$ norms of functions in $\Phi(L,W,S,B;H)$ have the upper bound $H$.

The framework of PINNs can be regarded as a form of multi-task learning (MTL), as a single neural network is designed to simultaneously learn multiple related tasks, involving the enforcement of physical laws or constraints within the learning process. In contrast to traditional single-task learning, MTL encompasses $T$ supervised learning tasks sampled from the input-output space $\mathcal{X}_{1}\times\mathcal{Y}_{1},\cdots,\mathcal{X}_{T}\times\mathcal{Y}_{T}$ respectively. Each task $t$
is represented by an independent random vector $(X_{t},Y_{t})$ distributed according to a probability distribution $\mu_{t}$.

Before presenting our results, we first introduce some notations. Let $(X_{t}^{i},Y_{t}^{i})_{i=1}^{N_{t}}$ be a sequence of i.i.d. random samples drawn from the distribution $\mu_{t}$ for $t=1,\cdots,T$. For any vector-valued function $\bm{f}=(f_{1},\cdots,f_{T})$, we denote its expectation and its empirical part as

$$ $P\bm{f}:=\frac{1}{T}\sum\limits_{t=1}^{T}Pf_{t},\ P_{N}\bm{f}:=\frac{1}{T}\sum \limits_{t=1}^{T}P_{N_{t}}f_{t},$ $$

where $N=(N_{1},\cdots,N_{T})$, $Pf_{t}:=\mathbb{E}[f_{t}(X_{t})]$ and $P_{N_{t}}f_{t}:=\frac{1}{N_{t}}\sum_{i=1}^{N_{t}}f_{t}(X_{t}^{i})$. We denote the componentwise exponentiation of $\bm{f}$ as $\bm{f}^{\alpha}=(f_{1}^{\alpha},\cdots,f_{T}^{\alpha})$ for any $\alpha\in\mathbb{R}$. In the following, we use bold lowercase letters to represent vector-valued functions and bold uppercase letters to indicate the class of functions consisting of vector-valued functions.

To derive sharper generalization bounds for PINNs, we require results from the field of MTL, with a core component being the Talagrand-type concentration inequality. [] has established a Talagrand-type inequality for MTL, which is based on so-called Logaithmic Sobolev inequality on log-moment generating function. Of independent interest, we provide a proof using the entropy method. This not only demonstrates the entropy method’s capability in proving results for the single-task scenario but also shows that it can be readily adapted to the multi-task scenario. Additionally, the concentration inequality yields better constants compared to those offered by Theorem 1 in [].

###### Theorem 3 .

Let $\bm{\mathcal{F}}=\{\bm{f}:=(f_{1},\cdots,f_{T})\}$ be a class of vector-valued functions satisfying $\max\limits_{1\leq t\leq T}\sup\limits_{x\in\mathcal{X}_{t}}|f_{t}(x)|\leq b$. Also assume that
$X:=(X_{t}^{i})_{(t,i)=(1,1)}^{(T,N_{t})}$ is a vector of $\sum_{t=1}^{T}N_{t}$ independent random variables. Let $\{\sigma_{t}^{i}\}_{t,i}$ be a sequence of independent Rademacher variables. If $\frac{1}{T}\sup\limits_{\bm{f}\in\bm{\mathcal{F}}}\sum\limits_{t=1}^{T}Var(f_{
t}(X_{t}^{1}))\leq r$, then for every $x>0$, with probability at least $1-e^{-x}$,

$$ $\sup\limits_{\bm{f}\in\bm{\mathcal{F}}}(P\bm{f}-P_{N}\bm{f})\leq\inf\limits_{ \alpha>0}\left(2(1+\alpha)\mathcal{R}(\bm{\mathcal{F}})+2\sqrt{\frac{xr}{nT}}+ (1+\frac{4}{\alpha})\frac{bx}{nT}\right),$ $$

where $n=\min_{1\leq t\leq T}N_{t}$ and the multi-task Rademacher complexity of function class $\bm{\mathcal{F}}$ is defined as

$$ $\mathcal{R}(\bm{\mathcal{F}}):=\mathbb{E}_{X,\sigma}\sup\limits_{\bm{f}\in\bm{ \mathcal{F}}}\frac{1}{T}\sum\limits_{t=1}^{T}\frac{1}{N_{t}}\sum\limits_{i=1}^ {N_{t}}\sigma_{t}^{i}f_{t}(X_{t}^{i}).$ $$

Moreover, the same bound also holds for $\sup_{\bm{f}\in\bm{\mathcal{F}}}(P_{N}\bm{f}-P\bm{f})$.

Note that the loss functions of the PINNs are all non-negative, thus we only need to consider the case of non-negative functions. Within this context, we can easily derive analogous results to those obtained in the single-task scenario.

In the following, we assume that for any $\bm{f}=(f_{1},\cdots,f_{T})\in\bm{\mathcal{F}}$, $0\leq f_{t}\leq b$ $(1\leq t\leq T)$.

Define

$$ $U_{N}(\bm{\mathcal{F}}):=\sup\limits_{\bm{f}\in\bm{\mathcal{F}}}(P\bm{f}-P_{N} \bm{f}).$ $$

###### Lemma 1 .

For normalized function class $\bm{\mathcal{F}}_{r}$,

$$ $\bm{\mathcal{F}}_{r}:=\{\frac{r}{P\bm{f}^{2}\lor r}\bm{f}:\ \bm{f}\in\bm{ \mathcal{F}}\}$ $$

and assume that for any fixed constants $K>1$ and $r>0$,

$$ $U_{N}(\bm{\mathcal{F}}_{r})\leq\frac{r}{bK}$ $$

Then for any $\bm{f}\in\bm{\mathcal{F}}$ the following inequality holds:

$$ $P\bm{f}\leq\frac{K}{K-1}P_{N}\bm{f}+\frac{r}{bK}.$ $$

###### Lemma 2 .

Let us consider a sub-root function $\psi(r)$ with fixed point $r^{*}$ and suppose that $\forall r>r^{*}$,

$$ $\psi(r)\geq b\mathcal{R}(\bm{\mathcal{F}}_{r}).$ $$

Then for any $K>1$, we have that, with probability at least $1-e^{-x}$ for $\forall\bm{f}\in\bm{\mathcal{F}}$

$$ $P\bm{f}\leq\frac{K}{K-1}P_{N}\bm{f}+\frac{32Kr^{*}}{b}+\frac{(10b+8bK)x}{nT}$ $$

With the preparations made above, we are now ready to derive the generalization bounds for the PINNs.

###### Theorem 4 (Generalization error for PINN loss of the linear second order elliptic equation) .

Let $u^{*}$ be the solution of the linear second order elliptic equation and $n=\min(N_{1},N_{2})$.

(1) If $u^{*}\in\mathcal{B}^{3}(\Omega)$, taking $\mathcal{F}=\mathcal{F}_{m,2}(c\|u^{*}\|_{\mathcal{B}^{3}(\Omega)})$, then with probability at least $1-e^{-t}$,

$$ $\mathcal{L}(u_{N})\leq cC_{1}(\Omega,M)(\frac{m\log n}{n}+(\frac{1}{m})^{1+ \frac{2}{3d}}+\frac{t}{n}),$ $$

where $c$ is a universal constant and $C_{1}(\Omega,M):=\max\{d^{2}M^{2},C(Tr,\Omega),|\Omega|d^{2}M^{4}+|\partial
\Omega|M^{2}\}$, $C(Tr,\Omega)$ is the constant in the Trace theorem for $\Omega$.

By taking $m=n^{\frac{3d}{2(3d+1)}}$, we have

$$ $\mathcal{L}(u_{N})\leq cC_{1}(\Omega,M)((\frac{1}{n})^{\frac{3d+2}{2(3d+1)}} \log n+\frac{t}{n}).$ $$

(2) If $u^{*}\in\mathcal{W}^{k,\infty}(\Omega)$ for $k>3$, taking $\mathcal{F}=\Phi(L,W,S,B;H)$ with $L=\mathcal{O}(1),W=\mathcal{O}(K^{d}),S=\mathcal{O}(K^{d}),B=1,H=\mathcal{O}(1)$, then with probability at least $1-e^{-t}$,

$$ $\mathcal{L}(u_{N})\leq C(\frac{K^{d}\log K\log n}{n}+(\frac{1}{K})^{2k-4}+ \frac{t}{n}),$ $$

where $C$ is a constant independent of $K,N$.

By taking $K=n^{\frac{1}{d+2k-4}}$, we have

$$ $\mathcal{L}(u_{N})\leq C(n^{-\frac{2k-4}{d+2k-4}}(\log n)^{2}+\frac{t}{n}).$ $$

Note that in certain cases, for instance, when $\Omega=(0,1)^{d}$, the constant $C(Tr,\Omega)$ is at most $d$, at this time, $\mathcal{L}(u_{N})$ in Theorem 4(1) only depends polynomially with the underlying dimension.

Theorem 4 provides a generalization error for the loss function of PINNs. However, it is often necessary to measure the generalization error between the empirical solution and the true solution under a certain norm. Fortunately, from Lemma 16, we can deduce that

$$ $\|u_{N}-u^{*}\|_{H^{\frac{1}{2}}(\Omega)}^{2}\leq C_{\Omega}(\|\mathcal{L}u_{N }-f\|_{L^{2}(\Omega)}^{2}+\|u_{N}-g\|_{L^{2}(\partial\Omega)}^{2})=C_{\Omega} \mathcal{L}(u_{N}).$ $$

Therefore, under the settings of Theorem 4, we can determine the generalization error for the linear second order elliptic equation in the $H^{\frac{1}{2}}$ norm.

###### Corollary 2 (Generalization error for linear second order elliptic equation in the $H^{\frac{1}{2}}$ norm) .

(1) If $u^{*}\in\mathcal{B}^{3}(\Omega)$, then

$$ $\|u_{N}-u^{*}\|_{H^{\frac{1}{2}}(\Omega)}^{2}\leq cC_{1}(\Omega,M)((\frac{1}{n })^{\frac{3d+2}{2(3d+1)}}\log n+\frac{t}{n}).$ $$

(2) If $u^{*}\in\mathcal{W}^{k,\infty}(\Omega)$, then

$$ $\|u_{N}-u^{*}\|_{H^{\frac{1}{2}}(\Omega)}^{2}\leq C(n^{-\frac{2k-4}{d+2k-4}}( \log n)^{2}+\frac{t}{n}).$ $$

For PINNs, we focus solely on the $L^{2}$ loss, as considered in the original study []. However, the design of the loss function should incorporate some priori estimation, which serves as a form of stability property []. Specifically, the design of the loss function should follow the principle that if the loss of PINNs is small for some function $u$, then $u$ should be close to the true solution under some appropriate norm. For instance, Theorem 1.2.19 in [] demonstrates that, under some suitable conditions for domain $\Omega$ and related functions $a_{ij},b_{i},c,f,g$, the solution $u^{*}$ of the linear second order elliptic equation satisfies that

$$ $\|u^{*}\|_{H^{2}(\Omega)}\leq C(\|f\|_{L^{2}(\Omega)}+\|g\|_{H^{\frac{3}{2}}( \partial\Omega)}).$ $$

Thus, if apply the loss

$$ $\mathcal{L}(u)=\|Lu-f\|_{L^{2}(\Omega)}^{2}+\|u-g\|_{H^{\frac{3}{2}}(\partial \Omega)}^{2},$ $$

we may obtain the generalization error in the $H^{2}$ norm. However, this term $\|g\|_{H^{\frac{3}{2}}(\partial\Omega)}$ is challenging to compute because it also requires ensuring Lipschitz continuity with respect to the parameters.

To better illustrate the principles of loss selection, we consider the following model semilinear parobolic equation:

$$ $\displaystyle u_{t}$ $\displaystyle=\Delta u+f(u)\quad\forall x\in D,t\in(0,T),$ $\displaystyle u(x,0)$ $\displaystyle=\bar{u}(x)\quad\forall x\in D,$ $\displaystyle u(x,t)$ $\displaystyle=0\quad\forall x\in\partial D,t\in(0,T).$ $$

Here $D\subset\mathbb{R}^{d}$ be a domain, i.e., an open connected bounded set with a $C^{k}$ boundary $\partial D$, $u_{0}\in C^{\bar{s}}(D;\mathbb{R})$ is the initial data for some positive constant $k,\bar{s}$ and $f:\mathbb{R}\times\mathbb{R}$ is the nonlinear source term. We assume that $f,f^{{}^{\prime}}$ are Lipschitz continuous and $f^{{}^{\prime}}$ is bounded.

For any smooth function $u$, we define the following residual:

(1) Interior residual given by

$$ $R_{int}(x,t):=u_{t}(x,t)-\Delta u(x,t)-f(u(x,t)),\ for\ (x,t)\in D\times(0,T].$ $$

(2) Spatial boundary residual given by

$$ $R_{sb}(x,t):=u(x,t),\ for\ (x,t)\in\partial D\times(0,T].$ $$

(3) Temporal boundary residual given by

$$ $R_{tb}(x):=u(x,0)-\bar{u}(x),\ for\ x\in D.$ $$

Then we have the estimation as follows.

###### Proposition 4 .

Let $u^{*}\in C^{k}([0,T]\times\bar{D})$ be the unique classical solution of the semilinear parobolic equation and any $u\in C^{k}([0,T]\times\bar{D})$, for the error $\hat{u}:=u^{*}-u$, we have

$$ $\displaystyle\int_{0}^{T}\int_{D}(|\hat{u}(x,t)|^{2}+|\nabla\hat{u}(x,t)|^{2}+ |\hat{u}_{t}(x,t)|^{2})dxdt$ $\displaystyle\leq C_{T}\left[\int_{0}^{T}\int_{D}(|R_{int}(x,t)|^{2}+|\partial _{t}R_{int}(x,t)|^{2})dxdt+\int_{0}^{T}\int_{\partial D}(|R_{sb}(x,t)|+| \partial_{t}R_{sb}(x,t)|)ds(x)dt\right]$ $\displaystyle\quad+C_{T}\left[\int_{D}(|R_{tb}(x)|^{2}+|\nabla R_{tb}(x)|^{2}+ |\Delta R_{tb}(x)|^{2}+|R_{int}(x,0)|^{2})dx,\right]$ $$

where $C_{T}=CTe^{CT}$ with a constant $C$ independent of $T$.

Contrary to most multi-task learning tasks, the framework of PINNs typically involves only 2 or 3 tasks. This allows us to align samples from different tasks and thus treat them using single-task methods. However, this approach requires that the sample sizes for the different tasks be identical. Specifically, for the linear second order elliptic equation, when $N=(N_{1},N_{2})=(n,n)$, the empirical loss can be expressed as

$$ $\displaystyle\mathcal{L}_{n}(u)$ $\displaystyle:=\frac{|\Omega|}{n}\sum\limits_{k=1}^{n}(-\sum\limits_{i,j=1}^{d }a_{ij}(X_{k})\partial_{ij}u(X_{k})+\sum\limits_{i=1}^{d}b_{i}(X_{k})\partial_ {i}u(X_{k})+c(X_{k})u(X_{k})-f(X_{k}))^{2}$ $\displaystyle\qquad+\frac{|\partial\Omega|}{n}\sum\limits_{k=1}^{n}(u(Y_{k})-g (Y_{k}))^{2}.$ $$

By applying LRC in the single-task context, we can obtain results analogous to those presented in Theorem 4. The reason for presenting the approach is that some variants of PINNs do not fit the standard MTL framework. For instance, within the extended physics-informed neural networks (XPINNs) framework, to ensure continuity, samples from adjacent regions have cross-correlations. The detailed theoretical framework for XPINNs remains an area for future research.

###### Proposition 3 (Approximation results in the H 2 superscript 𝐻 2 H^{2} italic_H start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT norm) .

###### Theorem 3 .

###### Lemma 1 .

###### Lemma 2 .

###### Theorem 4 (Generalization error for PINN loss of the linear second order elliptic equation) .

###### Corollary 2 (Generalization error for linear second order elliptic equation in the H 1 2 superscript 𝐻 1 2 H^{\frac{1}{2}} italic_H start_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT norm) .

###### Proposition 4 .

## 4. Conclusion and Discussion

Conclusion: In this paper, we have refined the generalization bounds for the DRM and PINNs through the localization techniques. For the DRM, our attention was centered on the Poisson equation and the static Schrödinger equation on the $d$-dimensional unit hypercube with Neumann boundary condition. As for the PINNs, our focus shifted to the general linear second elliptic PDEs with Dirichlet boundary condition. Additionally, in both neural networks based approaches for solving PDEs, we considered two scenarios: when the solutions of the PDEs belong to the Barron spaces and when they belong to the Sobolev spaces. Furthermore, we believe that the methodologies established in this paper can be extended to a variety of other methods involving machine learning for solving PDEs.

Discussion and future works:

- •
Through this paper, we made the assumption that all related functions are bounded, as required for localization analysis. However, these assumptions can sometimes be strict. Therefore, it is crucial to investigate settings where the boundedness is not imposed.
- •
In the DRM, we just considered the Neumann boundary condition that does not appear in the variational form. But for the Dirichlet condition, the DRM enforces it by adding a penalty. It’s interesting to investigate the generalization error for that form.
- •
For the PINNs, the loss functions play a crucial role for solving PDEs. It is worth paying more attention to the design of loss functions for different PDEs. Moreover, extending the results in Section 3 to other types of PDEs and other PDE solvers involving neural networks is also a topic for future research.
- •
The optimization error is beyound the scope of this paper. [, ] have considered the optimization error of the two-layer neural networks for the PINNs inspired by the work []. However, it remains open of the optimization aspect for the DRM.
- •
The requirement of the function class of deep neural networks may be impractical. In practice, we might achieve these requirements by restricting the weights of the networks, but this approach can make optimization more difficult. Thus it is worth exploring whether there are more efficient methods.

## 5. Proof of Section 2

### 5.1. Proof of Proposition 1

The proof follows a similar procedure to that in [], but the method in [] can only yield a slow rate of approximation. We start with a sketch of the proof. For any function in the Barron space, we first prove that it belongs to the closure of the convex hull of set in the $H^{1}$ norm. Then by estimating the metric entropy of the set and applying Theorem 1 in [] leads the fast rate of approximation.

For the function $f\in\mathcal{B}^{2}(\Omega)$, according to the definition of Barron space, we can assume that the infimum can be attained at the function $f_{e}$. To simplify the notation, we write $f_{e}$ as $f$, as $f_{e}|_{\Omega}=f$. From the formula of Fourier inverse transform and the fact that $f$ is real-valued,

$$ $\displaystyle f(x)$ $\displaystyle=Re\int_{\mathbb{R}^{d}}e^{i\omega\cdot x}\hat{f}(\omega)d\omega$ $\displaystyle=Re\int_{\mathbb{R}^{d}}e^{i\omega\cdot x}e^{i\theta(\omega)}| \hat{f}(\omega)|d\omega$ $\displaystyle=\int_{\mathbb{R}^{d}}\cos(\omega\cdot x+\theta(\omega))|\hat{f}( \omega)|d\omega$ $\displaystyle=\int_{\mathbb{R}^{d}}\frac{B\cos(\omega\cdot x+\theta(\omega))}{ (1+|\omega|_{1})^{2}}\Lambda(d\omega)$ $\displaystyle=\int_{\mathbb{R}^{d}}g(x,\omega)\Lambda(d\omega),$ $$

where $B=\int_{\mathbb{R}^{d}}(1+|\omega|_{1})^{2}|\hat{f}(\omega)|d\omega$, $\Lambda(d\omega)=\frac{(1+|\omega|_{1})^{2}|\hat{f}(\omega)|d\omega}{B}$ is a probability measure , $e^{i\theta(\omega)}$ is the phase of $\hat{f}(\omega)$ and

$$ $g(x,\omega)=\frac{B\cos(\omega\cdot x+\theta(\omega))}{(1+|\omega|_{1})^{2}}.$ $$

From the integral representation of $f$, we can deduce that $f$ is in the $H^{1}$ closure of the convex hull of the function class

$$ $\mathcal{G}_{cos}(B):=\{\frac{B\cos(\omega\cdot x+t)}{(1+|\omega|_{1})^{2}}: \omega\in\mathbb{R}^{d},t\in\mathbb{R}\}.$ $$

It could be easily verified via the probabilistic method. Assume that $\{\omega_{i}\}_{i=1}^{n}$ is a sequence of i.i.d. random variables distributed according to $\Lambda$, then

$$ $\displaystyle\mathbb{E}[\|f(x)-\frac{1}{n}\sum\limits_{i=1}^{n}g(x,\omega_{i}) \|_{H^{1}(\Omega)}^{2}]$ $\displaystyle=\int_{\Omega}\mathbb{E}[|f(x)-\frac{1}{n}\sum\limits_{i=1}^{n}g( x,\omega_{i})|^{2}+|\nabla f(x)-\frac{1}{n}\sum\limits_{i=1}^{n}\nabla g(x, \omega_{i})|^{2}]dx$ $\displaystyle=\frac{1}{n}\int_{\Omega}Var(g(x,\omega))dx+\frac{1}{n}\int_{ \Omega}Tr(Cov[\nabla g(x,\omega)])dx$ $\displaystyle\leq\frac{\mathbb{E}[\|g(x,\omega)\|_{H^{1}(\Omega)}^{2}]}{n}$ $\displaystyle\leq\frac{2B^{2}}{n},$ $$

where the first equality follows from Fubini’s theorem and the last inequality holds due to the facts that $|g(x,\omega)|\leq B$ and $|\nabla g(x,\omega)|\leq B$ for any $x,\omega$.

Then, for any given tolerance $\epsilon>0$, by Markov’s inequality,

$$ $P(\|f(x)-\frac{1}{n}\sum\limits_{i=1}^{n}g(x,\omega_{i})\|_{H^{1}(\Omega)}> \epsilon)\leq\frac{1}{\epsilon^{2}}\mathbb{E}[\|f(x)-\frac{1}{n}\sum\limits_{i =1}^{n}g(x,\omega_{i})\|_{H^{1}(\Omega)}^{2}]\leq\frac{2B^{2}}{n\epsilon^{2}}.$ $$

By choosing a large enough $n$ such that $\frac{2B^{2}}{n\epsilon^{2}}<1$, we have

$$ $P(\|f(x)-\frac{1}{n}\sum\limits_{i=1}^{n}g(x,\omega_{i})\|_{H^{1}(\Omega)}\leq \epsilon)>0,$ $$

which implies that there exist realizations of the random variables $\{\omega_{i}\}_{i=1}^{n}$ such that $\|f(x)-\frac{1}{n}\sum\limits_{i=1}^{n}g(x,\omega_{i})\|_{H^{1}(\Omega)}\leq\epsilon$. Therefore, the conclusion holds.

Next, we are going to show that those functions in $\mathcal{G}_{cos}(B)$ are in the $H^{1}$ closure of the convex hull of the function class
$\mathcal{F}_{\sigma}(5B)\cup\mathcal{F}_{\sigma}(-5B)\cup\{0\}$, where

$$ $\mathcal{F}_{\sigma}(b):=\{b\sigma(\omega\cdot x+t):|\omega|_{1}=1,t\in[-1,1]\}$ $$

for any constant $b\in\mathbb{R}$.

Note that although $\mathcal{G}_{cos}(B)$ consists of high-dimensinal functions, those functions depend only on the projection of multivariate variable $x$. Specifically, each function in $\mathcal{G}_{cos}(B)$ is the composition of a one-dimensional function $g(z)=\frac{B\cos(|\omega|_{1}z+t)}{(1+|\omega|_{1}^{2})}$ and a linear function $z=\frac{\omega}{|\omega|_{1}}\cdot x$ with value in $[-1,1]$. Therefore, it suffices to prove the conclusion for $g(z)$ on $[-1,1]$, i.e., to prove that for each $\omega$, $g$ is in the $H^{1}$ closure of convex hull of $\mathcal{F}_{\sigma}^{1}(5B)\cup\mathcal{F}_{\sigma}^{1}(-5B)\cup\{0\}$, where

$$ $\mathcal{F}_{\sigma}^{1}(b):=\{b\sigma(\epsilon z+t):\epsilon=-1\ or\ 1,t\in[- 1,1]\}$ $$

for any constant $b\in\mathbb{R}$. Then applying the variable substitution leads to the conclusion for $g(x,\omega)$.

In fact, it is easier to handle that in one-dimension due to the relationship between the ReLU functions and the basis function in the finite element method (FEM) [], i.e. the basis functions can be represented by ReLU functions. To make this more precise, let us consider the uniform mesh of interval $[-1,1]$ by taking $m+1$ points

$$ $-1=x_{0}<x_{1}<\cdots<x_{m}=1,$ $$

and set $h=\frac{2}{m}$, $x_{-1}=-1-h,x_{m+1}=1+h$. For $0\leq i\leq m$, introduce the function $\varphi_{i}(z)$, which is defined as follows:

$$ $\varphi_{i}(z)=\left\{\begin{aligned} \frac{1}{h}(z-z_{i-1})&,&if\ z\in[z_{i-1 ,},z_{i}],\\ \frac{1}{h}(z_{i+1}-z)&,&if\ z\in[z_{i},z_{i+1}],\\ 0&,&otherwise.\end{aligned}\right.$ (1) $$

Clearly, the set $\{\varphi_{0},\cdots,\varphi_{m}\}$ is a basis of $\mathcal{P}_{h}^{1}$ ($P_{1}$ lagrange finite element, see Chapter 1 of [] for more details). And $\varphi_{i}$ can be written as

$$ $\varphi_{i}(z)=\frac{\sigma(z-z_{i-1})-2\sigma(z-z_{i})+\sigma(z-z_{i+1})}{h}.$ $$

Now, we are ready to present the definition of interpolation operator and the estimation of interpolation error [].

Consider the so-called interpolation operator

$$ $\mathcal{I}_{h}^{1}:v\in C([-1,1])\rightarrow\sum\limits_{i=0}^{m}v(z_{i}) \varphi_{i}\in P_{h}^{1}.$ $$

Then for all $h$ and $v\in H^{2}([-1,1])$, the interpolation error can be bounded as

$$ $\|v-\mathcal{I}_{h}^{1}v\|_{L^{2}([-1,1])}\leq h^{2}\|v^{{}^{\prime\prime}}\|_ {L^{2}([-1,1])}\ and\ \|v^{{}^{\prime}}-(\mathcal{I}_{h}^{1}v)^{{}^{\prime}}\| _{L^{2}([-1,1])}\leq h\|v^{{}^{\prime\prime}}\|_{L^{2}([-1,1])}.$ $$

By using the interpolation operator and the connection between the ReLU functions and the basis functions, we can establish the conclusion for one-dimensional functions.

###### Lemma 3 .

Let $g\in\mathcal{C}^{2}([-1,1])$ with $\|g^{(s)}\|_{L^{\infty}}\leq B$ for $s=0,1,2$. Then there exists a two-layer ReLU network $g_{m}$ of the form

$$ $g_{m}(z)=\sum\limits_{i=1}^{6m-1}a_{i}\sigma(\epsilon_{i}z+t_{i}),$ $$

with $|a_{i}|\leq\frac{2B}{m}$, $\sum\limits_{i=1}^{6m-1}|a_{i}|\leq 5B$, $|t_{i}|\leq 1$, $\epsilon_{i}\in\{-1,1\}$, $1\leq i\leq 6m-1$ such that

$$ $\|g-g_{m}\|_{H^{1}([-1,1])}\leq\frac{4\sqrt{2}B}{m}.$ $$

Therefore, $g$ is in the $H^{1}([-1,1])$ closure of the convex hull of $\mathcal{F}_{\sigma}^{1}(5B)\cup\mathcal{F}_{\sigma}^{1}(-5B)\cup\{0\}$.

###### Proof.

Notice that the interpolant of $g$ can be written as a combination of ReLU functions as follows

$$ $\displaystyle\mathcal{I}_{h}^{1}(g)$ $\displaystyle=\sum\limits_{i=0}^{m}g(z_{i})\varphi_{i}(z)$ $\displaystyle=\frac{g(z_{0})(\sigma(z-z_{-1})-2\sigma(z-z_{0}))}{h}+\frac{g(z_ {1})\sigma(z-z_{0})}{h}+\sum\limits_{i=1}^{m-1}\frac{g(z_{i-1})-2g(z_{i})+g(z_ {i+1})}{h}\sigma(z-z_{i})$ $\displaystyle=g(z_{0})+\frac{g(z_{1})-g(z_{0})}{h}\sigma(z-z_{0})+\sum\limits_ {i=1}^{m-1}\frac{g(z_{i-1})-2g(z_{i})+g(z_{i+1})}{h}\sigma(z-z_{i}).$ $$

By the mean value theorem, there exist $\xi_{0}\in[z_{0},z_{1}]$ and $\xi_{i}\in[z_{i-1},z_{i+1}]$ for $1\leq i\leq m-1$ such that
$g(z_{1})-g(z_{0})=g^{{}^{\prime}}(\xi_{0})h$ and $g(z_{i-1})-2g(z_{i})+g(z_{i+1})=g^{{}^{\prime\prime}}(\xi_{i})h^{2}$ for $1\leq i\leq m-1$.

Therefore, $\mathcal{I}_{h}^{1}(g)$ can be rewritten as

$$ $\mathcal{I}_{h}^{1}(g)=g(z_{0})+g^{{}^{\prime}}(\xi_{0})\sigma(z-z_{0})+\sum \limits_{i=1}^{m-1}g^{{}^{\prime\prime}}(\xi_{i})\sigma(z-z_{i})h.$ $$

Notice that the constant can also be represented as a combination of ReLU functions on $[-1,1]$. By the observation that $\sigma(z)+\sigma(-z)=|z|$, we have that for any $z\in[-1,1]$

$$ $1=\frac{|1+z|+|1-z|}{2}=\frac{\sigma(z+1)+\sigma(-z-1)+\sigma(-z+1)+\sigma(z-1 )}{2}.$ $$

Plugging the above equality into the expression of $\mathcal{I}_{h}^{1}(g)$ yields that

$$ $\displaystyle\mathcal{I}_{h}^{1}(g)=$ $\displaystyle\sum\limits_{i=1}^{m}\frac{g(-1)(\sigma(z+1)+\sigma(-z-1)+\sigma( -z+1)+\sigma(z-1))}{2m}+\sum\limits_{i=1}^{m}\frac{g^{{}^{\prime}}(\xi_{0}) \sigma(z-z_{0})}{m}$ $\displaystyle+\sum\limits_{i=1}^{m-1}\frac{2g^{{}^{\prime\prime}}(\xi_{i}) \sigma(z-z_{i})}{m}.$ $$

Combining the expression of $\mathcal{I}_{h}^{1}(g)$ and the estimation for interpolation error leads to that there exists a two-layer neural netwok $g_{m}$ of the form

$$ $g_{m}(z)=\mathcal{I}_{h}^{1}(g)=\sum\limits_{i=1}^{6m-1}a_{i}\sigma(\epsilon_{ i}z+t_{i}),$ $$

with $|a_{i}|\leq\frac{2B}{m}$, $\sum\limits_{i=1}^{6m-1}|a_{i}|\leq 5B$,$|t_{i}|\leq 1$, $\epsilon_{i}\in\{-1,1\}$,$1\leq i\leq 6m-1$ such that

$$ $\|g-g_{m}\|_{H^{1}([-1,1])}\leq\frac{4\sqrt{2}B}{m}.$ $$

∎

Although the interpolation operator can be view as a piecewise linear interpolation of $g$, which is similar to Lemma 18 in [], our result does not require $g^{{}^{\prime}}(0)=0$ and the value of $g$ at the certain point is also expressed as a combination of ReLU functions. Specificlly, the $g_{m}$ in Lemma 18 of [] has the form $g_{m}(z)=c+\sum_{i=1}^{2m}a_{i}\sigma(\epsilon_{i}z+t_{i})$, where $c=g(0)$ and they partition $[-1,1]$ by $2m$ points with $z_{0}=-1,z_{m}=0,z_{2m}=1$. And our result can also be extended in $W^{1,\infty}$ norm like Lemma 18 of []. Note that on $[z_{i-1},z_{i}]$

$$ $\mathcal{I}_{h}^{1}(g)(z)=g(z_{i-1})\frac{z_{i}-z}{h}+g(z_{i})\frac{z-z_{i-1}} {h},$ $$

which is the piecewise linear interpolation of $g$. Then by bounding the remainder in Lagrange interpolation formula, we have $\|\mathcal{I}_{h}(g)-g\|_{L^{\infty}[z_{i-1},z_{i}]}\leq\frac{h^{2}}{8}\|g^{{}
^{\prime\prime}}\|_{L^{\infty}[z_{i-1},z_{i}]}$ and

$$ $\displaystyle|(\mathcal{I}_{h}^{1}(g))^{{}^{\prime}}(z)-g^{{}^{\prime}}(z)|$ $\displaystyle=|\frac{g(z_{i})-g(z_{i-1})}{h}-g^{{}^{\prime\prime}}(z)|$ $\displaystyle\leq|g^{{}^{\prime}}(\xi_{i})-g^{{}^{\prime}}(z_{i})|$ $\displaystyle\leq h\|g^{{}^{\prime\prime}}\|_{L^{\infty}[z_{i-1},z_{i}]},$ $$

where the first inequality follows from the mean value theorem.

Therefore, $\|\mathcal{I}_{h}^{1}(g)-g\|_{W^{1,\infty}([-1,1])}\leq\frac{2B}{m}$.

By combining the above results in the one-dimension, the following conclusion holds in the high-dimension through using the variable substitution.

###### Proposition 5 .

For any given function $f$ in $\mathcal{B}^{2}(\Omega)$, $f$ is in the $H^{1}$ closure of the convex hull of $\mathcal{F}_{\sigma}(5\|f\|_{\mathcal{B}^{2}(\Omega)})\cup\mathcal{F}_{\sigma}
(-5\|f\|_{\mathcal{B}^{2}(\Omega)})\cup\{0\}$. Specifically, for any $\epsilon>0$, there exist $m\in\mathbb{N}$ and $\omega_{i},t_{i},a_{i},1\leq i\leq m$ such that

$$ $\|f(x)-\sum\limits_{i=1}^{m}a_{i}\sigma(\omega_{i}\cdot x+t_{i})\|_{H^{1}( \Omega)}\leq\epsilon,$ $$

where $|\omega_{i}|_{1}=1,t_{i}\in[-1,1],1\leq i\leq m$ and $\sum\limits_{i=1}^{m}|a_{i}|\leq 5\|f\|_{\mathcal{B}^{2}(\Omega)}$.

Proposition 4 implies that functions in $\mathcal{B}^{2}(\Omega)$ can be approximated by a linear combination of functions in $\mathcal{F}_{\sigma}(1)$.

Recall that $\mathcal{F}_{\sigma}(1)=\{\sigma(\omega\cdot x+t):|\omega|_{1}=1,t\in[-1,1]\}$. For simplicity, we write $\mathcal{F}_{\sigma}$ for $\mathcal{F}_{\sigma}(1)$.

Then to invoke Theorem 1 in [], it remains to estimate the metric entropy of the function class $\mathcal{F}_{\sigma}$, which is defined as

$$ $\epsilon_{n}(\mathcal{F}_{\sigma}):=\inf\{\epsilon:\mathcal{F}_{\sigma}\ can\ be\ covered\ by\ at\ most\ n\ sets\ of\ diameter\ \leq\epsilon\ under\ the\ H^ {1}\ norm\}.$ $$

By Lemma 8, we just need to estimate the covering number of $\mathcal{F}_{\sigma}$, which is easier to handle.

###### Proposition 6 (Estimation of the metric entropy) .

For any $n\in\mathbb{N}$,

$$ $\epsilon_{n}(\mathcal{F}_{\sigma})\leq cn^{-\frac{1}{3d}},$ $$

where $c$ is a universal constant.

###### Proof.

For $(\omega_{1},t_{1}),(\omega_{2},t_{2})\in\partial B_{1}^{d}(1)\times[-1,1]$, we have

$$ $\displaystyle\|\sigma(\omega_{1}\cdot x+t_{1})-\sigma(\omega_{2}\cdot x+t_{2}) \|_{H^{1}(\Omega)}^{2}$ $\displaystyle=\int_{\Omega}|\sigma(\omega_{1}\cdot x+t_{1})-\sigma(\omega_{2} \cdot x+t_{2})|^{2}dx+\int_{\Omega}|\nabla\sigma(\omega_{1}\cdot x+t_{1})- \nabla\sigma(\omega_{2}\cdot x+t_{2})|^{2}dx$ $\displaystyle\leq\int_{\Omega}|(\omega_{1}-\omega_{2})\cdot x+(t_{1}-t_{2})|^{ 2}dx+\int_{\Omega}|\omega_{1}I_{\{\omega_{1}\cdot x+t_{1}\geq 0\}}-\omega_{2}I _{\{\omega_{2}\cdot x+t_{2}\geq 0\}}|^{2}dx$ $\displaystyle\leq 2(|\omega_{1}-\omega_{2}|_{1}^{2}+|t_{1}-t_{2}|^{2})+\int_{ \Omega}|(\omega_{1}-\omega_{2})I_{\{\omega_{1}\cdot x+t_{1}\geq 0\}}+\omega_{2 }(I_{\{\omega_{1}\cdot x+t_{1}\geq 0\}}-I_{\{\omega_{2}\cdot x+t_{2}\geq 0\}}) |^{2}dx$ $\displaystyle\leq 2(|\omega_{1}-\omega_{2}|_{1}^{2}+|t_{1}-t_{2}|^{2})+2| \omega_{1}-\omega_{2}|_{1}^{2}+2\int_{\Omega}|I_{\{\omega_{1}\cdot x+t_{1}\geq 0 \}}-I_{\{\omega_{2}\cdot x+t_{2}\geq 0\}}|^{2}dx$ $\displaystyle\leq 4(|\omega_{1}-\omega_{2}|_{1}^{2}+|t_{1}-t_{2}|^{2})+2\int_{ \Omega}|I_{\{\omega_{1}\cdot x+t_{1}\geq 0\}}-I_{\{\omega_{2}\cdot x+t_{2}\geq 0 \}}|^{2}dx,$ $$

where the first inequality is due to that $\sigma$ is 1-Lipschitz continuous, the second and the third inequalities follow the from the mean inequality and the fact that the $2$-norm is dominated by the $1$- norm.

It is challenging to handle the first and second terms simultaneously, thus we turn to handle two terms separately due to that the first term is related to the covering of $\partial B_{1}^{d}(1)\times[-1,1]$ and the second term is related to the covering of a VC-class of functions (see Chaper 2.6 of [] or Chapter 9 of []). Therefore, we consider a new space
$\mathcal{G}_{1}$ defined as

$$ $\mathcal{G}_{1}:=\{((\omega,t),I_{\{\omega\cdot x+t\geq 0\}}):\omega\in \partial B_{1}^{d}(1),t\in[-1,1]\}.$ $$

Obviously, it is a subset of the metric space

$$ $\mathcal{G}_{2}:=\{((\omega_{1},t_{1}),I_{\{\omega_{2}\cdot x+t_{2}\geq 0\}}): \omega_{1},\omega_{2}\in\partial B_{1}^{d}(1),t_{1},t_{2}\in[-1,1]\}$ $$

with the metric $d$ that for $b_{1}=((\omega_{1}^{1},t_{1}^{1}),I_{\{\omega_{2}^{1}\cdot x+t_{2}^{1}\geq 0\}
}),b_{2}=((\omega_{1}^{2},t_{1}^{2}),I_{\{\omega_{2}^{2}\cdot x+t_{2}^{2}\geq 0
\}})$,

$$ $d(b_{1},b_{2}):=\sqrt{2(|\omega_{1}^{1}-\omega_{1}^{2}|_{1}^{2}+|t_{1}^{1}-t_{ 1}^{2}|^{2})}+\|I_{\{\omega_{2}^{1}\cdot x+t_{2}^{1}\geq 0\}}-I_{\{\omega_{2}^ {2}\cdot x+t_{2}^{2}\geq 0\}}\|_{L^{2}(\Omega)}.$ $$

The key point is that $\mathcal{G}_{2}$ can be seen as a product space of $\partial B_{1}^{d}(1)\times[-1,1]$ and the function class $\mathcal{F}_{1}:=\{I_{\{\omega\cdot x+t\geq 0\}}:(\omega,t)\in\partial B_{1}^{
d}(1)\times[-1,1]\}$ is a VC-class. Therefore, we can handle the two terms separately.

Note that by defining the metric $d_{1}$ in $\partial B_{1}^{d}(1)\times[-1,1]$ as

$$ $d_{1}((\omega_{1}^{1},t_{1}^{1}),(\omega_{1}^{2},t_{1}^{2}))=\sqrt{2(|\omega_{ 1}^{1}-\omega_{1}^{2}|_{1}^{2}+|t_{1}^{1}-t_{1}^{2}|^{2})}$ $$

and the metric $d_{2}$ in $\mathcal{F}_{1}$ as

$$ $d_{2}(I_{\{\omega_{2}^{1}\cdot x+t_{2}^{1}\geq 0\}},I_{\{\omega_{2}^{2}\cdot x +t_{2}^{2}\geq 0\}})=\|I_{\{\omega_{2}^{1}\cdot x+t_{2}^{1}\geq 0\}}-I_{\{ \omega_{2}^{2}\cdot x+t_{2}^{2}\geq 0\}}\|_{L^{2}(\Omega)},$ $$

the covering number of $\mathcal{G}_{2}$ can be bounded as

$$ $\mathcal{N}(\mathcal{G}_{2},d,\epsilon)\leq\mathcal{N}(\partial B_{1}^{d}(1) \times[-1,1],d_{1},\frac{\epsilon}{2})\cdot\mathcal{N}(\mathcal{F}_{1},d_{2}, \frac{\epsilon}{2}).$ $$

Observe that $\mathcal{F}_{1}$ is a collection of all indicator functions of sets in a class with finite VC-dimension, then Theorem 2.6.4 in [] yields

$$ $\mathcal{N}(\mathcal{F}_{1},d_{2},\epsilon)\leq K(d+1)(4e)^{d+1}(\frac{1}{ \epsilon})^{2d}$ $$

with a universal constant $K$, since the collection of all half-spaces in $\mathbb{R}^{d}$ is a VC-class of dimension $d+1$ (see Lemma 9.12(i) in []).

By the inequality $\sqrt{|a|+|b|}\leq\sqrt{|a|}+\sqrt{|b|}$, we have

$$ $\sqrt{2|\omega_{1}^{1}-\omega_{1}^{2}|_{1}^{2}+|t_{1}^{1}-t_{1}^{2}|^{2}}\leq \sqrt{2}(|\omega_{1}^{1}-\omega_{1}^{2}|_{1}+|t_{1}^{1}-t_{1}^{2}|),$ $$

therefore

$$ $\mathcal{N}(\partial B_{1}^{d}(1)\times[-1,1],d_{1},\epsilon)\leq\mathcal{N}( \partial B_{1}^{d}(1),|\cdot|_{1},\frac{\sqrt{2}}{2}\epsilon)\cdot\mathcal{N}( [-1,1],|\cdot|,\frac{\sqrt{2}}{2}\epsilon).$ $$

Combining all results above and Lemma 10, we can compute an upper bound for the covering number of $\mathcal{G}_{1}$.

$$ $\displaystyle\mathcal{N}(\mathcal{G}_{1},d,\epsilon)$ $\displaystyle\leq\mathcal{N}(\mathcal{G}_{2},d,\frac{\epsilon}{2})$ $\displaystyle\leq\mathcal{N}(\partial B_{1}^{d}(1)\times[-1,1],d_{1},\frac{ \epsilon}{4})\cdot\mathcal{N}(\mathcal{F}_{1},d_{2},\frac{\epsilon}{4})$ $\displaystyle\leq\mathcal{N}(\partial B_{1}^{d}(1),|\cdot|_{1},\frac{\sqrt{2}} {8}\epsilon)\cdot\mathcal{N}([-1,1],|\cdot|,\frac{\sqrt{2}}{8}\epsilon)\cdot \mathcal{N}(\mathcal{F}_{1},d_{2},\frac{\epsilon}{4})$ $\displaystyle\leq K(d+1)(4e)^{d+1}(\frac{c}{\epsilon})^{3d},$ $$

where $c$ is a universal constant.

Therefore, applying Lemma 11 yields the desired conclusion.
∎

###### Lemma 3 .

###### Proof.

###### Proposition 5 .

###### Proposition 6 (Estimation of the metric entropy) .

###### Proof.

### 5.2. Proof of Theorem 1

The proof is based on a new error decomposition and the peeling method. The
key point is the fact that $\int_{\Omega}u^{*}(x)dx=0$, thus for any $u\in H^{1}(\Omega)$,

$$ $(\int_{\Omega}u(x)dx)^{2}=(\int_{\Omega}(u(x)-u^{*}(x))dx)^{2}\leq\int_{\Omega }(u(x)-u^{*}(x))^{2}dx\leq\|u-u^{*}\|_{H^{1}(\Omega)}^{2},$ $$

which implies that if $u$ is close enough to $u^{*}$ in the $H^{1}$ norm, then
$(\int_{\Omega}u(x)dx)^{2}$ is small. Furthermore, if $u$ is bounded, we can also prove that the empirical part of $(\int_{\Omega}u(x)dx)^{2}$ ,i.e. $(\frac{1}{n}\sum\limits_{i=1}^{n}u(X_{i}))^{2}$ is also small in high probability via the Hoeffding’s inequality.

Through the proof, we omit the notation for the Poisson equation, i.e., we write $\mathcal{E}$ and $\mathcal{E}_{n}$ for the population loss $\mathcal{E}_{P}$ and empirical loss $\mathcal{E}_{n,P}$ respectively.
Additionally, we assume that there is a constant $M$ such that
$|u^{*}|,|\nabla u^{*}|,|f|\leq M$.

Assume that $u_{n}$ is the minimal solution from minimizing the empirical loss $\mathcal{E}_{n}$ in the function class $\mathcal{F}$, here we just take $\mathcal{F}$ as a parametrized hypothesis function class. When considering the specific setting, we can choose $\mathcal{F}$ to be the function class of two-layer neural networks or deep neural networks. Additionally, we also assume that those functions in $\mathcal{F}$ and their partial derivatives are bounded by $M$ in absolut value.

Recall that the population loss and its empirical part are

$$ $\mathcal{E}(u)=\int_{\Omega}|\nabla u(x)|^{2}dx-\int_{\Omega}2f(x)u(x)dx+(\int _{\Omega}u(x)dx)^{2}$ $$

and

$$ $\mathcal{E}_{n}(u)=\frac{1}{n}\sum\limits_{i=1}^{n}|\nabla u(X_{i})|^{2}-\frac {2}{n}\sum\limits_{i=1}^{n}f(X_{i})u(X_{i})+(\frac{1}{n}\sum\limits_{i=1}^{n}u (X_{i}))^{2}.$ $$

We begin by introducing the error decomposition:

$$ $\displaystyle\mathcal{E}(u_{n})-\mathcal{E}(u^{*})$ $\displaystyle=\mathcal{E}(u_{n})-\lambda\mathcal{E}_{n}(u_{n})+\lambda( \mathcal{E}_{n}(u_{n})-\mathcal{E}_{n}(u_{\mathcal{F}}))+\lambda\mathcal{E}_{n }(u_{\mathcal{F}})-\mathcal{E}(u^{*})$ $\displaystyle\leq\mathcal{E}(u_{n})-\lambda\mathcal{E}_{n}(u_{n})+\lambda \mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}(u^{*})$ $\displaystyle=\mathcal{E}(u_{n})-\lambda\mathcal{E}_{n}(u_{n})+\lambda( \mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}_{n}(u^{*}))+\lambda\mathcal{E}_{n }(u^{*})-\mathcal{E}(u^{*})$ $\displaystyle=(\mathcal{E}(u_{n})-\mathcal{E}(u^{*}))-\lambda(\mathcal{E}_{n}( u_{n})-\mathcal{E}_{n}(u^{*}))+\lambda(\mathcal{E}_{n}(u_{\mathcal{F}})- \mathcal{E}_{n}(u^{*}))$ $\displaystyle\leq\sup\limits_{u\in\mathcal{F}}[(\mathcal{E}(u)-\mathcal{E}(u^{ *}))-\lambda(\mathcal{E}_{n}(u)-\mathcal{E}_{n}(u^{*}))]+\lambda(\mathcal{E}_{ n}(u_{\mathcal{F}})-\mathcal{E}_{n}(u^{*})),$ $$

where the first inequality follows from the definition of $u_{n}$ and $\lambda$ is a constant to be determined.

Rearranging the term $\mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}_{n}(u^{*})$ yields

$$ $\displaystyle\mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}_{n}(u^{*})$ $\displaystyle=\frac{1}{n}\sum\limits_{i=1}^{n}|\nabla u_{\mathcal{F}}(X_{i})|^ {2}+(\frac{1}{n}\sum\limits_{i=1}^{n}u_{\mathcal{F}}(X_{i}))^{2}-\frac{2}{n} \sum\limits_{i=1}^{n}f(X_{i})u_{\mathcal{F}}(X_{i})$ $\displaystyle\ \ \ -[\frac{1}{n}\sum\limits_{i=1}^{n}|\nabla u^{*}(X_{i})|^{2} +(\frac{1}{n}\sum\limits_{i=1}^{n}u^{*}(X_{i}))^{2}-\frac{2}{n}\sum\limits_{i= 1}^{n}f(X_{i})u^{*}(X_{i})]$ $\displaystyle=\frac{1}{n}\sum\limits_{i=1}^{n}[(|\nabla u_{\mathcal{F}}(X_{i}) |^{2}-2f(X_{i})u_{\mathcal{F}}(X_{i}))-(|\nabla u^{*}(X_{i})|^{2}-2f(X_{i})u^{ *}(X_{i}))]$ $\displaystyle\ \ \ +[(\frac{1}{n}\sum\limits_{i=1}^{n}u_{\mathcal{F}}(X_{i}))^ {2}-(\frac{1}{n}\sum\limits_{i=1}^{n}u^{*}(X_{i}))^{2}]$ $\displaystyle:=\phi_{n}^{1}+\phi_{n}^{2},$ $$

where in the last equality, we denote the right two terms in the second equality as $\phi_{n}^{1}$ and $\phi_{n}^{2}$ respectively.

Define

$$ $h(x)=(|\nabla u_{\mathcal{F}}(x)|^{2}-2f(x)u_{\mathcal{F}}(x))-(|\nabla u^{*}( x)|^{2}-2f(x)u^{*}(x)),$ $$

then by the boundedness of $u_{\mathcal{F}},\nabla u_{\mathcal{F}},u^{*},\nabla u^{*}$ and $f$, we can deduce that

$$ $Var(h)\leq P(h^{2})\leq 8M^{2}\|u_{\mathcal{F}}-u^{*}\|_{H^{1}(\Omega)}^{2}=8M ^{2}\epsilon_{app}^{2}\ and\ |h-\mathbb{E}[h]|\leq 2\sup|h|\leq 12M^{2}.$ $$

Therefore, from Bernstein inequality (see Lemma 7), with probability at least $1-e^{-t}$,

$$ $\displaystyle\phi_{n}^{1}$ $\displaystyle=\frac{1}{n}\sum\limits_{i=1}^{n}[(|\nabla u_{\mathcal{F}}(X_{i}) |^{2}-2f(X_{i})u_{\mathcal{F}}(X_{i}))-(|\nabla u^{*}(X_{i})|^{2}-2f(X_{i})u^{ *}(X_{i}))]$ $\displaystyle\leq\mathbb{E}[h(X)]+\sqrt{\frac{24M^{2}t}{n}\epsilon_{app}^{2}}+ \frac{4M^{2}t}{n}$ $\displaystyle=\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^{*})-(\int_{\Omega}u_ {\mathcal{F}}dx)^{2}+\sqrt{\frac{24M^{2}t}{n}\epsilon_{app}^{2}}+\frac{4M^{2}t }{n}$ $\displaystyle\leq C(\epsilon_{app}^{2}+\frac{M^{2}t}{n}),$ $$

where the last inequality follows by the basic inequality $2\sqrt{ab}\leq a+b$ for any $a,b>0$ and Proposition 1(1).

For $\phi_{n}^{2}$, the Hoeffding inequality (see Lemma 8) implies

$$ $P(|\frac{1}{n}\sum\limits_{i=1}^{n}u_{\mathcal{F}}(X_{i})-\int_{\Omega}u_{ \mathcal{F}}(x)dx|\geq 2M\sqrt{\frac{2t}{n}})\leq 2e^{-t}.$ $$

Therefore with probabilty at least $1-2e^{-t}$,

$$ $\displaystyle\phi_{n}^{2}$ $\displaystyle=(\frac{1}{n}\sum\limits_{i=1}^{n}u_{\mathcal{F}}(X_{i}))^{2}-( \frac{1}{n}\sum\limits_{i=1}^{n}u^{*}(X_{i}))^{2}$ $\displaystyle\leq(\frac{1}{n}\sum\limits_{i=1}^{n}u_{\mathcal{F}}(X_{i}))^{2}$ $\displaystyle\leq 2(|\frac{1}{n}\sum\limits_{i=1}^{n}u_{\mathcal{F}}(x_{i})- \int_{\Omega}u_{\mathcal{F}}(x)dx|^{2}+|\int_{\Omega}u_{\mathcal{F}}(x)dx|^{2})$ $\displaystyle\leq C(\epsilon_{app}^{2}+\frac{M^{2}t}{n}).$ $$

Combining the bounds for $\phi_{n}^{1}$ and $\phi_{n}^{2}$, we can deduce that with probability as least $1-3e^{-t}$,

$$ $\mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}_{n}(u_{*})\leq C(\epsilon_{app}^{ 2}+\frac{M^{2}t}{n}).$ $$

Plugging it into the error decomposition yields that with probability as least $1-3e^{-t}$,

$$ $\mathcal{E}(u_{n})-\mathcal{E}(u^{*})\leq(\mathcal{E}(u_{n})-\mathcal{E}(u^{*} ))-\lambda(\mathcal{E}_{n}(u_{n})-\mathcal{E}_{n}(u^{*}))+\lambda C(\epsilon_{ app}^{2}+\frac{M^{2}t}{n}).$ $$

One may employ the peeling technique to establish an upper bound for the first term.

Let $\rho_{0}$ be a positive constant to be determined and $\rho_{k}=2\rho_{k-1}$ for $k\geq 1$.

Consider the sets $\mathcal{F}_{k}:=\{u\in\mathcal{F}:\rho_{k-1}<\|u-u^{*}\|_{H^{1}(\Omega)}^{2}
\leq\rho_{k}\}$ for $k\geq 1$ and $\mathcal{F}_{0}=\{u\in\mathcal{F}:\|u-u^{*}\|_{H^{1}(\Omega)}^{2}\leq\rho_{0}\}$.

The boundedness of the functions in $\mathcal{F}$, $u^{*}$ and their respective gradients implies that

$$ $K:=\max k\leq\log\frac{2M^{2}}{\rho_{0}},$ $$

since $\rho_{K}=2^{K}\rho_{0}$.

Then set $\delta_{k}=\frac{\delta}{K+1}$ for $0\leq k\leq K$ with a fixed constant $\delta\in(0,1)$.

From Lemma 13, we know that with probability at least $1-\delta_{k}$

$$ $\displaystyle\sup\limits_{u\in\mathcal{F}_{k}}(\mathcal{E}(u)-\mathcal{E}(u^{* }))-(\mathcal{E}_{n}(u)-\mathcal{E}(u^{*}))$ $\displaystyle\leq C(\frac{\alpha M^{2}\log(2\beta\sqrt{n})}{n}+\sqrt{\frac{M^{ 2}\rho_{k}\alpha\log(2\beta\sqrt{n})}{n}}+\sqrt{\frac{M^{2}\rho_{k}\log\frac{1 }{\delta_{k}}}{n}}$ $\displaystyle\ \ \ +\frac{M^{2}\log\frac{1}{\delta_{k}}}{n}+\sqrt{\frac{aM^{2} \rho_{k}}{n}\log\frac{4b}{M}}),$ $$

where $\alpha,\beta,a,b$ are constants depending on the complexity of $\mathcal{F}$ (see the definitions in Lemma 14).

Note that

$$ $\displaystyle\rho_{k}$ $\displaystyle\leq\max\{\rho_{0},2\rho_{k-1}\}$ $\displaystyle\leq\max\{\rho_{0},2\|u-u^{*}\|_{H^{1}(\Omega)}^{2}\}$ $\displaystyle\leq\max\{\rho_{0},2C_{P}(\mathcal{E}(u)-\mathcal{E}(u^{*}))\}$ $\displaystyle\leq\rho_{0}+2C_{P}(\mathcal{E}(u)-\mathcal{E}(u^{*}))$ $$

holds for any $u\in\mathcal{F}_{k}$
and

$$ $\log\frac{1}{\delta_{k}}=\log\frac{K+1}{\delta}\leq\log\frac{1}{\delta}+\log \log\frac{2eM^{2}}{\rho_{0}}.$ $$

Therefore, setting $\rho_{0}=1/n$, then for any $u\in\mathcal{F}_{k}$, we have

$$ $\displaystyle C\sqrt{\frac{M^{2}\rho_{k}\alpha\log(2\beta\sqrt{n})}{n}}$ $\displaystyle\leq C\sqrt{\frac{M^{2}(\rho_{0}+2C_{P}(\mathcal{E}(u)-\mathcal{E }(u^{*})))\alpha\log(2\beta\sqrt{n})}{n}}$ $\displaystyle\leq C\sqrt{\frac{M^{2}\rho_{0}\alpha\log(2\beta\sqrt{n})}{n}}+C \sqrt{\frac{M^{2}2C_{P}(\mathcal{E}(u)-\mathcal{E}(u^{*}))\alpha\log(2\beta \sqrt{n})}{n}}$ $\displaystyle\leq C\sqrt{\frac{M^{2}\rho_{0}\alpha\log(2\beta\sqrt{n})}{n}}+C( \frac{\mathcal{E}(u)-\mathcal{E}(u^{*})}{4C}+\frac{2CM^{2}C_{P}\alpha\log(2 \beta\sqrt{n})}{n})$ $\displaystyle=\frac{\mathcal{E}(u)-\mathcal{E}(u^{*})}{4}+C(\sqrt{\frac{M^{2} \rho_{0}\alpha\log(2\beta\sqrt{n})}{n}}+\frac{M^{2}C_{P}\alpha\log(2\beta\sqrt {n})}{n})$ $\displaystyle\leq\frac{\mathcal{E}(u)-\mathcal{E}(u^{*})}{4}+\frac{CM^{2}C_{P} \alpha\log(2\beta\sqrt{n})}{n},$ $$

where the third inequality follows from the basic inequality $2\sqrt{ab}\leq a+b$ for any $a,b\geq 0$.

Similarly, we can deduce that

$$ $C\sqrt{\frac{M^{2}\rho_{k}\log\frac{1}{\delta_{k}}}{n}}\leq\frac{\mathcal{E}(u )-\mathcal{E}(u^{*})}{4}+\frac{CC_{P}M^{2}(\log\frac{1}{\delta}+\log\log(2enM^ {2}))}{n},$ $$

$$ $\frac{M^{2}\log\frac{1}{\delta_{k}}}{n}\leq\frac{M^{2}(\log\frac{1}{\delta}+ \log\log(2enM^{2}))}{n}$ $$

and

$$ $C\sqrt{\frac{aM^{2}\rho_{k}}{n}\log\frac{4b}{M}}\leq\frac{\mathcal{E}(u)- \mathcal{E}(u^{*})}{4}+\frac{CM^{2}C_{P}a\log\frac{4b}{M}}{n}.$ $$

Combining all results yields that with probability at least $1-\delta_{k}$ for all $u\in\mathcal{F}_{k}$,

$$ $\displaystyle(\mathcal{E}(u)-\mathcal{E}(u^{*}))-4(\mathcal{E}_{n}(u)-\mathcal {E}_{n}(u^{*}))$ $\displaystyle\leq C(\frac{M^{2}C_{P}\alpha\log(2\beta\sqrt{n})}{n}+\frac{CC_{P }M^{2}(\log\frac{1}{\delta}+\log\log(2enM^{2}))}{n}+\frac{M^{2}C_{P}a\log\frac {4b}{M}}{n}.)$ $$

Note that $\sum\limits_{k=1}^{K}\delta_{k}=\delta$, therefore the above inequality holds with probability at least $1-\delta$ uniformly for all $u\in\mathcal{F}$, i.e.

$$ $\displaystyle\sup\limits_{u\in\mathcal{F}}(\mathcal{E}(u)-\mathcal{E}(u^{*}))- 4(\mathcal{E}_{n}(u)-\mathcal{E}(u^{*}))$ $\displaystyle\leq C(\frac{M^{2}C_{P}\alpha\log(2\beta\sqrt{n})}{n}+\frac{C_{P} M^{2}(\log\frac{1}{\delta}+\log\log(2enM^{2}))}{n}+\frac{M^{2}C_{P}a\log\frac{ 4b}{M}}{n}).$ $$

By taking $\lambda=4$ and $\delta=e^{-t}$, we can deduce that with probability at least $1-4e^{-t}$,

$$ $\displaystyle\mathcal{E}(u_{n})-\mathcal{E}(u^{*})$ $\displaystyle\leq C(\frac{M^{2}C_{P}\alpha\log(2\beta\sqrt{n})}{n}+\frac{C_{P} M^{2}(t+\log\log(2enM^{2}))}{n}+\frac{M^{2}C_{P}a\log\frac{4b}{M}}{n}+\epsilon _{app}^{2}+\frac{M^{2}t}{n}).$ $$

From Lemma 14, we know that

(1) when $\mathcal{F}=\mathcal{F}_{m,1}(5\|u_{P}^{*}\|_{\mathcal{B}_{2}(\Omega)})$,

$$ $b=cM,a=cmd,\beta=cM^{2},\alpha=cmd,$ $$

where $c$ is a universal constant.

(2) when $\mathcal{F}=\Phi(N,L,B\|u_{P}^{*}\|_{W^{k,\infty}(\Omega)})$,

$$ $b=Cn,a=CN^{2}L^{2}(\log N\log L)^{3},\beta=Cn,\alpha=CN^{2}L^{2}(\log N\log L) ^{3},$ $$

where $n\geq CN^{2}L^{2}(\log N\log L)^{3}$ and $C$ is a constant independent of $N,L$.

Finally, recall the tensorization of variance:

$$ $Var[f(X_{1},\cdots,X_{n})]\leq\mathbb{E}[\sum\limits_{i=1}^{n}Var_{i}f(X_{1}, \cdots,X_{n})],$ $$

whenever $X_{1},\cdots,X_{n}$ are independent. And for each function $f(x_{1},\cdots,x_{n})$,

$$ $Var_{i}f(x_{1},\cdots,x_{n}):=Var[f(x_{1},\cdots,x_{i-1},X_{i},x_{i+1},\cdots, x_{n})].$ $$

Combining this fact and observation of the product structue of $[0,1]^{d}$ yields that the Poincaré constant is a universal constant.

Hence, the conclusion follows.

###### Remark 1 .

In the proof of Theorem 1, we made an implicit assumption that the empirical processes are measurable. Typically, when considering some empirical process, functions are Lipschitz continuous with respect to the parameters and the parameter space is separable, thus the measurability holds directly. However, in our setting where ReLU neural networks are used in the DRM, the functions fail to satisfy the Lipschitz continuity with respect to the parameters. Thus, it’s necessary to discuss the measurability of the empirical processes. For simplicity, we just consider the two-layer neural networks.

Here, we need the concept of pointwise measurability. Recall that a function class $\mathcal{F}$ of measurable functions in $\mathcal{X}$ is pointwise measurable if there exists a countable subset $\mathcal{G}\subset\mathcal{F}$ such that for every $f\in\mathcal{F}$, there exists a sequence $\{g_{m}\}\in\mathcal{G}$ with $g_{m}(x)\rightarrow f(x)$ for every $x\in\mathcal{X}$ (see Chapter 2.3 in [] or Chapter 8.2 in []).

Note that when applying two-layer neural networks in the DRM, the term $I_{\{\omega\cdot x+t\geq 0\}}$ is not Lipschitz continuous with respect to $\omega$ and $t$. Fortunatelly, we can adapt the proof of Lemma 8.12 in [] to show that the function class is pointwise measurable. Specifically, consider the function class

$$ $\mathcal{G}=\{I_{\{-\omega\cdot x\leq t\}}:\ \omega\in\partial B_{1}^{d}(1) \cap\mathbb{Q}^{d},t\in[-1,1+\epsilon)\cap\mathbb{Q}\},$ $$

where $\mathbb{Q}$ are rationals and $\epsilon$ is a sufficiently small positive number. Note that here, we have slightly expanded the interval $[-1,1]$. It does not influence the conclusion, as in the final, we can let $\epsilon\rightarrow 0$.

Fix $\omega$ and $t$, we can construct $\{(\omega_{m},t_{m})\}$ as follows: pick $\omega_{k}\in\partial B_{1}^{d}(1)\cap\mathbb{Q}^{d}$ such that $|\omega_{m}-\omega|_{1}\leq 1/(2m)$ and pick $t_{m}\in(t+1/(2m),t+1/m]$. Now, for any $x\in[0,1]^{d}$, we have that

$$ $I_{\{-\omega_{m}\cdot x\leq t_{m}\}}=I_{\{-\omega\cdot x\leq t_{m}+(\omega_{m} -\omega)\cdot x\}}.$ $$

Since $|(\omega_{m}-\omega)\cdot x|\leq|\omega_{m}-\omega|_{1}\leq 1/(2m)$, we have that $r_{m}:=t_{m}+(\omega_{m}-\omega)\cdot x-t>0$ for all $m$ and $r_{m}\rightarrow 0$ as $m\rightarrow\infty$. Note that the function $t\mapsto I_{\{u\leq t\}}$ is right-continuous, so that $I_{\{-\omega_{m}\cdot x\leq t_{m}\}}\rightarrow I_{\{-\omega\cdot x\leq t\}}$ for all $x\in[0,1]^{d}$. Thus, the pointwise measurability is established.

Therefore, for the function class of two-layer neural networks $\mathcal{F}_{m,1}(B)$,

$$ $\mathcal{F}_{m,1}(B)=\{\sum\limits_{i=1}^{m}\gamma_{i}\sigma(\omega_{i}\cdot x +t_{i}):\ |\omega_{i}|_{1}=1,t_{i}\in[-1,1+\epsilon),\sum\limits_{i=1}^{m}| \gamma_{i}|\leq B\},$ $$

we can pick $\gamma_{i},\omega_{i},t_{i}$ to be rationals. To prove the measurability for the empirical process $\sup_{f\in\mathcal{F}}(P-P_{n})f$ where $\mathcal{F}$ is related to ReLU functions and their gradients, it remains to focus on the term $Pf$.

Note that for $u,\hat{u}\in\mathcal{F}_{m,1}(B)$ with the forms

$$ $u(x)=\sum\limits_{i=1}^{m}\gamma_{i}\sigma(\omega_{i}\cdot x+t_{i}),\hat{u}(x) =\sum\limits_{i=1}^{m}\hat{\gamma_{i}}\sigma(\hat{\omega_{i}}\cdot x+\hat{t_{i }}),$ $$

we have that

$$ $\displaystyle|P(|\nabla u|^{2}-2fu)-P(|\nabla\hat{u}|^{2}-2f\hat{u})|$ $\displaystyle\lesssim P|\nabla u-\nabla\hat{u}|+P|u-\hat{u}|$ $\displaystyle\lesssim\sum\limits_{i=1}^{m}|\gamma_{i}-\hat{\gamma_{i}}|+| \omega_{i}-\hat{\omega_{i}}|_{1}+|t_{i}-\hat{t}_{i}|+P|I_{\{\omega_{i}\cdot x+ t_{i}\geq 0\}}-I_{\{\hat{\omega_{i}}\cdot x+\hat{t_{i}}\geq 0\}}|.$ $$

The dominated convergence theorem implies that

$$ $P|I_{\{\omega\cdot x+t\geq 0\}}-I_{\{\omega_{m}\cdot x+t_{m}\geq 0\}}| \rightarrow 0.$ $$

Therefore, with a little abuse of notation, we have $\sup_{f\in\mathcal{F}}(P-P_{n})f=\sup_{f\in\mathcal{G}}(P-P_{n})f$, which implies that the empirical processes in the proof of Theorem 1 are measurable, as the parameters in $\mathcal{F}$ can be replaced by rationals.

###### Remark 1 .

### 5.3. Proof of Theorem 2

For the static Schrödinger equation, we can also use the method in the proof of Theorem 1 or other methods in [][], due to the similarity between the problem and the generalization error of $L^{2}$ regression with bounded noise. However, the methods mentioned are quite complex. Here, we present a simple proof through a different error decomposition and LRC, which can be easily adapted for other problems with the strongly convex structures.

As before, through the proof, we write $\mathcal{E}$ and $\mathcal{E}_{n}$ for the population loss $\mathcal{E}_{S}$ and empirical loss $\mathcal{E}_{n,S}$ respectively. Additionally, we assume that
$|u^{*}|,|\nabla u^{*}|,|V|,|f|\leq M$ for a positive constant $M$.

Recall that

$$ $u^{*}=\mathop{\arg\min}\limits_{u\in H^{1}(\Omega)}\mathcal{E}(u):=\int_{ \Omega}|\nabla u|^{2}+V|u|^{2}dx-2\int_{\Omega}fudx$ $$

and $u_{n}$ is the minimal solution to the empirical loss $\mathcal{E}_{n}$ in the function class $\mathcal{F}$. We also assume that $\sup_{u\in\mathcal{F}}|u|$, $\sup_{u\in\mathcal{F}}|\nabla u|\leq M$.

Through a error decomposition similar as that for the Poisson equation, we have

$$ $\displaystyle\mathcal{E}(u_{n})-\mathcal{E}(u^{*})$ $\displaystyle=\mathcal{E}(u_{n})-\lambda\mathcal{E}_{n}(u_{n})+\lambda( \mathcal{E}_{n}(u_{n})-\mathcal{E}_{n}(u_{\mathcal{F}}))+\lambda\mathcal{E}_{n }(u_{\mathcal{F}})-\mathcal{E}(u^{*})$ $\displaystyle\leq\mathcal{E}(u_{n})-\lambda\mathcal{E}_{n}(u_{n})+\lambda \mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}(u^{*})$ $\displaystyle=\mathcal{E}(u_{n})-\lambda\mathcal{E}_{n}(u_{n})+\lambda( \mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}_{n}(u^{*}))+\lambda\mathcal{E}_{n }(u^{*})-\mathcal{E}(u^{*})$ $\displaystyle=(\mathcal{E}(u_{n})-\mathcal{E}(u^{*}))-\lambda(\mathcal{E}_{n}( u_{n})-\mathcal{E}_{n}(u^{*}))+\lambda(\mathcal{E}_{n}(u_{\mathcal{F}})- \mathcal{E}_{n}(u^{*}))$ $\displaystyle\leq\sup\limits_{u\in\mathcal{F}}[(\mathcal{E}(u)-\mathcal{E}(u^{ *}))-\lambda(\mathcal{E}_{n}(u)-\mathcal{E}_{n}(u^{*}))]+\lambda(\mathcal{E}_{ n}(u_{\mathcal{F}})-\mathcal{E}_{n}(u^{*})),$ $$

where the first inequality follows from the definition of $u_{n}$ and $\lambda$ is a constant to be determined.

Let $\epsilon_{app}:=\|u_{\mathcal{F}}-u^{*}\|_{H^{1}(\Omega)}$ be the approximation error.

By Bernstein inequality, with probability at least $1-e^{-t}$

$$ $(\mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}_{n}(u^{*}))-(\mathcal{E}(u_{ \mathcal{F}})-\mathcal{E}(u^{*}))\leq\sqrt{\frac{2tVar(g)}{n}}+\frac{t\|g\|_{L ^{\infty}}}{3n},$ $$

where

$$ $g(x):=(|\nabla u_{\mathcal{F}}|^{2}+V(x)|u_{\mathcal{F}}(x)|^{2}-2f(x)u_{ \mathcal{F}}(x))-(|\nabla u^{*}(x)|^{2}+V(x)|u^{*}(x)|^{2}-2f(x)u^{*}(x)).$ $$

From the boundedness of $u_{\mathcal{F}},u^{*},f$ and $V$, we can deduce that $|g|\leq 8M^{2}$ and

$$ $Var(g)\leq Pg^{2}\leq cM^{2}\|u_{\mathcal{F}}-u^{*}\|_{H^{1}(\Omega)}^{2}=cM^{ 2}\epsilon_{app}^{2}.$ $$

Therefore, with probability at least $1-e^{-t}$

$$ $\displaystyle\mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}_{n}(u^{*})$ $\displaystyle\leq c\epsilon_{app}^{2}+\sqrt{\frac{2tcM^{2}\epsilon_{app}^{2}}{ n}}+\frac{8tM^{2}}{3n}$ $\displaystyle\leq c(\epsilon_{app}^{2}+\frac{tM^{2}}{n}),$ $$

where the first inequality follows from Proposition 1(2) and the second inequality follows from the mean inequality.

Plugging it into the error decomposition yields

$$ $\mathcal{E}(u_{n})-\mathcal{E}(u^{*})\leq\sup\limits_{u\in\mathcal{F}}[( \mathcal{E}(u)-\mathcal{E}(u^{*}))-\lambda(\mathcal{E}_{n}(u)-\mathcal{E}_{n}( u^{*}))]+\lambda c(\epsilon_{app}^{2}+\frac{M^{2}t}{n})$ $$

with probability at least $1-e^{-t}$.

Note that $(\mathcal{E}(u)-\mathcal{E}(u^{*}))-\lambda(\mathcal{E}_{n}(u)-\mathcal{E}_{n}
(u^{*}))$ can be rewritten as

$$ $(\mathcal{E}(u)-\mathcal{E}(u^{*}))-\lambda(\mathcal{E}_{n}(u)-\mathcal{E}_{n} (u^{*}))=Ph-\lambda P_{n}h,$ $$

where $h(x):=(|\nabla u(x)|^{2}+V(x)|u(x)|^{2}-2f(x)u(x))-(|\nabla u^{*}(x)|^{2}+V(x)
|u^{*}(x)|^{2}-2f(x)u^{*}(x))$. And the form motivates the use of LRC.

We begin by defining the function class

$$ $\mathcal{H}:=\{(|\nabla u(x)|^{2}+V(x)|u(x)|^{2}-2f(x)u(x))-(|\nabla u^{*}(x)| ^{2}+V(x)|u^{*}(x)|^{2}-2f(x)u^{*}(x)):\ u\in\mathcal{F}\}.$ $$

To invoke the LRC, we define a functional on $\mathcal{H}$ as $T(h):=Ph^{2}$. It is easy to check that

$$ $Var(h)\leq T(h)\leq cM^{2}Ph,$ $$

as $Ph^{2}\leq cM^{2}\|u-u^{*}\|_{H^{1}(\Omega)}^{2}\leq cM^{2}(\mathcal{E}(u)-
\mathcal{E}(u^{*}))=cM^{2}Ph$.
It implies that the functional $T$ satisfies the condition of Theorem 3.3 in [].

Following the procedure of Theorem 3.3 in [], we are going to seek a sub-root function and compute its fixed point.

Define the sub-root function

$$ $\psi(r):=80M^{2}\mathbb{E}\mathcal{R}_{n}(h\in star(\mathcal{H},0):Ph^{2}\leq r )+704\frac{M^{4}\log n}{n},$ $$

where invoking the star-hull of $\mathcal{H}$ around 0 0 is to make $\psi$ to be a sub-root function.

Next, our goal is to bound the fixed point of $\psi$.

If $r\geq\psi(r)$, then Corollary 2.2 in [] implies that, with probability at least $1-\frac{1}{n}$,

$$ $\{h\in star(\mathcal{H},0):Ph^{2}\leq r\}\subset\{h\in star(\mathcal{H},0):P_{ n}h^{2}\leq 2r\},$ $$

and thus

$$ $\mathbb{E}\mathcal{R}_{n}(h\in star(\mathcal{H},0):Ph^{2}\leq r)\leq\mathbb{E} \mathcal{R}_{n}(h\in star(\mathcal{H},0):P_{n}h^{2}\leq 2r)+\frac{8M^{2}}{n}.$ $$

Assume that $r^{*}$ is the fixed point of $\psi$, then

$$ $r^{*}=\psi(r^{*})\leq cM^{2}\mathbb{E}\mathcal{R}_{n}(h\in star(\mathcal{H},0) :P_{n}h^{2}\leq 2r^{*})+c\frac{M^{4}\log n}{n},$ $$

where we use a universal constant $c$ to represent the upper bound for the constants in $\psi(r)$.

To estimate the first term, we need the assumption about the covering number of $\mathcal{H}$.

###### Assumption 1 .

For any $\epsilon>0$, assume that

$$ $\mathcal{N}(\mathcal{H},L_{2}(P_{n}),\epsilon)\leq(\frac{\beta}{\epsilon})^{ \alpha}\ a.s.,$ $$

for some constant $\beta>\sup_{h\in\mathcal{H}}|h|$.

Then by Dudley’s entropy integral ineuqality,

$$ $\displaystyle\mathbb{E}\mathcal{R}_{n}(h\in star(\mathcal{H},0):P_{n}h^{2}\leq 2 r^{*})$ $\displaystyle\leq\frac{c}{\sqrt{n}}\mathbb{E}\int_{0}^{\sqrt{2r^{*}}}\sqrt{ \log\mathcal{N}(\epsilon,star(\mathcal{H},0),L_{2}(P_{n}))}d\epsilon$ $\displaystyle\leq\frac{c}{\sqrt{n}}\mathbb{E}\int_{0}^{\sqrt{2r^{*}}}\sqrt{ \log\mathcal{N}(\frac{\epsilon}{2},\mathcal{H},L_{2}(P_{n}))(\frac{2}{\epsilon }+1)}d\epsilon$ $\displaystyle\leq c\sqrt{\frac{\alpha}{n}}\int_{0}^{\sqrt{2r^{*}}}\sqrt{\log( \frac{\beta}{\epsilon})}d\epsilon$ $\displaystyle=c\beta\sqrt{\frac{\alpha}{n}}\int_{0}^{\frac{\sqrt{2r^{*}}}{ \beta}}\sqrt{\log(\frac{1}{\epsilon})}d\epsilon$ $\displaystyle\leq c\sqrt{\frac{\alpha}{n}}\sqrt{r^{*}\log(\frac{\beta}{\sqrt{r ^{*}}})}$ $\displaystyle\leq c\sqrt{\frac{\alpha}{n}}\sqrt{r^{*}\log(\frac{\sqrt{n}\beta} {M^{2}})},$ $$

where the fourth inequality follows from Lemma 12 and the last ineuqality follows by the fact that $r^{*}\geq c\frac{M^{4}\log n}{n}$.

Therefore,

$$ $r^{*}\leq cM^{2}\sqrt{\frac{\alpha}{n}}\sqrt{r^{*}\log(\frac{\sqrt{n}\beta}{M^ {2}})}+c\frac{M^{4}\log n}{n},$ $$

which implies

$$ $r^{*}\leq cM^{4}(\frac{\alpha}{n}\log(\frac{\sqrt{n}\beta}{M^{2}})+\frac{\log n }{n}).$ $$

The final step is to estimate the covering numbers of the function classes of two-layer neural networks and deep neural networks, i.e. to determine $\alpha$ and $\beta$ for $\mathcal{F}=\mathcal{F}_{m}(5\|u_{S}^{*}\|_{\mathcal{B}^{2}(\Omega)})$ and $\mathcal{F}=\Phi(N,L,B\|u_{S}^{*}\|_{W^{1,\infty}(\Omega)})$.

(1) When $\mathcal{F}=\mathcal{F}_{m,1}(5\|u_{S}^{*}\|_{\mathcal{B}^{2}(\Omega)})$, estimation of the covering number of $\mathcal{H}$ is almost same as the estimation of $\mathcal{G}$ for the two-layer neural networks in Lemma 14 (1). It is not difficult to deduce that $\alpha=cmd$, $\beta=cM^{2}$. For simplicity, we omit the proof.

(2) When $\mathcal{F}=\Phi(N,L,B\|u_{S}^{*}\|_{W^{1,\infty}(\Omega)})$, we can also deduce that $\alpha=CN^{2}L^{2}(\log N\log L)^{3},\beta=Cn$ by a similar method as that in Lemma 14 (2).

As a result, the conclusion holds.

###### Assumption 1 .

## 6. Proof of Section 3

### 6.1. Proof of Proposition 3

###### Proof.

(1) The proof mainly follows the procedure in the proof the Proposition 2 (1), but the tools from the FEM may not work for $\text{ReLU}^{2}$ functions. Therefore, we trun to use Taylor’s theorem with integral remainder, which enables us to establish a connection between the one-dimensional $C^{2}$ functions and the $\text{ReLU}^{2}$ functions. The method has been also used in [, ].

Recall that Taylor’s theorem with integral remainder states that for $f:\mathbb{R}\rightarrow\mathbb{R}$ that has $k+1$ continuous derivatives in some neighborhood $U$ of $x=a$, then for $x\in U$

$$ $f(x)=f(a)+f^{{}^{\prime}}(a)(x-a)+\cdots+\frac{f^{(k)}(a)}{k!}(x-a)^{k}+\int_{ a}^{x}f^{(k+1)}(t)\frac{(x-t)^{k}}{k!}dt.$ $$

For

$$ $g(z)=\frac{B\cos(|\omega|_{1}z+b)}{(1+|\omega|_{1})^{3}}=\frac{B(\cos(|\omega| _{1}z)\cos b-\sin(|\omega|_{1}z)\sin b)}{(1+|\omega|_{1})^{3}},$ $$

applying Taylor’s theorem with integral remainder for $\cos(|\omega|_{1}z)$ and $\sin(|\omega|_{1}z)$ at the point 0 0, we have

$$ $\cos(|\omega|_{1}z)=1-\frac{|\omega|_{1}^{2}}{2}z^{2}+\int_{0}^{z}|\omega|_{1} ^{3}\sin(|\omega|_{1}t)\frac{(z-t)^{2}}{2}dt$ $$

and

$$ $\sin(|\omega|_{1}z)=|\omega|_{1}z-\int_{0}^{z}|\omega|_{1}^{3}\cos(|\omega|_{1 }t)\frac{(z-t)^{2}}{2}dt.$ $$

For the integral remainder, we can consider the general form $h(z)=\int_{0}^{z}f(t)(z-t)^{2}dt$. By the fact that $(z-t)^{2}=(z-t)_{+}^{2}+(t-z)_{+}^{2}$,

$$ $\int_{0}^{z}f(t)(z-t)^{2}dt=\int_{0}^{z}f(t)(z-t)_{+}^{2}dt+\int_{0}^{z}f(t)(- z+t)_{+}^{2}dt:=A_{1}+A_{2}$ $$

In the next, we aim to prove that

$$ $A_{1}+A_{2}=\int_{0}^{1}f(t)(z-t)_{+}^{2}dt-\int_{0}^{1}f(-t)(-z-t)_{+}^{2}dt: =B_{1}+B_{2},$ $$

which enables the method used in the proof of Proposition 1(1) to be feasible.

(1) When $z\geq 0$, it is easy to obtain that

$$ $A_{1}=\int_{0}^{z}f(t)(z-t)_{+}^{2}dt=\int_{0}^{1}f(t)(z-t)_{+}^{2}dt=B_{1},\ and\ A_{2}=0,B_{2}=0.$ $$

Therefore, $A_{1}+A_{2}=B_{1}+B_{2}$.

(2) When $z<0$, it is easy to check that $A_{1}=B_{1}=0$. Therefore, it remains only to check that $A_{2}=B_{2}$.

For $A_{2}$, we can deduce that

$$ $\displaystyle A_{2}$ $\displaystyle=\int_{0}^{z}f(t)(-z+t)_{+}^{2}dt$ $\displaystyle=-\int_{z}^{0}f(t)(-z+t)_{+}^{2}dt$ $\displaystyle=-[\int_{z}^{0}f(t)(-z+t)_{+}^{2}dt+\int_{-1}^{z}f(t)(-z-t)_{+}^{ 2}dt]$ $\displaystyle=-\int_{-1}^{0}f(t)(-z+t)_{+}^{2}dt$ $\displaystyle=-\int_{0}^{1}f(-y)(-z-y)_{+}^{2}dy=B_{2},$ $$

where the third equality follows by that $\int_{-1}^{z}f(t)(-z-t)_{+}^{2}dt$=0 and the fifth equality is due to the variable substitution $t=-y$.

Combining (1) and (2), we can deduce that

$$ $h(z)=\int_{0}^{z}f(t)(z-t)^{2}dt=\int_{0}^{1}f(t)(z-t)_{+}^{2}dt-\int_{0}^{1}f (-t)(-z-t)_{+}^{2}dt.$ $$

The next step is to prove that $h$ is the $H^{2}$ closure of convex hull of $\text{ReLU}^{2}$ functions.

Let $h_{1}(z)=\int_{0}^{1}f(t)(z-t)_{+}^{2}dt$, $h_{2}(z)=\int_{0}^{1}f(t)(-z-t)_{+}^{2}dt$, then $h(z)=h_{1}(z)-h_{2}(z)$.

Notice that $h_{1}^{{}^{\prime}}(z)=\int_{0}^{1}2f(t)(z-t)_{+}dt$ and $h_{1}^{{}^{\prime\prime}}(z)=\int_{0}^{1}2f(t)I_{\{z-t\geq 0\}}dt$ a.e., since $(z-t)_{+}$ is differentiable a.e. .

Let $\{t_{i}\}_{i=1}^{n}$ be an i.i.d. sequence of random variables distributed according the uniform distribution of the interval $[0,1]$, then by Fubini’s theorem

$$ $\displaystyle\mathbb{E}\|h_{1}(z)-\sum\limits_{i=1}^{n}\frac{f(t_{i})(z-t_{i}) _{+}^{2}}{n}\|_{H^{2}([-1,1])}^{2}$ $\displaystyle=\int_{-1}^{1}\mathbb{E}[|h_{1}(z)-\sum\limits_{i=1}^{n}\frac{f(t _{i})(z-t_{i})_{+}^{2}}{n}|^{2}+|h_{1}^{{}^{\prime}}(z)-\sum\limits_{i=1}^{n} \frac{2f(t_{i})(z-t_{i})_{+}}{n}|^{2}++|h_{1}^{{}^{\prime\prime}}(z)-\sum \limits_{i=1}^{n}\frac{2f(t_{i})I_{\{z-t_{i}\geq 0\}}}{n}|^{2}]dz$ $\displaystyle=\int_{-1}^{1}\frac{Var(f(\cdot)(z-\cdot)_{+}^{2})+Var(2f(\cdot)( z-\cdot)_{+})+Var(2f(\cdot)I_{\{z-\cdot\geq 0\}})}{n}dz$ $\displaystyle\leq\frac{C}{n},$ $$

where the last ineuqality follows from the boundedness of $f$. And the same conclusion also holds for $h_{2}(z)$ and $h(z)$. Therefore, we can deduce that $h$ is in the $H^{2}([-1,1])$ closure of convex hull of the function class $\mathcal{F}_{2}(cB)\cup\mathcal{F}_{2}(-cB)\cup\{0\},$
where

$$ $\mathcal{F}_{2}(B):=\{B\sigma_{2}(\epsilon z+t):\epsilon=-1\ or\ 1,t\in[-1,1]\}$ $$

and $c$ is a universal constant.

Observe that the linear function $z$ and constant can be represented by two-layer $\sigma_{2}$ neural networks, since $z=\frac{(z+1)^{2}-(z-1)^{2}}{4}$, $1=(z+1)-z$.

Therefore, we can deduce that $\frac{B\cos b\cos(|\omega|_{1}z)}{(1+|\omega|_{1})^{3}}$ and $\frac{B\sin b\sin(|\omega|_{1}z)}{(1+|\omega|_{1})^{3}}$ are also in the $H^{2}([-1,1])$ closure of convex hull of the function class $\mathcal{F}_{2}(cB)\cup\mathcal{F}_{2}(-cB)\cup\{0\}.$ Finally, the conclusion holds for $g$.

Then the variable substitution yields that for any $f\in\mathcal{B}^{3}(\Omega)$ and $\epsilon>0$, there exists a two-layer $\sigma_{2}$ neural network such that

$$ $\|f(x)-\sum\limits_{i=1}^{m}a_{i}\sigma_{2}(\omega_{i}\cdot x+t_{1})\|_{H^{2}( \Omega)}\leq\epsilon,$ $$

where $|\omega_{i}|_{1}=1,|t_{i}|\leq 1,\sum\limits_{i=1}^{m}|a_{i}|\leq c\|f\|_{
\mathcal{B}^{3}(\Omega)}$ and $c$ is a universal constant.

Just as the proof of Proposition 2(1), it remains only to estimate the metric entropy of the function class

$$ $\mathcal{F}_{2}:=\{\sigma_{2}(\omega\cdot x+t):|\omega|_{1}=1,t\in[-1,1]\}$ $$

under the $H^{2}$ norm.

For $(\omega_{1},t_{1}),(\omega_{2},t_{2})\in\partial B_{1}^{d}(1)\times[-1,1]$, we have

$$ $\displaystyle\|\sigma_{2}(\omega_{1}\cdot x+t_{1})-\sigma_{2}(\omega_{2}\cdot x +t_{2})\|_{H^{2}(\Omega)}^{2}$ $\displaystyle=\|\sigma_{2}(\omega_{1}\cdot x+t_{1})-\sigma_{2}(\omega_{2}\cdot x +t_{2})\|_{L^{2}(\Omega)}^{2}+\||2\omega_{1}\sigma(\omega_{1}\cdot x+t_{1})-2 \omega_{2}\sigma(\omega_{2}\cdot x+t_{2})|\|_{L^{2}(\Omega)}^{2}$ $\displaystyle\quad+\sum\limits_{i=1}^{d}\sum\limits_{j=1}^{d}\|2\omega_{1i} \omega_{1j}I_{\{\omega_{1}\cdot x+t_{1}\geq 0\}}-2\omega_{2i}\omega_{2j}I_{\{ \omega_{2}\cdot x+t_{2}\geq 0\}}\|_{L^{2}(\Omega)}^{2}$ $\displaystyle:=(i)+(ii)+(iii),$ $$

where we denote the $i$-th element of the vector $\omega_{k}$ by $\omega_{ki}$ for $k=1,2$, $1\leq i\leq d$.

For $(i)$, since $\sigma_{2}$ is 4-Lipschitz in $[-2,2]$,

$$ $\displaystyle(i)$ $\displaystyle=\|\sigma_{2}(\omega_{1}\cdot x+t_{1})-\sigma_{2}(\omega_{2}\cdot x +t_{2})\|_{L^{2}(\Omega)}^{2}$ $\displaystyle\leq 4\|(\omega_{1}-\omega_{2})\cdot x+(t_{1}-t_{2})\|_{L^{2}( \Omega)}^{2}$ $\displaystyle\leq 8(|\omega_{1}-\omega_{2}|_{1}^{2}+|t_{1}-t_{2}|^{2}).$ $$

For $(ii)$,

$$ $\displaystyle(ii)$ $\displaystyle=\||2\omega_{1}\sigma(\omega_{1}\cdot x+t_{1})-2\omega_{2}\sigma( \omega_{2}\cdot x+t_{2})|\|_{L^{2}(\Omega)}^{2}$ $\displaystyle=2\||(\omega_{1}-\omega_{2})\sigma(\omega_{1}\cdot x+t_{1})+ \omega_{2}(\sigma(\omega_{1}\cdot x+t_{1})-\sigma(\omega_{2}\cdot x+t_{2}))|\| \|_{L^{2}(\Omega)}^{2}$ $\displaystyle\leq 4\||(\omega_{1}-\omega_{2})\sigma(\omega_{1}\cdot x+t_{1})| \|_{L^{2}(\Omega)}^{2}+4\||\omega_{2}(\sigma(\omega_{1}\cdot x+t_{1})-\sigma( \omega_{2}\cdot x+t_{2}))|\|_{L^{2}(\Omega)}^{2}$ $\displaystyle\leq 16|\omega_{1}-\omega_{2}|_{1}^{2}+8(|\omega_{1}-\omega_{2}|_ {1}^{2}+|t_{1}-t_{2}|^{2}),$ $$

where the first inequality follows from the mean ineuqality and the boundedness of $\sigma$.

For $(iii)$,

$$ $\displaystyle(iii)$ $\displaystyle=\sum\limits_{i=1}^{d}\sum\limits_{j=1}^{d}\|2\omega_{1i}\omega_{ 1j}I_{\{\omega_{1}\cdot x+t_{1}\geq 0\}}-2\omega_{2i}\omega_{2j}I_{\{\omega_{2 }\cdot x+t_{2}\geq 0\}}\|_{L^{2}(\Omega)}^{2}$ $\displaystyle=4\sum\limits_{i=1}^{d}\sum\limits_{j=1}^{d}\|(\omega_{1i}\omega_ {1j}-\omega_{2i}\omega_{2j})I_{\{\omega_{1}\cdot x+t_{1}\geq 0\}}+\omega_{2i} \omega_{2j}(I_{\{\omega_{1}\cdot x+t_{1}\geq 0\}}-I_{\{\omega_{2}\cdot x+t_{2} \geq 0\}})\|_{L^{2}(\Omega)}^{2}$ $\displaystyle\leq 8\sum\limits_{i=1}^{d}\sum\limits_{j=1}^{d}|\omega_{1i} \omega_{1j}-\omega_{2i}\omega_{2j}|^{2}+(\omega_{2i}\omega_{2j})^{2}\|I_{\{ \omega_{1}\cdot x+t_{1}\geq 0\}}-I_{\{\omega_{2}\cdot x+t_{2}\geq 0\}}\|_{L^{2 }(\Omega)}^{2}$ $\displaystyle\leq 8\sum\limits_{i=1}^{d}\sum\limits_{j=1}^{d}|\omega_{1i}- \omega_{2i}|^{2}|\omega_{1j}|^{2}+|\omega_{1j}-\omega_{2j}|^{2}|\omega_{2i}|^{ 2}+(\omega_{2i}\omega_{2j})^{2}\|I_{\{\omega_{1}\cdot x+t_{1}\geq 0\}}-I_{\{ \omega_{2}\cdot x+t_{2}\geq 0\}}\|_{L^{2}(\Omega)}^{2}$ $\displaystyle\leq 16|\omega_{1}-\omega_{2}|_{1}^{2}+8\|I_{\{\omega_{1}\cdot x+ t_{1}\geq 0\}}-I_{\{\omega_{2}\cdot x+t_{2}\geq 0\}}\|_{L^{2}(\Omega)}^{2},$ $$

where the last inequality follows from the fact that $|\omega_{1}|\leq|\omega_{1}|_{1}=1,|\omega_{2}|\leq|\omega_{2}|_{1}=1$.

Combining the upper bounds for $(i),(ii),(iii)$, we obtain that

$$ $\|\sigma_{2}(\omega_{1}\cdot x+t_{1})-\sigma_{2}(\omega_{2}\cdot x+t_{2})\|_{H ^{2}(\Omega)}^{2}\leq 48(|\omega_{1}-\omega_{2}|_{1}^{2}+|t_{1}-t_{2}|^{2})+8 \|I_{\{\omega_{1}\cdot x+t_{1}\geq 0\}}-I_{\{\omega_{2}\cdot x+t_{2}\geq 0\}} \|_{L^{2}(\Omega)}^{2}.$ $$

Therefore, based on the same method used in the proof of Proposition 6, we can deduce that

$$ $\epsilon_{n}(\mathcal{F}_{2})\leq cn^{-\frac{1}{3d}}.$ $$

Finally, applying Theorem 1 in [] yields the conclusion for $f$ in $\mathcal{B}^{3}(\Omega)$.

(2) Recall that based on the spline theory, [] has demonstrated the approximation rates for Hölder continous functions with sparse $\text{ReLU}^{2}$ neural networks. Then [] extended these results for sparse $\text{ReLU}^{3}$ neural networks. In fact, the approximation results also hold for Sobolev functions, we only need to replace the Theorem 3 in [] with the results from [] on approximating Sobolev functions with multivariate splines. For simplicity, we omit the proof. ∎

###### Proof.

### 6.2. Proof of Theorem 3

###### Proof.

Assume that $\sup\limits_{1\leq t\leq T}\sup\limits_{x\in\mathcal{X}_{t}}|f_{t}(x)|\leq b$ and $\frac{1}{T}\sup\limits_{\bm{f}\in\bm{\mathcal{F}}}\sum\limits_{t=1}^{T}Var(f_{
t}(X_{t}^{1}))\leq r$.

Let

$$ $Z:=\sup\limits_{\bm{f}\in\bm{\mathcal{F}}}\frac{1}{T}\sum\limits_{t=1}^{T}Pf_{ t}-P_{n}f_{t}=\sup\limits_{\bm{f}\in\bm{\mathcal{F}}}\frac{1}{T}\sum\limits_{t =1}^{T}\frac{1}{N_{t}}\sum\limits_{i=1}^{N_{t}}f_{t}(X_{t}^{i})-\mathbb{E}f_{t }(X_{t}^{i})$ $$

and

$$ $F(x):=\frac{1}{2b}\sup\limits_{\bm{f}\in\bm{\mathcal{F}}}\sum\limits_{t=1}^{T} \frac{n}{N_{t}}\sum\limits_{i=1}^{N_{t}}f_{t}(x_{t}^{i})-\mathbb{E}f_{t}(X_{t} ^{i}),$ $$

where $n=\min\limits_{1\leq i\leq T}N_{t}$ and $x=(x_{1}^{1},\cdots,x_{t}^{i},\cdots,x_{T}^{N_{t}})$.

$$ $W(x):=\frac{1}{4b^{2}}\sup\limits_{\bm{f}\in\bm{\mathcal{F}}}\sum\limits_{t=1} ^{T}\frac{n^{2}}{N_{t}^{2}}\sum\limits_{i=1}^{N_{t}}(f_{t}(x_{t}^{i})-\mathbb{ E}f_{t}(X_{t}^{i}))^{2}+\mathbb{E}(f_{t}(X_{t}^{i})-\mathbb{E}f_{t}(X_{t}^{i}) )^{2}.$ $$

Similar to Theorem 38 in [],
fix $(x_{t,i})_{1\leq t\leq T,1\leq i\leq n}$ and assume that the maximum in the definition of $F$ is achieved at $\hat{\bm{f}}=(\hat{f}_{1},\cdots,\hat{f}_{T})\in\bm{\mathcal{F}}$.

Then for any $y$, $(F-S_{y}^{t,i}F)_{+}\leq\frac{n}{2bN_{t}}(\hat{f}_{t}(x_{t,i})-\hat{f}_{t}(y))
_{+}$, where

$$ $S_{y}^{k}(G)(y_{1},\cdots,y_{n}):=G(y_{1},\cdots,y_{k-1},y,y_{k+1},\cdots,y_{n})$ $$

for any function $G$.

Therefore,

$$ $\displaystyle V_{+}^{2}F(x)$ $\displaystyle=\sum\limits_{t=1}^{T}\sum\limits_{i=1}^{N_{t}}\mathbb{E}_{y\sim \mu_{t,i}}[((F-S_{y}^{t,i}F)_{+})^{2}]$ $\displaystyle\leq\frac{1}{4b^{2}}\sum\limits_{t=1}^{T}\frac{n^{2}}{N_{t}^{2}} \sum\limits_{i=1}^{N_{t}}\mathbb{E}_{y\sim\mu_{t,i}}(\hat{f}_{t}(x_{t}^{i})- \hat{f}_{t}(y))_{+}^{2}$ $\displaystyle\leq\frac{1}{4b^{2}}\sum\limits_{t=1}^{T}\frac{n^{2}}{N_{t}^{2}} \sum\limits_{i=1}^{N_{t}}\mathbb{E}_{y\sim\mu_{t,i}}(\hat{f}_{t}(x_{t}^{i})- \hat{f}_{t}(y))^{2}$ $\displaystyle=\frac{1}{4b^{2}}\sum\limits_{t=1}^{T}\frac{n^{2}}{N_{t}^{2}}\sum \limits_{i=1}^{N_{t}}\mathbb{E}_{y\sim\mu_{t,i}}(\hat{f}_{t}(x_{t}^{i})- \mathbb{E}\hat{f}_{t}(X_{t}^{i})-(\hat{f}_{t}(y)-\mathbb{E}\hat{f}_{t}(X_{t}^{ i})))^{2}$ $\displaystyle=\frac{1}{4b^{2}}\sum\limits_{t=1}^{T}\frac{n^{2}}{N_{t}^{2}}\sum \limits_{i=1}^{N_{t}}(\hat{f}_{t}(x_{t}^{i})-\mathbb{E}\hat{f}_{t}(X_{t}^{i})) ^{2}+\mathbb{E}(\hat{f}_{t}(X_{t}^{i})-\mathbb{E}\hat{f}_{t}(X_{t}^{i}))^{2}$ $\displaystyle\leq W,$ $$

where $X_{t}^{i}$ follows the distribution $\mu_{t}^{i}$, i.e. $\mu_{t}^{i}=\mu_{t}$.

Therefore, $V_{+}^{2}(F)\leq W$. Then equation (26) in [] yields that

$$ $Ent_{F}(\gamma)\leq\frac{\gamma}{2-\gamma}\log\mathbb{E}e^{\gamma V_{+}^{2}(F) }\leq\frac{\gamma}{2-\gamma}\log\mathbb{E}e^{\gamma V_{+}^{2}(W)}.$ $$

Next, we are going the prove that $W$ is self-bounding, so that Lemma 32 (i) in [] can be applied. Assume that the maximum in the definition of $W$ is achieved at $\bar{\bm{f}}=(\bar{f}_{1},\cdots,\bar{f}_{T})\in\bm{\mathcal{F}}$, then for any $y$,

$$ $(W-S_{y}^{t,i}W)_{+}\leq\frac{n^{2}}{4b^{2}N_{t}^{2}}((\bar{f}_{t}(x_{t}^{i})- \mathbb{E}\bar{f}_{t}(X_{t}^{i}))^{2}-(\bar{f}_{t}(y)-\mathbb{E}\bar{f}_{t}(X_ {t}^{i}))^{2})_{+},$ $$

therefore

$$ $\displaystyle V_{+}^{2}W(x)$ $\displaystyle=\sum\limits_{t=1}^{T}\sum\limits_{i=1}^{N_{t}}\mathbb{E}_{y\sim \mu_{t,i}}(W(x)-S_{y}^{t,i}W(x))_{+}^{2}$ $\displaystyle\leq\frac{1}{16b^{4}}\sum\limits_{t=1}^{T}\frac{n^{4}}{N_{t}^{4}} \sum\limits_{i=1}^{N_{t}}\mathbb{E}_{y\sim\mu_{t,i}}((\bar{f}_{t}(x_{t}^{i})- \mathbb{E}\bar{f}_{t}(X_{t}^{i}))^{2}-(\bar{f}_{t}(y)-\mathbb{E}\bar{f}_{t}(X_ {t}^{i}))^{2})_{+}^{2}$ $\displaystyle\leq\frac{1}{16b^{4}}\sum\limits_{t=1}^{T}\frac{n^{4}}{N_{t}^{4}} \sum\limits_{i=1}^{N_{t}}(\bar{f}_{t}(x_{t}^{i})-\mathbb{E}\bar{f}_{t}(X_{t}^{ i}))^{4}$ $\displaystyle\leq\frac{1}{4b^{2}}\sum\limits_{t=1}^{T}\frac{n^{2}}{N_{t}^{2}} \sum\limits_{i=1}^{N_{t}}(\bar{f}_{t}(x_{t}^{i})-\mathbb{E}\bar{f}_{t}(X_{t}^{ i}))^{2}$ $\displaystyle\leq W.$ $$

Therefore, from Lemma 32(i) in [], we have

$$ $\log\mathbb{E}[e^{\gamma W}]\leq\frac{\gamma^{2}\mathbb{E}[W]}{2-\gamma}+ \gamma\mathbb{E}[W]=\frac{\gamma\mathbb{E}[W]}{1-\gamma/2}.$ $$

Thus,

$$ $Ent_{F}(\gamma)\leq\frac{\gamma}{2-\gamma}(\frac{\gamma\mathbb{E}[W]}{1-\gamma /2})=\frac{\gamma^{2}}{(1-\gamma/2)^{2}}\frac{\mathbb{E}[W]}{2}.$ $$

From Theorem 12 in [], we conclude that

$$ $\displaystyle\log\mathbb{E}e^{\beta(F-\mathbb{E}F)}$ $\displaystyle=\beta\int_{0}^{\beta}\frac{Ent_{F}(\gamma)}{\gamma^{2}}d\gamma$ $\displaystyle\leq\beta\frac{\mathbb{E}[W]}{2}\int_{0}^{\beta}\frac{1}{(2- \gamma/2)^{2}}d\gamma$ $\displaystyle=\frac{\beta^{2}}{1-\beta/2}\frac{\mathbb{E}[W]}{2}.$ $$

The above ineuqality implies that $F$ is a sub-gamma random variable. With the following lemma, we can derive the concentration inequality for $F$.

###### Lemma 4 .

Let $Z$ be a random variable, $A,B>0$ be some constants. If for any $\lambda\in(0,1/B)$ it holds

$$ $\log\mathbb{E}[e^{\lambda(Z-\mathbb{E}Z)}]\leq\frac{A\lambda^{2}}{2(1-B\lambda )},$ $$

then for all $x\geq 0$,

$$ $P(Z\geq\mathbb{E}Z+\sqrt{2Ax}+Bx)\leq e^{-x}.$ $$

Now, applying Lemma 4 with $A=\mathbb{E}[W],B=1/2$ for $F$, we can deduce that with probability at least $1-e^{-x}$

$$ $F\leq\mathbb{E}F+\sqrt{2x\mathbb{E}W}+\frac{x}{2}.$ $$

From the definitions of $F$ and $Z$, we have $Z=\frac{2bF}{nT}$, then with probability at least $1-e^{-x}$

$$ $Z\leq\mathbb{E}Z+\frac{2b}{nT}\sqrt{2x\mathbb{E}W}+\frac{bx}{nT}.$ $$

Note that $\mathbb{E}Z\leq 2\mathcal{R}(\mathcal{F})$ and

$$ $\displaystyle\mathbb{E}W$ $\displaystyle=\frac{1}{4b^{2}}\mathbb{E}\sup\limits_{\bm{f}\in\bm{\mathcal{F}} }\sum\limits_{t=1}^{T}\frac{n^{2}}{N_{t}^{2}}\sum\limits_{i=1}^{N_{t}}(f_{t}(X _{t}^{i})-\mathbb{E}f_{t}(X_{t}^{i}))^{2}+\mathbb{E}(f_{t}(X_{t}^{i})-\mathbb{ E}f_{t}(X_{t}^{i}))^{2}$ $\displaystyle=\frac{1}{4b^{2}}\mathbb{E}\sup\limits_{\bm{f}\in\bm{\mathcal{F}} }\sum\limits_{t=1}^{T}\frac{n^{2}}{N_{t}^{2}}\sum\limits_{i=1}^{N_{t}}[(f_{t}( X_{t}^{i})-\mathbb{E}f_{t}(X_{t}^{i}))^{2}-\mathbb{E}(f_{t}(X_{t}^{i})-\mathbb {E}f_{t}(X_{t}^{i}))^{2}]+2\mathbb{E}(f_{t}(X_{t}^{i})-\mathbb{E}f_{t}(X_{t}^{ i}))^{2}$ $\displaystyle\leq\frac{1}{4b^{2}}(2\mathbb{E}\sup\limits_{\bm{f}\in\bm{ \mathcal{F}}}\sum\limits_{t=1}^{T}\frac{n^{2}}{N_{t}^{2}}\sum\limits_{i=1}^{N_ {t}}\epsilon_{t,i}(f_{t}(X_{t}^{i})-\mathbb{E}f_{t}(X_{t}^{i}))^{2}+\sup \limits_{\bm{f}\in\bm{\mathcal{F}}}2\sum\limits_{t=1}^{T}\frac{n^{2}}{N_{t}} \mathbb{E}(f_{t}(X_{t}^{1})-\mathbb{E}f_{t}(X_{t}^{1}))^{2})$ $\displaystyle\leq\frac{1}{4b^{2}}(2nT4b2\mathcal{R}(\mathcal{F})+2nTr)$ $\displaystyle\leq\frac{4nT\mathcal{R}(\mathcal{F})}{b}+\frac{nTr}{2b^{2}},$ $$

where the first inequality follows from the standard symmetrization technique and the second ineuqality follows from the contraction property of the Rademacher complexity.

Plugging it into the concentration inequality for $Z$, we have

$$ $\displaystyle Z$ $\displaystyle\leq\mathbb{E}Z+\frac{2b}{nT}\sqrt{2x\mathbb{E}W}+\frac{bx}{nT}$ $\displaystyle\leq 2\mathcal{R}(\mathcal{F})+\frac{2b}{nT}\sqrt{2x(\frac{4nT \mathcal{R}(\mathcal{F})}{b}+\frac{nTr}{2b^{2}})}+\frac{bx}{nT}$ $\displaystyle=2\mathcal{R}(\mathcal{F})+2\sqrt{\frac{8bx\mathcal{R}(\mathcal{F })}{nT}+\frac{xr}{nT}}+\frac{bx}{nT}$ $\displaystyle\leq 2\mathcal{R}(\mathcal{F})+2\sqrt{\frac{8bx\mathcal{R}( \mathcal{F})}{nT}}+2\sqrt{\frac{xr}{nT}}+\frac{bx}{nT}$ $\displaystyle\leq 2(1+\alpha)\mathcal{R}(\mathcal{F})+2\sqrt{\frac{xr}{nT}}+(1 +\frac{4}{\alpha})\frac{bx}{nT},$ $$

where the last inequality follows from the inequality $2\sqrt{ab}\leq\alpha a+\frac{b}{\alpha}$ for any $\alpha>0,a>0,b>0$.
∎

###### Proof.

###### Lemma 4 .

### 6.3. Proof of Lemma 1

###### Proof.

Let us consider two cases:

1: $P\bm{f}^{2}\leq r$,

2: $P\bm{f}^{2}>r$.

For the first case, $\bm{f}=\frac{r}{P\bm{f}^{2}\lor r}\bm{f}\in\bm{\mathcal{F}}_{r}$, therefore

$$ $P\bm{f}\leq P_{N}\bm{f}+U_{N}(\bm{\mathcal{F}}_{r})\leq P_{N}\bm{f}+\frac{r}{K }\leq\frac{K}{K-1}P_{N}\bm{f}+\frac{r}{bK}.$ $$

For the second case, $\frac{r}{P\bm{f}^{2}\lor r}\bm{f}=\frac{r}{P\bm{f}^{2}}\bm{f}\in\bm{\mathcal{F
}}_{r}$, thus

$$ $P\frac{r}{Pf^{2}}\bm{f}\leq P_{N}\frac{r}{P\bm{f}^{2}}\bm{f}+U_{N}(\bm{ \mathcal{F}}_{r})\leq P_{N}\frac{r}{P\bm{f}^{2}}\bm{f}+\frac{r}{bK}.$ $$

Basic algebraic transformation yields that

$$ $P\bm{f}\leq P_{N}\bm{f}+\frac{P\bm{f}^{2}}{bK}\leq P_{N}\bm{f}+\frac{P\bm{f}}{ K},$ $$

which implies

$$ $P\bm{f}\leq\frac{K}{K-1}P_{N}\bm{f}\leq\frac{K}{K-1}P_{N}\bm{f}+\frac{r}{bK}.$ $$

∎

###### Proof.

### 6.4. Proof of Lemma 2

###### Proof.

The aim is to find some $r$ such that $U_{N}(\bm{\mathcal{F}}_{r})\leq\frac{r}{bK}$, then applying Lemma 2 yields the conclusion.

Note that the variance of functions in $\bm{\mathcal{F}}_{r}$ is at most $r$. For any $\bm{f}\in\bm{\mathcal{F}}_{r}$, we consider two cases:

1: $P\bm{f}^{2}\leq r$,

2: $P\bm{f}^{2}>r$.

For the first case, $\bm{f}=\frac{r}{P\bm{f}^{2}\lor r}\bm{f}\in\bm{\mathcal{F}}_{r}$, thus $Var(\frac{r}{P\bm{f}^{2}\lor r}\bm{f})=Var(\bm{f})\leq P\bm{f}^{2}\leq r$.

For the second case,

$$ $Var(\frac{r}{P\bm{f}^{2}\lor r}\bm{f})=Var(\frac{r}{P\bm{f}^{2}}\bm{f})\leq P( \frac{r}{P\bm{f}^{2}}\bm{f})^{2}=\frac{r^{2}}{P\bm{f}^{2}}<r.$ $$

Then applying Theorem 3 for $U_{N}(\mathcal{F}_{r})$, we have that with probability at least $1-e^{-x}$,

$$ $\displaystyle U_{N}(\mathcal{F}_{r})$ $\displaystyle\leq 4\mathcal{R}(\mathcal{F}_{r})+2\sqrt{\frac{xr}{nT}}+\frac{5 bx}{nT}$ $\displaystyle\leq 4\frac{\psi(r)}{b}+2\sqrt{\frac{xr}{nT}}+\frac{5bx}{nT}$ $\displaystyle\leq 4\frac{\sqrt{rr^{*}}}{b}+2\sqrt{\frac{xr}{nT}}+\frac{5bx}{nT}$ $\displaystyle:=A\sqrt{r}+B,$ $$

where the third inequality follows from the property of the sub-root function, i.e., $\psi(r)/\sqrt{r}\leq\psi(r^{*})/\sqrt{r^{*}}=\sqrt{r^{*}}$ for any $r>r^{*}$ and $A=\frac{4\sqrt{r^{*}}}{b}+2\sqrt{\frac{x}{nT}}$, $B=\frac{5bx}{nT}$.

Solving the equation

$$ $A\sqrt{r}+B=\frac{r}{bK}$ $$

yields that

$$ $\sqrt{r}=\frac{bKA+\sqrt{b^{2}K^{2}A^{2}+4bKB}}{2}.$ $$

Thus

$$ $r\geq\frac{b^{2}K^{2}A^{2}}{2}>r^{*}$ $$

and

$$ $r\leq b^{2}K^{2}A^{2}+2bKB.$ $$

Therefore by Lemma 2, we have

$$ $\displaystyle P\bm{f}$ $\displaystyle\leq\frac{K}{K-1}P_{N}\bm{f}+\frac{r}{bK}$ $\displaystyle\leq\frac{K}{K-1}P_{N}\bm{f}+bKA^{2}+2B$ $\displaystyle\leq\frac{K}{K-1}P_{N}\bm{f}+2bK(\frac{16r^{*}}{b^{2}}+\frac{4x}{ nT})+\frac{10bx}{nT}$ $\displaystyle=\frac{K}{K-1}P_{N}\bm{f}+\frac{32Kr^{*}}{b}+\frac{(10b+8bK)x}{nT}.$ $$

∎

###### Proof.

### 6.5. Proof of Theorem 4

There remain some problems regarding the selection of the sub-root function $\psi$ and the computation of its fixed point. Just as in the single-task scenario, we can take $\psi$ as the local Rademacher averages of the star-hull of $\mathcal{F}$ around 0 0.

Specifically, let

$$ $\psi(r):=16b\mathbb{E}\mathcal{R}_{N}\{\bm{f}:\bm{f}\in star(\bm{\mathcal{F}}, 0),P\bm{f}^{2}\leq r\}+\frac{14b^{2}\log(nT)}{nT},$ $$

where $star(\bm{\mathcal{F}},0):=\{\alpha\bm{f}:\ \bm{f}\in\bm{\mathcal{F}},\alpha\in
[0,1]\}$.

Recall that the normalized function class $\bm{\mathcal{F}}_{r}$ is a subset of the function class $\{\bm{f}:\bm{f}\in star(\bm{\mathcal{F}},0),P\bm{f}^{2}\leq r\}$, thus $\psi(r)\geq b\mathcal{R}(\bm{\mathcal{F}}_{r})$.

For the first term in the definition of $\psi(r)$, with the following lemma, we can translate the ball in $L^{2}(P)$ into the ball in $L^{2}(P_{N})$, so that Dudley’s inequality can be applied.

###### Lemma 5 .

Let $\bm{\mathcal{G}}$ be a class of vector-valued functions that map $\mathcal{X}$ into $[-b,b]^{T}$ with $b>0$. For every $x>0$ and $r$ satisfy

$$ $r\geq 16b\mathbb{E}\mathcal{R}_{N}\{\bm{g}:\bm{g}\in\bm{\mathcal{G}},P\bm{g}^{ 2}\leq r\}+\frac{14b^{2}x}{nT},$ $$

then with probability at least $1-e^{-x}$

$$ $\{\bm{g}\in\bm{\mathcal{G}}:P\bm{g}^{2}\leq r\}\subset\{\bm{g}\in\bm{\mathcal{ G}}:P_{N}\bm{g}^{2}\leq 2r\}.$ $$

###### Proof.

Define $\bm{\mathcal{G}}_{r}:=\{\bm{g}^{2}:\bm{g}\in\bm{\mathcal{G}}:P\bm{g}^{2}\leq r\}$.

Note that $\|\bm{g}^{2}\|_{\infty}\leq b^{2},Var(\bm{g}^{2})\leq P\bm{g}^{4}\leq b^{2}P
\bm{g}^{2}\leq b^{2}r$. Then applying the Theorem 3 for $\bm{\mathcal{G}}_{r}$ with $\alpha=1$ yields that with probability at least $1-e^{-x}$, for every $\bm{g}_{0}=\bm{g}^{2}\in\bm{\mathcal{G}}_{r}$,

$$ $\displaystyle P_{N}\bm{g}^{2}$ $\displaystyle\leq P\bm{g}^{2}+4\mathbb{E}\mathcal{R}_{N}\{\bm{g}^{2}:\bm{g}\in \bm{\mathcal{G}},P\bm{g}^{2}\leq r\}+2\sqrt{\frac{b^{2}xr}{nT}}+\frac{5b^{2}x} {nT}$ $\displaystyle\leq r+8b\mathbb{E}\mathcal{R}_{N}\{\bm{g}:\bm{g}\in\bm{\mathcal{ G}},P\bm{g}^{2}\leq r\}+\frac{r}{2}+\frac{7b^{2}x}{nT}$ $\displaystyle\leq 2r,$ $$

where the second inequality follows from the contraction property of the Rademacher complexity and the mean inequality. ∎

###### Remark 2 .

Although the contraction property used in the proof of Lemma 5 is slightly different from the standard form (see Lemma 5.7 in []), it is just an adaptation of the standard one.

Specifically, let $\Phi_{i}$ be $l_{i}$-Lipschitz functions from $\mathbb{R}$ to $\mathbb{R}$ for $i=1,\cdots,m$ and $\sigma_{1},\cdots,\sigma_{m}$ be Rademacher random variables. Then for any set $A\subset\mathbb{R}^{m}$, the following inequality holds:

$$ $\mathbb{E}_{\sigma}\sup\limits_{a\in A}\sum\limits_{i=1}^{m}\sigma_{i}\Phi_{i} (a_{i})\leq\mathbb{E}_{\sigma}\sup\limits_{a\in A}\sum\limits_{i=1}^{m}\sigma_ {i}l_{i}a_{i}.$ $$

For completeness, we give a brief proof.

By the Fubini’s theorem, we have

$$ $\mathbb{E}_{\sigma}\sup\limits_{a\in A}\sum\limits_{i=1}^{m}\sigma_{i}\Phi_{i} (a_{i})=\mathbb{E}_{\sigma_{1},\cdots,\sigma_{m-1}}\mathbb{E}_{\sigma_{m}}[ \sup\limits_{a\in A}u_{m-1}(a)+\sigma_{m}\Phi_{m}(a_{m})],$ $$

where $u_{m-1}(a)=\sum\limits_{i=1}^{m-1}\sigma_{i}\Phi_{i}(a_{i})$.

From the proof of Lemma 5.7 in [], we know

$$ $\mathbb{E}_{\sigma_{m}}[\sup\limits_{a\in A}u_{m-1}(a)+\sigma_{m}\Phi_{m}(a_{m })]\leq\mathbb{E}_{\sigma_{m}}[\sup\limits_{a\in A}u_{m-1}(a)+\sigma_{m}l_{m}a _{m}].$ $$

Proceeding in the same way for all other $\sigma_{i}$($i\neq m$) leads to the conclusion. In fact, we have used the conclusion with $\Phi_{t,i}(x)=\frac{x^{2}}{N_{t}}$ in the proof of Lemma 5.

With Lemma 5, we can bound $r^{*}$ as follows.

###### Lemma 6 .

$$ $r^{*}\leq 16b\mathbb{E}\mathcal{R}_{N}\{\bm{f}:\bm{f}\in star(\bm{\mathcal{F}} ,0),P_{N}\bm{f}^{2}\leq 2r^{*}\}+\frac{16b^{2}+14b^{2}\log(nT)}{nT}.$ $$

###### Proof.

From Lemma 5 and the fact that

$$ $r^{*}=\psi(r^{*})=16b\mathbb{E}\mathcal{R}_{N}\{\bm{f}:\bm{f}\in star(\bm{ \mathcal{F}},0),P\bm{f}^{2}\leq r^{*}\}+\frac{14b^{2}\log(nT)}{nT},$ $$

we can deduce that with probability at least $1-\frac{1}{nT}$,

$$ $\{\bm{f}:\bm{f}\in star(\bm{\mathcal{F}},0),P\bm{f}^{2}\leq r^{*}\}\subset\{ \bm{f}:\bm{f}\in star(\bm{\mathcal{F}},0),P_{N}\bm{f}^{2}\leq 2r^{*}\}.$ $$

Therefore,

$$ $\displaystyle r^{*}$ $\displaystyle\leq 16b\left[\mathbb{E}\mathcal{R}_{N}\{\bm{f}:\bm{f}\in star( \bm{\mathcal{F}},0),P_{N}\bm{f}^{2}\leq 2r^{*}\}+\frac{b}{nT}\right]+\frac{14b ^{2}\log(nT)}{nT}$ $\displaystyle=16b\mathbb{E}\mathcal{R}_{N}\{\bm{f}:\bm{f}\in star(\bm{\mathcal {F}},0),P_{N}\bm{f}^{2}\leq 2r^{*}\}+\frac{16b^{2}+14b^{2}\log(nT)}{nT}.$ $$

∎

Now, we are ready to use the Dudley’s inequality to bound the first term in the right.

Define $\bm{\mathcal{F}}_{s,r}:=\{\bm{f}:\ \bm{f}\in star(\bm{\mathcal{F}},0),P_{N}\bm
{f}^{2}\leq 2r\}$.

Specifically, with the samples $(X_{t}^{i})_{(t,i)=(1,1)}^{(T,N_{t})}$ fixed, define a random process $(X_{\bm{f}})_{\bm{f}\in\bm{\mathcal{F}}_{s,r}}$ with

$$ $X_{\bm{f}}:=\frac{1}{T}\sum\limits_{t=1}^{T}\frac{1}{N_{t}}\sum\limits_{i=1}^{ N_{t}}\sigma_{t}^{i}f_{t}(X_{t}^{i})\ for\ \bm{f}=(f_{1},\cdots,f_{T})\in\bm{ \mathcal{F}}_{s,r}.$ $$

From the fact that $\sigma_{t}^{i}$ is sub-gaussian, we can deduce that for any $\lambda\in\mathbb{R}$ and $\bm{f}^{{}^{\prime}}=(f_{1}^{{}^{\prime}},\cdots,f_{T}^{{}^{\prime}})\in\bm{
\mathcal{F}}_{s,r}$

$$ $\displaystyle\mathbb{E}e^{\lambda(X_{\bm{f}}-X_{\bm{f}^{{}^{\prime}}})}$ $\displaystyle\leq\mathbb{E}e^{\frac{\lambda}{T}\sum\limits_{t=1}^{T}\frac{1}{N _{t}}\sum\limits_{i=1}^{N_{t}}\sigma_{t}^{i}(f_{t}(X_{t}^{i})-f_{t}^{{}^{ \prime}}(X_{t}^{i}))}$ $\displaystyle\leq e^{\frac{\lambda^{2}}{2T^{2}}\sum\limits_{t=1}^{T}\frac{1}{N _{t}^{2}}\sum\limits_{i=1}^{N_{t}}(f_{t}(X_{t}^{i})-f_{t}^{{}^{\prime}}(X_{t}^ {i}))^{2}}$ $\displaystyle\leq e^{\frac{\lambda^{2}}{2}K^{2}d^{2}(\bm{f},\bm{f}^{{}^{\prime }})},$ $$

where $K=\frac{1}{\sqrt{nT}}$ and

$$ $d(\bm{f},\bm{f}^{{}^{\prime}}):=\sqrt{\frac{1}{T}\sum\limits_{t=1}^{T}\frac{1} {N_{t}}\sum\limits_{i=1}^{N_{t}}(f_{t}(X_{t}^{i})-f_{t}^{{}^{\prime}}(X_{t}^{i }))^{2}}.$ $$

It implies that $\|X_{\bm{f}}-X_{\bm{f}^{{}^{\prime}}}\|_{\psi_{2}}\leq CKd(\bm{f},\bm{f}^{{}^{
\prime}})$ with a universal constant $C$.

Then using Dudley’s inequality yields that

$$ $\mathbb{E}\sup\limits_{\bm{f}\in\bm{\mathcal{F}}_{s,r}}X_{\bm{f}}\leq CK\int_{ 0}^{diam(\bm{\mathcal{F}}_{s,r})}\sqrt{\log\mathcal{N}(\bm{\mathcal{F}}_{s,r}, d,\epsilon)}d\epsilon\leq CK\int_{0}^{2\sqrt{r}}\sqrt{\log\mathcal{N}(\bm{ \mathcal{F}}_{s,r},d,\epsilon)}d\epsilon,$ $$

where $diam(\bm{\mathcal{F}}_{s,r}):=\sup_{\bm{f},\bm{f}^{{}^{\prime}}\in\bm{\mathcal
{F}}_{s,r}}d(\bm{f},\bm{f}^{{}^{\prime}})$.

###### Proof of Theorem 4:.

In the following, we assume that $\mathcal{F}$ is a parameterized hypothesis function class to be determined. When considering the framework of PINNs for the linear second order elliptic equation as MTL, the function class in MTL associated with $\mathcal{F}$ is defined as

$$ $\bm{\mathcal{F}}:=\{\bm{u}=(|\Omega|(Lu(x)-f(x))^{2},|\partial\Omega|(u(y)-g(y ))^{2}):\ u\in\mathcal{F}\}.$ $$

Note that here we use notation $u$ to represent some function in $\mathcal{F}$ and $\bm{u}$ to denote the corresponding vector-valued function associated with $u$.

Then the empirical loss can be written as

$$ $\mathcal{L}_{N}(u)=2P_{N}\bm{u},$ $$

where $N=(N_{1},N_{2})$ and $n=\min(N_{1},N_{2})$.

The aim is to seek $u_{N}\in\mathcal{F}$ which minimizes $\mathcal{L}_{N}$. It is equivalent to seek $\bm{u}_{N}\in\bm{\mathcal{F}}$ which minimizes $P_{N}\bm{u}$ i.e.

$$ $\bm{u}_{N}\in\mathop{{\arg\min}}\limits_{\bm{u}\in\bm{\mathcal{F}}}P_{N}\bm{u}.$ $$

Assume that $u^{*}$ is the solution of the linear second order elliptic PDE and there is a constant $M$ such that $|a_{ij}|,|b_{i}|,|c|,|g|,|u^{*}|,|\partial_{i}u^{*}|,|\partial_{ij}u^{*}|\leq M$ and $|u|,|\partial_{i}u|,|\partial_{ij}u|\leq M$ for any $u\in\mathcal{F}$, $1\leq i,j\leq d$.

Then $\sup\limits_{u\in\mathcal{F}}\max(|\Omega|(Lu-f)^{2},|\partial\Omega|(u-g)^{2}
)\leq b:=c(|\Omega|d^{2}M^{4}+|\partial\Omega|M^{2})$ with a universal constant $c$.

Therefore, with probability at least $1-e^{-t}$

$$ $\displaystyle P_{N}\bm{u}_{N}\leq P_{N}\bm{u}_{\mathcal{F}}$ $\displaystyle\leq P\bm{u}_{\mathcal{F}}+2\sqrt{\frac{tVar(\bm{u}_{\mathcal{F}} ^{2})}{2n}}+\frac{2bt}{2n}$ $\displaystyle\leq P\bm{u}_{\mathcal{F}}+2\sqrt{\frac{tbP\bm{u}_{\mathcal{F}}}{ 2n}}+\frac{bt}{n}$ $\displaystyle\leq\frac{3}{2}P\bm{u}_{\mathcal{F}}+\frac{2bt}{n},$ $$

where $\bm{u}_{\mathcal{F}}=(|\Omega|(Lu_{{}_{\mathcal{F}}}-f)^{2},|\partial\Omega|(u
_{\mathcal{F}}-g)^{2})$, $u_{\mathcal{F}}\in\mathop{\arg\min}\limits_{u\in\mathcal{F}}\|u-u^{*}\|_{H^{2}
(\Omega)}^{2}$ and the second inequality follows from Theorem 3 by taking $\mathcal{F}=\{\bm{u}_{\mathcal{F}}\}$ and $\alpha=4,T=2$, which can be seen as a vector version of the Bernstein inequality. Here, we define the approximation error as $\epsilon_{app}:=\|u_{\mathcal{F}}-u^{*}\|_{H^{2}(\Omega)}$.

Then applying Lemma 2 with $K=2$ yields that with probability at least $1-2e^{-t}$

$$ $\displaystyle P\bm{u}_{N}$ $\displaystyle\leq 2P_{N}\bm{u}_{N}+\frac{64r^{*}}{b}+\frac{13bt}{n}$ $\displaystyle\leq 3P\bm{u}_{\mathcal{F}}+\frac{64r^{*}}{b}+\frac{17bt}{n},$ $$

which implies that

$$ $\mathcal{L}(u_{N})=2P_{N}\bm{u}_{N}\leq 3\mathcal{L}(u_{\mathcal{F}})+\frac{12 8r^{*}}{b}+\frac{34bt}{n}.$ $$

Note that $\mathcal{L}(u_{\mathcal{F}})$ can be bounded by the approximation error, as for any $u\in H^{2}(\Omega)$

$$ $\displaystyle\mathcal{L}(u)$ $\displaystyle=\int_{\Omega}(Lu-f)^{2}dx+\int_{\partial\Omega}(u-g)^{2}dy$ $\displaystyle=\int_{\Omega}(-\sum\limits_{i,j=1}^{d}a_{ij}\partial_{ij}u+\sum \limits_{i=1}^{d}b_{i}\partial_{i}u+cu-f)^{2}dx+\int_{\partial\Omega}(u-g)^{2}dy$ $\displaystyle=\int_{\Omega}(-\sum\limits_{i,j=1}^{d}a_{ij}\partial_{ij}(u-u^{* })+\sum\limits_{i=1}^{d}b_{i}\partial_{i}(u-u^{*})+c(u-u^{*}))^{2}dx+\int_{ \partial\Omega}(u-u^{*})^{2}dy$ $\displaystyle\leq 3\int_{\Omega}(-\sum\limits_{i,j=1}^{d}a_{ij}\partial_{ij}(u -u^{*}))^{2}+(\sum\limits_{i=1}^{d}b_{i}\partial_{i}(u-u^{*}))^{2}+(c(u-u^{*}) )^{2}dx+\int_{\partial\Omega}(u-u^{*})^{2}dy$ $\displaystyle\leq 3d^{2}M^{2}\|u-u^{*}\|_{H^{2}(\Omega)}^{2}+C(Tr,\Omega)^{2} \|u-u^{*}\|_{H^{1}(\Omega)}^{2}$ $\displaystyle\leq(3d^{2}M^{2}+C(Tr,\Omega)^{2})\|u-u^{*}\|_{H^{2}(\Omega)}^{2},$ $$

where in the last inequality, we use the boundedness of $a_{ij},b_{i},c$ and the Sobolev trace theorem with the constant $C(Tr,\Omega)$ that depends only on the domain $\Omega$.

Thus,

$$ $\mathcal{L}(u_{\mathcal{F}})\leq(3d^{2}M^{2}+C(Tr,\Omega)^{2})\epsilon_{app}^{2}$ $$

and with probability at least $1-2e^{-t}$

$$ $\mathcal{L}(u_{N})=2P_{N}\bm{u}_{N}\leq 3(3d^{2}M^{2}+C(Tr,\Omega)^{2}) \epsilon_{app}^{2}+\frac{128r^{*}}{b}+\frac{34bt}{n}.$ $$

It remains only to bound the fixed point $r^{*}$. With Lemma 6, we only to bound the covering number of $\bm{\mathcal{F}}$ under $d$, which is done in the Lemma 15.

(1) For the two-layer neural neworks, we know

$$ $\log\mathcal{N}(\bm{\mathcal{F}},d,\epsilon)\leq cmd\log(\frac{b}{\epsilon}),$ $$

where $c$ is a universal constant.

Therefore

$$ $\displaystyle r^{*}$ $\displaystyle\leq cb\sqrt{\frac{md}{n}}\int_{0}^{2\sqrt{r^{*}}}\sqrt{\log( \frac{b}{\epsilon})}d\epsilon+\frac{cb^{2}\log n}{n}$ $\displaystyle=cb^{2}\sqrt{\frac{md}{n}}\int_{0}^{2\sqrt{\frac{r^{*}}{b^{2}}}} \sqrt{\log(\frac{1}{\epsilon})}d\epsilon+\frac{cb^{2}\log n}{n}$ $\displaystyle\leq cb\sqrt{\frac{mdr^{*}}{n}}\sqrt{\log(\frac{2b}{\sqrt{r^{*}}} )}+c\frac{cb^{2}\log n}{n}$ $\displaystyle\leq cb\sqrt{\frac{mdr^{*}}{n}}\sqrt{\log n}+\frac{cb^{2}\log n}{ n},$ $$

where second inequality follows from Lemma 12.

It implies that

$$ $r^{*}\leq\frac{cb^{2}md\log n}{n}.$ $$

(2) For the deep neural networks, we know

$$ $\log\mathcal{N}(\bm{\mathcal{F}},d,\epsilon)\leq CK^{d}\log(\frac{K}{\epsilon}),$ $$

where $C$ is a constant independent of $K$.

Similar to that in (1), we have

$$ $r^{*}\leq\frac{CK^{d}\log K\log n}{n}$ $$

with a constant $C$ independent of $K,N,n$.

∎

###### Lemma 5 .

###### Proof.

###### Remark 2 .

###### Lemma 6 .

###### Proof.

###### Proof of Theorem 4:.

### 6.6. Proof of Proposition 4

Recall that $R_{int}=u_{t}-\Delta u-f(u)$, $R_{tb}=\bar{u}-u$, $R_{sb}=u$.

We can verify that the error $\hat{u}=u^{*}-u$ satisfies the following parabolic equation:

$$ $\displaystyle\hat{u}_{t}$ $\displaystyle=\Delta\hat{u}+f(u^{*})-f(u)+R_{int}(x,t)\quad\forall x\in D,t\in (0,T),$ $\displaystyle\hat{u}(x,0)$ $\displaystyle=R_{tb}(x)\quad\forall x\in D,$ $\displaystyle\hat{u}(x,t)$ $\displaystyle=R_{sb}(x,t)\quad\forall x\in\partial D,t\in(0,T).$ $$

(1) Multiplying both sides by $\hat{u}$, integrating over the domain and integrating by parts yields

$$ $\displaystyle\frac{1}{2}\frac{d}{dt}\int_{D}|\hat{u}(x,t)|^{2}dx$ $\displaystyle=-\int_{D}|\nabla\hat{u}|^{2}dx+\int_{\partial D}R_{sb}(x,t)\frac {\partial\hat{u}}{\partial n}ds(x)+\int_{D}\hat{u}(f(u^{*})-f(u))dx+\int_{D}R_ {int}\hat{u}dx$ $\displaystyle\leq C\int_{\partial D}|R_{sb}(x,t)|ds(x)+\int_{D}|\hat{u}|^{2}dx +\frac{1}{2}(\int_{D}|R_{int}|^{2}dx+\int_{D}|\hat{u}|^{2}dx)$ $\displaystyle\leq C(\int_{D}|\hat{u}|^{2}dx+\int_{\partial D}|R_{sb}(x,t)|ds(x )+\int_{D}|R_{int}|^{2}dx),$ $$

where the first inequality follows from the fact that $\|u\|_{C^{1}([0,T]\times\partial D)},\|u^{*}\|_{C^{1}([0,T]\times\partial D)}<\infty$.

(2) Multiplying both sides by $\hat{u}_{t}$, integrating over the domain yields

$$ $\displaystyle\int_{D}|\hat{u}_{t}(x,t)|^{2}dx$ $\displaystyle=\int_{D}(\Delta\hat{u})\hat{u}_{t}dx+\int_{D}\hat{u}_{t}(f(u)-f( u^{*}))dx+\int_{D}R_{int}\hat{u}_{t}dx$ $\displaystyle=-\int_{D}\nabla\hat{u}\cdot\nabla\hat{u}_{t}dx+\int_{\partial D} \hat{u}_{t}\frac{\partial\hat{u}}{\partial n}ds(x)+\int_{D}\hat{u}_{t}(f(u)-f( u^{*}))dx+\int_{D}R_{int}\hat{u}_{t}dx.$ $$

Rearranging it and integrating by parts yields

$$ $\displaystyle\frac{1}{2}\frac{d}{dt}\int_{D}|\nabla\hat{u}(x,t)|^{2}dx$ $\displaystyle=-\int_{D}|\hat{u}_{t}|^{2}dx+\int_{\partial D}\hat{u}_{t}\frac{ \partial\hat{u}}{\partial n}ds+\int_{D}\hat{u}_{t}(f(u)-f(u^{*}))dx+\int_{D}R_ {int}\hat{u}_{t}dx$ $\displaystyle\leq C\int_{\partial D}|\hat{u}_{t}|ds+C\int_{D}|\hat{u}_{t}\hat{ u}|dx+\frac{1}{2}(\int_{D}|R_{int}|^{2}dx+\int_{D}|\hat{u}|^{2}dx)$ $\displaystyle\leq C(\int_{\partial D}|\partial_{t}R_{sb}|ds+\int_{D}|R_{int}|^ {2}dx+\int_{D}|\hat{u}_{t}|^{2}dx+\int_{D}|\hat{u}|^{2}dx),$ $$

where the second ineuqality follows from the fact that $\hat{u}_{t}(x,t)=\partial_{t}R_{sb}(x,t)$ for $x\in\partial D$ and the mean inequality.

(3) Differentiating both sides with respect to $t$ yields

$$ $\hat{u}_{tt}=\Delta\hat{u}_{t}+f^{{}^{\prime}}(u^{*})u^{*}_{t}-f^{{}^{\prime}} (u)u_{t}+\partial_{t}R_{int},\ for\ (x,t)\in D\times(0,T).$ $$

Then multiplying both sides by $\hat{u}_{t}$, integrating over the domain and integrating by parts yields

$$ $\displaystyle\frac{1}{2}\frac{d}{dt}\int_{D}|\hat{u}_{t}|^{2}dx$ $\displaystyle=\int_{D}(\Delta\hat{u}_{t})\hat{u}_{t}dx+\int_{D}\hat{u}_{t}(f^{ {}^{\prime}}(u^{*})u^{*}_{t}-f^{{}^{\prime}}(u)u_{t})dx+\int_{D}\hat{u}_{t} \partial_{t}R_{int}dx$ $\displaystyle=-\int_{D}|\nabla\hat{u}_{t}|^{2}dx+\int_{\partial D}\hat{u}_{t} \frac{\partial\hat{u}_{t}}{\partial n}ds+\int_{D}\hat{u}_{t}(f^{{}^{\prime}}(u ^{*})u^{*}_{t}-f^{{}^{\prime}}(u)u_{t})dx+\int_{D}\hat{u}_{t}\partial_{t}R_{ int}dx$ $\displaystyle\leq C\int_{\partial D}|\partial_{t}R_{sb}|ds+\int_{D}\hat{u}_{t} (f^{{}^{\prime}}(u^{*})u^{*}_{t}-f^{{}^{\prime}}(u)u_{t})dx+\frac{1}{2}(\int_{ D}|\hat{u}_{t}|^{2}dx+\int_{D}|\partial_{t}R_{int}|^{2}dx)$ $\displaystyle\leq C(\int_{\partial D}|\partial_{t}R_{sb}|ds+\int_{D}|\partial_ {t}R_{int}|^{2}dx+\int_{D}|\hat{u}_{t}|^{2}dx+\int_{D}|\hat{u}|^{2}dx),$ $$

where the second inequality follows from the fact that

$$ $\displaystyle|\int_{D}\hat{u}_{t}(f^{{}^{\prime}}(u^{*})u^{*}_{t}-f^{{}^{ \prime}}(u)u_{t})dx|$ $\displaystyle\leq\int_{D}|\hat{u}_{t}(f^{{}^{\prime}}(u^{*})u^{*}_{t}-f^{{}^{ \prime}}(u)u_{t})|dx$ $\displaystyle\leq\int_{D}|\hat{u}_{t}|(|f^{{}^{\prime}}(u^{*})-f^{{}^{\prime}} (u)||u_{t}^{*}|+|f^{{}^{\prime}}(u)||u_{t}-u_{t}^{*}|)dx$ $\displaystyle\leq C\int_{D}|\hat{u}_{t}||\hat{u}|dx+C\int_{D}|\hat{u}_{t}|^{2}dx$ $\displaystyle\leq C\int_{D}|\hat{u}_{t}|^{2}dx+C\int_{D}|\hat{u}|^{2}dx.$ $$

Combining all results above yields

$$ $\displaystyle\frac{1}{2}\frac{d}{dt}(\int_{D}|\hat{u}(x,t)|^{2}dx+\int_{D}| \nabla\hat{u}(x,t)|^{2}dx+\int_{D}|\hat{u}_{t}(x,t)|^{2}dx)$ $\displaystyle\leq C(\int_{D}|\hat{u}(x,t)|^{2}dx+\int_{D}|\nabla\hat{u}|^{2}dx +\int_{D}|\hat{u}_{t}(x,t)|^{2}dx+\int_{D}|R_{int}(x,t)|^{2}dx+\int_{\partial D }|R_{sb}(x,t)|ds$ $\displaystyle\quad+\int_{D}|\partial_{t}R_{int}(x,t)|^{2}dx+\int_{\partial D}| \partial_{t}R_{sb}(x,t)|ds).$ $$

Integrating the above inequality over $[0,\bar{T}]$ for any $\bar{T}\leq T$, we obtain

$$ $\displaystyle\int_{D}|\hat{u}(x,\bar{T})|^{2}dx+\int_{D}|\nabla\hat{u}(x,\bar{ T})|^{2}dx+\int_{D}|\hat{u}_{t}(x,\bar{T})|^{2}dx$ $\displaystyle\leq\int_{D}|\hat{u}(x,0)|^{2}dx+\int_{D}|\nabla\hat{u}(x,0)|^{2} dx+\int_{D}|\hat{u}_{t}(x,0)|^{2}dx+$ $\displaystyle\quad+C(\int_{0}^{\bar{T}}\int_{D}|\hat{u}(x,t)|^{2}dx+\int_{0}^{ \bar{T}}\int_{D}|\nabla\hat{u}|^{2}dx+\int_{0}^{\bar{T}}\int_{D}|\hat{u}_{t}(x ,t)|^{2}dx)$ $\displaystyle\quad+C(\int_{0}^{\bar{T}}\int_{D}|R_{int}(x,t)|^{2}dx+\int_{0}^{ \bar{T}}\int_{\partial D}|R_{sb}(x,t)|ds+\int_{0}^{\bar{T}}\int_{D}|\partial_{ t}R_{int}(x,t)|^{2}dx+\int_{0}^{\bar{T}}\int_{\partial D}|\partial_{t}R_{sb}(x ,t)|ds).$ $$

Note that

$$ $\int_{D}|\hat{u}(x,0)|^{2}dx=\int_{D}|R_{tb}(x)|^{2}dx,$ $$

$$ $\int_{D}|\nabla\hat{u}(x,0)|^{2}dx=\int_{D}|\nabla R_{tb}(x)|^{2}dx$ $$

and

$$ $\displaystyle\int_{D}|\hat{u}_{t}(x,0)|^{2}dx$ $\displaystyle=\int_{D}|\Delta\hat{u}(x,0)+f(u^{*}(x,0))-f(u(x,0))+R_{int}(x,0) |^{2}dx$ $\displaystyle\leq 3\int_{D}|\Delta R_{tb}(x)|^{2}dx+3C\int_{D}|\hat{u}(x,0)|^{ 2}dx+3\int_{D}|R_{int}(x,0)|^{2}dx.$ $$

Define $E(\bar{T})=\int_{D}|\hat{u}(x,\bar{T})|^{2}dx+\int_{D}|\nabla\hat{u}(x,\bar{T}
)|^{2}dx+\int_{D}|\hat{u}_{t}(x,\bar{T})|^{2}dx$, the above results yields

$$ $E(\bar{T})\leq C\int_{0}^{\bar{T}}E(t)dt+CR(\bar{T}),$ $$

where

$$ $\displaystyle R(\bar{T})$ $\displaystyle:=(\int_{D}|\Delta R_{tb}(x)|^{2}dx+\int_{D}|R_{int}(x,0)|^{2}dx+ \int_{D}|R_{tb}(x)|^{2}dx+\int_{D}|\nabla R_{tb}(x)|^{2}dx)$ $\displaystyle+(\int_{0}^{\bar{T}}\int_{D}|R_{int}(x,t)|^{2}dx+\int_{0}^{\bar{T }}\int_{\partial D}|R_{sb}(x,t)|ds+\int_{0}^{\bar{T}}\int_{D}|\partial_{t}R_{ int}(x,t)|^{2}dx+\int_{0}^{\bar{T}}\int_{\partial D}|\partial_{t}R_{sb}(x,t)| ds).$ $$

From the integral form of the Grönwall’s inequality, we have

$$ $E(\bar{T})\leq CR(\bar{T})e^{C\bar{T}}.$ $$

Intrgrating over $\bar{T}$, we can deduce that

$$ $\int_{0}^{T}E(\bar{T})d\bar{T}\leq\int_{0}^{T}CR(\bar{T})e^{C\bar{T}}d\bar{T} \leq CTR(T)e^{CT}.$ $$

## 7. Auxiliary Lemmas

###### Lemma 7 (Bernstein inequality) .

Let $X_{i},1\leq i\leq n$ be i.i.d. centred random variables a.s. bounded by $b<\infty$ in absolute value. Set
$\sigma^{2}=\mathbb{E}X_{1}^{2}$ and $S_{n}=\frac{1}{n}\sum\limits_{i=1}^{n}X_{i}$. Then, for all $t>0$,

$$ $P(S_{n}\geq\sqrt{\frac{2\sigma^{2}t}{n}}+\frac{bt}{3n})\leq e^{-t}.$ $$

###### Lemma 8 (Hoeffding inequality) .

Let $X_{i},1\leq i\leq n$ be i.i.d. centred random variables a.s. bounded by $b<\infty$ in absolute value. Set $S_{n}=\frac{1}{n}\sum\limits_{i=1}^{n}X_{i}$, then for all $t>0$,

$$ $P(|S_{n}|\geq b\sqrt{\frac{2t}{n}})\leq 2e^{-t}.$ $$

###### Lemma 9 (Bounded difference inequality) .

Let $X_{1},\cdots,X_{m}\in\mathcal{X}^{m}$ be a set of $m\geq 1$ independent random variables and assume that there exists $c_{1},\cdots,c_{m}$ such that $f:\mathcal{X}^{m}\rightarrow\mathbb{R}$ satisfies the following conditions:

$$ $|f(x_{1},\cdots,x_{i},\cdots,x_{m})-f(x_{1},\cdots,x_{i}^{{}^{\prime}},\cdots, x_{m})|\leq c_{i},$ $$

for all $i\in[m]$ and any points $x_{1},\cdots,x_{m},x_{i}^{{}^{\prime}}\in\mathcal{X}$. Let $f(S)$ denote $f(X_{1},\cdots,X_{m})$, then, for all $\epsilon>0$, the following inequalities hold:

$$ $P(f(S)-\mathbb{E}(f(S))\geq\epsilon)\leq\exp(\frac{-2\epsilon^{2}}{\sum_{i=1}^ {m}c_{i}^{2}})$ $$

$$ $P(f(S)-\mathbb{E}(f(S))\leq-\epsilon)\leq\exp(\frac{-2\epsilon^{2}}{\sum_{i=1} ^{m}c_{i}^{2}})$ $$

###### Lemma 10 (Covering number of $\partial B_{1}^{d}(1)$ in the $L^{1}$ norm) .

For any $\epsilon>0$,

$$ $\mathcal{N}(\partial B_{1}(1),|\cdot|_{1},\epsilon)\leq 2(\frac{12}{\epsilon}) ^{d-1}.$ $$

###### Proof.

By the symmetry of $\partial B_{1}^{d}(1)$, it suffices to consider the set

$$ $S:=\{(x_{1},\cdots,x_{d})\in\partial B_{1}^{d}(1),x_{i}\geq 0,1\leq i\leq d\},$ $$

as $\mathcal{N}(\partial B_{1}(1),|\cdot|_{1},\epsilon)\leq 2^{d}\mathcal{N}(S,|
\cdot|_{1},\epsilon)$.

Note that for $(x_{1},\cdots,x_{d})\in\partial B_{1}^{d}(1)$, $x_{d}$ is determined by $x_{1},\cdots,x_{d-1}$. Thus the problem of estimating the covering number of $\partial B_{1}^{d}(1)$ can be reduced to estimating the covering number of

$$ $S_{1}:=\{(x_{1},\cdots,x_{d-1}):x_{1}+\cdots+x_{d-1}\leq 1,x_{i}\geq 0,1\leq i \leq d-1\},$ $$

which is a subset of $B_{1}^{d-1}(1)$.

By Lemma 5.7 in [], we know $\mathcal{N}(B_{1}^{d-1}(1),|\cdot|_{1},\epsilon)\leq(\frac{2}{\epsilon}+1)^{d-
1}\leq(\frac{3}{\epsilon})^{d-1}$. Thus, there exists a $\frac{\epsilon}{2}$- cover of $B_{1}^{d-1}(1)$ with cardinality $(\frac{6}{\epsilon})^{d-1}$ which we denote by $C$. Although $C$ is also a $\frac{\epsilon}{2}$- cover of $S_{1}$, the elements in $C$ may not belong to $S_{1}$. To fix this issue, we can transform $C$ to a subset of $S_{1}$ and the transformation doesn’t change the property that $C$ is a $\frac{\epsilon}{2}$- cover of $S_{1}$. Specifically, for $(y_{1},\cdots,y_{d-1})\in C$, we do the transformation as follows

$$ $(y_{1},\cdots,y_{d-1})\rightarrow(y_{1}I_{\{y_{1}\geq 0\}},\cdots,y_{d-1}I_{\{ y_{d-1}\geq 0\}}).$ $$

Note that

$$ $y_{1}I_{\{y_{1}\geq 0\}}+\cdots+y_{d-1}I_{\{y_{d-1}\geq 0\}}\leq|y_{1}|+\cdots +|y_{d-1}|\leq 1,$ $$

and for any $(x_{1},\cdots,x_{d-1})\in S_{1}$

$$ $|x_{1}-y_{1}I_{\{y_{1}\geq 0\}}|+\cdots|x_{d-1}-y_{d-1}I_{\{y_{d-1}\geq 0\}}| \leq|x_{1}-y_{1}|+\cdots+|x_{d-1}-y_{d-1}|,$ $$

which imply that after transformation, $C$ is a subset of $S_{1}$ and also a $\frac{\epsilon}{2}$- cover of $S_{1}$. For simplicity, we still denote it by $C$.

Now we are ready to give a $\epsilon$-cover of $S$ via extending $C$ to a subset of $\partial B_{1}^{d}(1)$. Define $C_{e}:=\{(y_{1},\cdots,y_{d}):(y_{1},\cdots,y_{d-1})\in C,y_{d}=1-(y_{1}+
\cdots+y_{d-1})\}$.

Thus for any $(x_{1},\cdots,x_{d})\in S$, since $(x_{1},\cdots,x_{d-1})\in S_{1}$ and $C$ is a $\frac{\epsilon}{2}$-cover of $S_{1}$, there exists a element of $C$, we denote it by $(z_{1},\cdots,z_{d-1})$, such that

$$ $|x_{1}-z_{1}|+\cdots+|x_{d-1}-z_{d-1}|\leq\frac{\epsilon}{2}.$ $$

Note that for $z_{d}=1-(z_{1}+\cdots+z_{d-1})$, $(z_{1},\cdots,z_{d})\in C_{e}$ and

$$ $\displaystyle|x_{1}-z_{1}|+\cdots+|x_{d-1}-z_{d-1}|+|x_{d}-z_{d}|$ $\displaystyle=|x_{1}-z_{1}|+\cdots+|x_{d-1}-z_{d-1}|+|x_{1}-z_{1}+\cdots+x_{d- 1}-z_{d-1}|$ $\displaystyle\leq 2(|x_{1}-z_{1}|+\cdots+|x_{d-1}-z_{d-1}|)$ $\displaystyle\leq\epsilon,$ $$

which implies that $C_{e}$ is a $\epsilon$-cover of $S$.

Recall that $|C_{e}|=|C|=(\frac{6}{\epsilon})^{d-1}$, then
$\mathcal{N}(\partial B_{1}(1),|\cdot|_{1},\epsilon)\leq 2^{d}(\frac{6}{
\epsilon})^{d-1}=2(\frac{12}{\epsilon})^{d-1}.$

Note that in this lemma, our goal is not to investigate the optimal upper bound, but to give an upper bound with explicit dependence on the dimension.

∎

###### Lemma 11 (Equivalence between metric entropy and covering number) .

Let $(T,d)$ be a metric space and there is a continuous and strictly increasing function $f:\mathbb{R}_{+}\rightarrow\mathbb{R}_{+}$ such that for any $\epsilon>0$,

$$ $\mathcal{N}(T,d,\epsilon)\leq f(\epsilon),$ $$

Then for any $\epsilon>0$,

$$ $\epsilon_{n}(T)\leq f^{-1}(n),$ $$

where $f^{-1}$ represents the inverse of $f$.

###### Proof.

It’s obvious, since $\mathcal{N}(T,d,f^{-1}(n))\leq f(f^{-1}(n))=n$.
∎

###### Lemma 12 .

For any $0<x\leq 1$, we have

$$ $\int_{0}^{x}\sqrt{\log\frac{1}{\epsilon}}d\epsilon\leq 2x\sqrt{\log\frac{4}{x}}.$ $$

###### Proof.

For $0<x\leq 1$, let $f(x)=\sqrt{x\log\frac{1}{x}}$, $g(x)=\sqrt{x}$,$h(x)=x\log\frac{1}{x}$, then $f(x)=g(h(x))$. Note that $g$ is increasing and concave and $h$ is concave, thus

$$ $\displaystyle f(\lambda x+(1-\lambda)y)$ $\displaystyle=h(g(\lambda x+(1-\lambda)y))$ $\displaystyle\leq h(\lambda g(x)+(1-\lambda)g(y))$ $\displaystyle\leq\lambda h(g(x))+(1-\lambda)h(g(y))$ $\displaystyle=\lambda f(x)+(1-\lambda)f(y),$ $$

which means $f$ is concave in $[0,1]$.

Let $\epsilon=y^{\frac{3}{2}}$, then

$$ $\displaystyle\int_{0}^{x}\sqrt{\log\frac{1}{\epsilon}}d\epsilon$ $\displaystyle=(\frac{3}{2})^{\frac{3}{2}}\int_{0}^{x^{\frac{2}{3}}}\sqrt{y\log \frac{1}{y}}dy$ $\displaystyle\leq(\frac{3}{2})^{\frac{3}{2}}x^{\frac{2}{3}}\sqrt{\frac{x^{ \frac{2}{3}}}{2}\log\frac{2}{x^{\frac{2}{3}}}}$ $\displaystyle=(\frac{3}{2})^{\frac{3}{2}}x\sqrt{\frac{1}{3}\log\frac{2^{\frac{ 3}{2}}}{x}}$ $\displaystyle\leq 2x\sqrt{\log\frac{4}{x}},$ $$

where the first inequality follows from Jensen’s inequality.

∎

###### Lemma 13 (The remaining part of the proof of Theorem 1) .

For the function class $\mathcal{F}$ and

$$ $\mathcal{G}:=\{(|\nabla u(x)|^{2}-2f(x)u(x))-(|\nabla u^{*}(x)|^{2}-2f(x)u^{*} (x)):\ u\in\mathcal{F}\},$ $$

we assume that for any $\epsilon>0$,

$$ $\mathcal{N}(\mathcal{F},\|\cdot\|_{L^{2}(P_{n})},\epsilon)\leq(\frac{b}{ \epsilon})^{a}\ a.s.\ and\ \mathcal{N}(\mathcal{G},\|\cdot\|_{L^{2}(P_{n})}, \epsilon)\leq(\frac{\beta}{\epsilon})^{\alpha}\ a.s.$ $$

for some positive constants $a,b,\alpha,\beta$, where $b>\sup\limits_{f\in\mathcal{F}}|f|$, $\beta>\sup\limits_{g\in\mathcal{G}}|g|$.

Then we have that with probability at least $1-e^{-t}$

$$ $\displaystyle\sup\limits_{u\in\mathcal{F}_{\delta}}(\mathcal{E}(u)-\mathcal{E} (u^{*}))-(\mathcal{E}_{n}(u)-\mathcal{E}(u^{*}))$ $\displaystyle\leq C(\frac{\alpha M^{2}\log(2\beta\sqrt{n})}{n}+\sqrt{\frac{M^{ 2}\delta\alpha\log(2\beta\sqrt{n})}{n}}+\sqrt{\frac{M^{2}\delta t}{n}}$ $\displaystyle\ \ \ +\frac{M^{2}t}{n}+\sqrt{\frac{aM^{2}\delta}{n}\log\frac{4b} {M}}),$ $$

where

$$ $\mathcal{F}_{\delta}:=\{u\in\mathcal{F}:\ \|u-u^{*}\|_{H^{1}(\Omega)}^{2}\leq \delta\}.$ $$

###### Proof.

As before, rearranging $\sup\limits_{u\in\mathcal{F}_{\delta}}(\mathcal{E}(u)-\mathcal{E}(u^{*}))-(
\mathcal{E}_{n}(u)-\mathcal{E}(u^{*}))$ yields that

$$ $\displaystyle\sup\limits_{u\in\mathcal{F}_{\delta}}(\mathcal{E}(u)-\mathcal{E} (u^{*}))-(\mathcal{E}_{n}(u)-\mathcal{E}(u^{*}))$ $\displaystyle=\sup\limits_{u\in\mathcal{F}(\delta)}[\int_{\Omega}[(|\nabla u(x )|^{2}-2f(x)u(x))-(|\nabla u^{*}(x)|^{2}-2f(x)u^{*}(x))]dx$ $\displaystyle\ \ \ -\frac{1}{n}\sum\limits_{i=1}^{n}[(|\nabla u(X_{i})|^{2}-2f (X_{i})u(X_{i}))-(|\nabla u^{*}(X_{i})|^{2}-2f(X_{i})u^{*}(X_{i}))]$ $\displaystyle\ \ \ +(\int_{\Omega}u(x)dx)^{2}-(\frac{1}{n}\sum\limits_{i=1}^{n }u(X_{i}))^{2}+(\frac{1}{n}\sum\limits_{i=1}^{n}u^{*}(X_{i}))^{2}]$ $\displaystyle\leq\sup\limits_{g\in\mathcal{G}(\delta)}(P-P_{n})g+\sup\limits_{ u\in\mathcal{F}(\delta)}[(\int_{\Omega}u(x)dx)^{2}-(\frac{1}{n}\sum\limits_{i= 1}^{n}u(X_{i}))^{2}]+(\frac{1}{n}\sum\limits_{i=1}^{n}u^{*}(X_{i}))^{2}$ $\displaystyle:=\psi_{n}^{1}(\delta)+\psi_{n}^{2}(\delta)+\psi_{n}^{3}(\delta),$ $$

where

$$ $\mathcal{G}:=\{(|\nabla u(x)|^{2}-2f(x)u(x))-(|\nabla u^{*}(x)|^{2}-2f(x)u^{*} (x)):\ u\in\mathcal{F}\}$ $$

and

$$ $\mathcal{G}(\delta):=\{(|\nabla u(x)|^{2}-2f(x)u(x))-(|\nabla u^{*}(x)|^{2}-2f (x)u^{*}(x)):\ u\in\mathcal{F},\|u-u^{*}\|_{H^{1}(\Omega)}^{2}\leq\delta\}.$ $$

Applying the Hoeffding inequality for $\psi_{n}^{3}(\delta)$, we can obtain that with probabilty at least $1-e^{-t}$

$$ $\psi_{n}^{3}(\delta)=(\frac{1}{n}\sum\limits_{i=1}^{n}u^{*}(X_{i}))^{2}\leq \frac{2M^{2}t}{n}.$ $$

For $\phi_{n}^{(12)}(\delta)$, we can deduce that

$$ $\displaystyle\psi_{n}^{2}(\delta)$ $\displaystyle=\sup\limits_{u\in\mathcal{F}(\delta)}[(\int_{\Omega}u(x)dx)^{2}- (\frac{1}{n}\sum\limits_{i=1}^{n}u(X_{i}))^{2}]$ $\displaystyle=\sup\limits_{u\in\mathcal{F}(\delta)}[(Pu)^{2}-(P_{n}u)^{2}]$ $\displaystyle=\sup\limits_{u\in\mathcal{F}(\delta)}[(Pu)^{2}-((P_{n}u-Pu)+Pu)^ {2}]$ $\displaystyle=\sup\limits_{u\in\mathcal{F}(\delta)}[2Pu((P-P_{n})u)-(P_{n}u-Pu )^{2}]$ $\displaystyle\leq\sqrt{\delta}\sup\limits_{u\in\mathcal{F}(\delta)}|(P-P_{n})u|,$ $$

where the last inequality follows from the fact that for any $u\in\mathcal{F}(\delta)$,

$$ $|Pu|=|\int_{\Omega}udx|=|\int_{\Omega}(u-u^{*})dx|\leq(\int_{\Omega}(u-u^{*})^ {2}dx)^{\frac{1}{2}}\leq\sqrt{\delta}.$ $$

Therefore, to bound $\psi_{n}^{2}(\delta)$, it suffices to bound the empirical process $\sup\limits_{u\in\mathcal{F}(\delta)}|(P-P_{n})u|$. By applying the bounded difference inequality and the symmetrization, we can deduce that with probability at least $1-e^{-t}$

$$ $\displaystyle\sup\limits_{u\in\mathcal{F}(\delta)}|(P-P_{n})u|$ $\displaystyle\leq\mathbb{E}\sup\limits_{u\in\mathcal{F}(\delta)}|(P-P_{n})u|+M \sqrt{\frac{2t}{n}}$ $\displaystyle\leq 2\mathbb{E}\sup\limits_{u\in\mathcal{F}(\delta)}|\frac{1}{n} \sum\limits_{i=1}^{n}\epsilon_{i}u(X_{i})|+M\sqrt{\frac{2t}{n}}$ $\displaystyle\leq 2\mathbb{E}\sup\limits_{u\in\mathcal{F}}|\frac{1}{n}\sum \limits_{i=1}^{n}\epsilon_{i}u(X_{i})|+M\sqrt{\frac{2t}{n}}.$ $$

The first term is the expectation of the empirical process and it can be easily bounded by using Dudley’s inequality.

Specifically,

$$ $\displaystyle\mathbb{E}\sup\limits_{u\in\mathcal{F}}|\frac{1}{n}\sum\limits_{i =1}^{n}\epsilon_{i}u(X_{i})|$ $\displaystyle=\mathbb{E}_{X}\mathbb{E}_{\epsilon}\sup\limits_{u\in\mathcal{F} \cup(-\mathcal{F})}\frac{1}{n}\sum\limits_{i=1}^{n}\epsilon_{i}u(X_{i})$ $\displaystyle\leq\mathbb{E}_{X}[\frac{12}{\sqrt{n}}\int_{0}^{M}\sqrt{\log \mathcal{N}(\mathcal{F}\cup(-\mathcal{F}),\|\cdot\|_{L^{2}(P_{n})},u)}du]$ $\displaystyle\leq\mathbb{E}_{X}[\frac{12}{\sqrt{n}}\int_{0}^{M}\sqrt{\log 2 \mathcal{N}(\mathcal{F},\|\cdot\|_{L^{2}(P_{n})},u)}du]$ $\displaystyle\leq\frac{12}{\sqrt{n}}\int_{0}^{M}\sqrt{\log 2+a\log\frac{b}{u}}du$ $\displaystyle\leq\frac{12}{\sqrt{n}}(\sqrt{\log 2}M+\sqrt{a}b\int_{0}^{\frac{M }{b}}\sqrt{\log\frac{1}{u}}du)$ $\displaystyle\leq\frac{12}{\sqrt{n}}(\sqrt{\log 2}M+2\sqrt{a}M\sqrt{\log\frac{ 4b}{M}})$ $\displaystyle\leq C\sqrt{\frac{aM^{2}}{n}\log\frac{4b}{M}},$ $$

where the fifth inequality follows by the fact that $b>M$ and Lemma 9.

Now, it remains only to bound $\psi_{n}^{1}(\delta)$.

Recall that

$$ $\mathcal{G}(\delta)=\{(|\nabla u(x)|^{2}-2f(x)u(x))-(|\nabla u^{*}(x)|^{2}-2f( x)u^{*}(x)):\ u\in\mathcal{F},\|u-u^{*}\|_{H^{1}(\Omega)}^{2}\leq\delta\}.$ $$

Therefore, we can deduce that $|g|\leq 6M^{2}$ and $Var(g)\leq P(g^{2})\leq 4M^{2}\delta$ for any $g\in\mathcal{G}(\delta)$. Then, from Talagrand’s inequality for empirical processes (Theorem 2.1 in [] with $\alpha=1$), we obtain that with probability at least $1-e^{-t}$

$$ $\sup\limits_{g\in\mathcal{G}(\delta)}(P-P_{n})g\leq 4\mathbb{E}R_{n}(\mathcal{ G}(\delta))+\sqrt{\frac{8M^{2}t\delta}{n}}+\frac{16M^{2}t}{n},$ $$

where $R_{n}(\mathcal{G}(\delta)$ is the empirical Rademacher complexity of function class $R_{n}(\mathcal{G}(\delta)$, i.e.

$$ $R_{n}(\mathcal{G}(\delta)=\mathbb{E}_{\epsilon}\sup\limits_{g\in\mathcal{G}( \delta)}\frac{1}{n}\sum\limits_{i=1}^{n}\epsilon_{i}g(x_{i}).$ $$

Note that $Pg^{2}\leq 4M^{2}\delta$ for any $g\in\mathcal{G}(\delta)$, therefore

$$ $\mathbb{E}R_{n}(\mathcal{G}(\delta))\leq\mathbb{E}R_{n}(g\in\mathcal{G}:Pg^{2} \leq 4M^{2}\delta).$ $$

The right term frequently appears in the articles related to the LRC and can be more easily handled than the term on the left.

By applying Corollary 2.1 in [] under the assumption 1, we know

$$ $\mathbb{E}R_{n}(g\in\mathcal{G}:Pg^{2}\leq 4M^{2}\delta)\leq C(\frac{\alpha M^ {2}\log(2\beta\sqrt{n})}{n}+\sqrt{\frac{M^{2}\delta\alpha\log(2\beta\sqrt{n})} {n}}),$ $$

where $C$ is a universal constant.

∎

###### Lemma 14 .

For the empirical covering number of $\mathcal{F}$ and $\mathcal{G}$ defined in the Lemma 13, we can deduce that

(1) when $\mathcal{F}=\mathcal{F}_{m,1}(B)$, we have

$$ $\mathcal{N}(\mathcal{F},L^{2}(P_{n}),\epsilon)\leq(\frac{cB}{\epsilon})^{m(d+1 )}\ and\ \mathcal{N}(\mathcal{G},L^{2}(P_{n}),\epsilon)\leq(\frac{c\max(MB,B^{ 2})}{\epsilon})^{cmd},$ $$

where $M$ is a upper bound for $|f|$ and $c$ is a universal constant.

(2) when $\mathcal{F}=\Phi(N,L,B)$, we have

$$ $\mathcal{N}(\mathcal{F},L^{2}(P_{n}),\epsilon)\leq(\frac{Cn}{\epsilon})^{CN^{2 }L^{2}(\log N\log L)^{3}}\ and\ \mathcal{N}(\mathcal{G},L^{2}(P_{n}),\epsilon) \leq(\frac{Cn}{\epsilon})^{CN^{2}L^{2}(\log N\log L)^{3}},$ $$

where $C$ is a constant independent of $N,L$ and $n\geq CN^{2}L^{2}(\log N\log L)^{3}$.

###### Proof.

(1) For the function class of two-layer neural networks, recall that

$$ $\mathcal{F}_{m,1}(B)=\{\sum\limits_{i=1}^{m}\gamma_{i}\sigma(\omega_{i}\cdot x +t_{i}):\sum\limits_{i=1}^{m}|\gamma_{i}|\leq B,|\omega_{i}|_{1}=1,t_{i}\in[-1 ,1]\},$ $$

we can just consider the covering number in the $L^{\infty}$ norm.

Without loss of generality, we can assume that $B=1$. Then for

$$ $u_{k}(x)=\sum\limits_{i=1}^{m}\gamma_{i}^{k}\sigma(\omega_{i}^{k}\cdot x+t_{i} ^{k})\in\mathcal{F}_{m,1}(1),k=1,2,$ $$

we have

$$ $\displaystyle|u_{1}(x)-u_{2}(x)|$ $\displaystyle=|\sum\limits_{i=1}^{m}\gamma_{i}^{1}\sigma(\omega_{i}^{1}\cdot x +t_{i}^{1})-\gamma_{i}^{2}\sigma(\omega_{i}^{2}\cdot x+t_{i}^{2})|$ $\displaystyle\leq\sum\limits_{i=1}^{m}|\gamma_{i}^{1}\sigma(\omega_{i}^{1} \cdot x+t_{i}^{1})-\gamma_{i}^{2}\sigma(\omega_{i}^{2}\cdot x+t_{i}^{2})|$ $\displaystyle=\sum\limits_{i=1}^{m}|(\gamma_{i}^{1}-\gamma_{i}^{2})\sigma( \omega_{i}^{1}\cdot x+t_{i}^{1})+\gamma_{i}^{2}(\sigma(\omega_{i}^{1}\cdot x+t _{i}^{1})-\sigma(\omega_{i}^{2}\cdot x+t_{i}^{2}))|$ $\displaystyle\leq\sum\limits_{i=1}^{m}2|\gamma_{i}^{1}-\gamma_{i}^{2}|+|\gamma _{i}^{2}|(|\omega_{i}^{1}-\omega_{i}^{2}|_{1}+|t_{i}^{1}-t_{i}^{2}|),$ $$

where the last inequality follows from that $\sigma$ is bounded by $2$ in absolute value and is 1-Lipschitz continuous.

Therefore, when

$$ $\sum\limits_{i=1}^{m}|\gamma_{i}^{1}-\gamma_{i}^{2}|\leq\frac{\epsilon}{4}\ and\ |\omega_{i}^{1}-\omega_{i}^{2}|_{1}\leq\frac{\epsilon}{4},|t_{i}^{1}-t_{i }^{2}|\leq\frac{\epsilon}{4},1\leq i\leq m,$ $$

we have that $\sup_{x\in\Omega}|u_{1}(x)-u_{2}(x)|\leq\epsilon$, which implies

$$ $\mathcal{N}(\mathcal{F}_{m,1}(1),L^{2}(P_{n}),\epsilon)\leq\mathcal{N}( \mathcal{F}_{m,1}(1),L^{\infty},\epsilon)\leq(\frac{c}{\epsilon})^{m}(\frac{c} {\epsilon})^{m(d-1)}(\frac{c}{\epsilon})^{m}=(\frac{c}{\epsilon})^{m(d+1)},$ $$

where $c$ is a universal constant.

Therefore, $\mathcal{N}(\mathcal{F}_{m,1}(B),L^{2}(P_{n}),\epsilon)\leq(\frac{cB}{\epsilon
})^{m(d+1)}$, where we assume that $B\geq 1$.

Recall that

$$ $\mathcal{G}=\{(|\nabla u(x)|^{2}-2f(x)u(x))-(|\nabla u^{*}(x)|^{2}-2f(x)u^{*}( x)):\ u\in\mathcal{F}\}.$ $$

As $u^{*}$ is fixed and the estimation for the term $f(x)u(x)$ can be done as that for $\mathcal{F}$, thus we only need to estimate the first term.

For

$$ $u_{k}=\sum\limits_{i=1}^{m}\gamma_{i}^{k}\sigma(\omega_{i}^{k}\cdot x+t_{i}^{k })\in\mathcal{F}_{m}(1),k=1,2$ $$

we have

$$ $\displaystyle\||\nabla u_{1}|^{2}-|\nabla u_{2}|^{2}\|_{L^{2}(P_{n})}$ $\displaystyle\leq 2\||\nabla u_{1}-\nabla u_{2}|\|_{L^{2}(P_{n})}$ $\displaystyle\leq 2\|\sum\limits_{i=1}^{m}|\gamma_{i}^{1}\omega_{i}^{1}I_{\{ \omega_{i}^{1}\cdot x+t_{i}^{1}\geq 0\}}-\gamma_{i}^{2}\omega_{i}^{2}I_{\{ \omega_{i}^{2}\cdot x+t_{i}^{2}\geq 0\}}|\|_{L^{2}(P_{n})}$ $\displaystyle\leq 2\sum\limits_{i=1}^{m}\||\gamma_{i}^{1}\omega_{i}^{1}I_{\{ \omega_{i}^{1}\cdot x+t_{i}^{1}\geq 0\}}-\gamma_{i}^{2}\omega_{i}^{2}I_{\{ \omega_{i}^{2}\cdot x+t_{i}^{2}\geq 0\}}|\|_{L^{2}(P_{n})}$ $\displaystyle=2\sum\limits_{i=1}^{m}\||(\gamma_{i}^{1}-\gamma_{i}^{2})\omega_{ i}^{1}I_{\{\omega_{i}^{1}\cdot x+t_{i}^{1}\geq 0\}}+\gamma_{i}^{2}(\omega_{i}^ {1}I_{\{\omega_{i}^{1}\cdot x+t_{i}^{1}\geq 0\}}-\omega_{i}^{2}I_{\{\omega_{i} ^{2}\cdot x+t_{i}^{2}\geq 0\}})|\|_{L^{2}(P_{n})}$ $\displaystyle\leq 2\sum\limits_{i=1}^{m}|\gamma_{i}^{1}-\gamma_{i}^{2}|+2\sum \limits_{i=1}^{m}|\gamma_{i}^{2}|\||\omega_{i}^{1}I_{\{\omega_{i}^{1}\cdot x+t _{i}^{1}\geq 0\}}-\omega_{i}^{2}I_{\{\omega_{i}^{2}\cdot x+t_{i}^{2}\geq 0\}}| \|_{L^{2}(P_{n})}$ $\displaystyle\leq 2\sum\limits_{i=1}^{m}|\gamma_{i}^{1}-\gamma_{i}^{2}|+2\sum \limits_{i=1}^{m}|\gamma_{i}^{2}|(|\omega_{i}^{1}-\omega_{i}^{2}|_{1}+\|I_{\{ \omega_{i}^{1}\cdot x+t_{i}^{1}\geq 0\}}-I_{\{\omega_{i}^{2}\cdot x+t_{i}^{2} \geq 0\}}\|_{L^{2}(P_{n})}),$ $$

where the first inequality follows from that $|\nabla u_{k}|\leq 1,k=1,2$ and the second, third, fourth and the last inequalities follow from the triangle inequality.

Thus if

$$ $\sum\limits_{i=1}^{m}|\gamma_{i}^{1}-\gamma_{i}^{2}|\leq\frac{\epsilon}{4}\ and\ |\omega_{i}^{1}-\omega_{i}^{2}|_{1}+\|I_{\{\omega_{i}^{1}\cdot x+t_{i}^{1 }\geq 0\}}-I_{\{\omega_{i}^{2}\cdot x+t_{i}^{2}\geq 0\}}\|_{L^{2}(P_{n})}\leq \frac{\epsilon}{4},1\leq i\leq m,$ $$

we can deduce that $\||\nabla u_{1}|^{2}-|\nabla u_{2}|^{2}\|_{L^{2}(P_{n})}\leq\epsilon$.

Based on same method in the proof of Proposition 6, the $L^{2}(P_{n})$ covering number of the function class $\{|\nabla u|^{2}:\ u\in\mathcal{F}\}$ can be bounded as

$$ $(\frac{c}{\epsilon})^{m}(\frac{c}{\epsilon})^{(3d-1)m}=(\frac{c}{\epsilon})^{3 md}.$ $$

Combining the result for $\mathcal{F}$, we obtain that

$$ $\mathcal{N}(\mathcal{G},L^{2}(P_{n}),\epsilon)\leq(\frac{c\max(MB,B^{2})}{ \epsilon})^{cmd},$ $$

where $M$ is a upper bound for $|f|$ and $c$ is a universal constant.

(2) Note that the empirical covering number $\mathcal{N}(\mathcal{F},L^{2}(P_{n}),\epsilon)$ can be bounded by the uniform covering number $\mathcal{N}(\mathcal{F},n,\epsilon)$, which is defined as

$$ $\mathcal{N}(\mathcal{F},n,\epsilon):=\sup\limits_{Z_{n}\in\mathcal{X}^{n}} \mathcal{N}(\mathcal{F}|_{Z_{n}},\epsilon,\|\cdot\|_{\infty}),$ $$

where $Z_{n}=(z_{1},\cdots,z_{n})$ and $\mathcal{F}|_{Z_{n}}:=\{(f(z_{1}),\cdots,f(z_{n})):\ f\in\mathcal{F}\}$.

As for the uniform covering numbe, it can be estimated by the pseudo-dimension. Specifically, let $\mathcal{F}$ be a class of function from $\mathcal{X}$ to $[-B,B]$. Then for any $\epsilon>0$, we have

$$ $\mathcal{N}(\mathcal{F},n,\epsilon)\leq(\frac{2enB}{\epsilon Pdim(\mathcal{F}) })^{Pdim(\mathcal{F})}$ $$

for $n\geq Pdim(\mathcal{F})$ (See Theorem 12.2 in []).

From [] and [], we know that

$$ $Pdim(\Psi)\leq CN^{2}L^{2}\log L\log N\ and\ Pdim(D\Psi)\leq CN^{2}L^{2}\log L\log N$ $$

with a constant $C$ independent with $N,L$, where $\Psi$ is a ReLU neural network with width $N$ and depth $L$.

Therefore, we can deduce that for $\mathcal{F}=\Phi(N,L,B)$, we have

$$ $\mathcal{N}(\mathcal{F},L^{2}(P_{n}),\epsilon)\leq(\frac{Cn}{\epsilon})^{CN^{2 }L^{2}(\log N\log L)^{3}}$ $$

and

$$ $\mathcal{N}(\mathcal{G},L^{2}(P_{n}),\epsilon)\leq(\frac{Cn}{\epsilon})^{CN^{2 }L^{2}(\log N\log L)^{3}}$ $$

with a constant $C$ independent of $N,L$ and $n\geq CN^{2}L^{2}(\log N\log L)^{3}$, as the width and depth of $\Phi(N,L,B)$ are $\mathcal{O}(N\log N)$ and $\mathcal{O}(L\log L)$ respectively.

∎

###### Lemma 15 (Estimation of covering numbers for the PINNs) .

(1) For $\mathcal{F}=\mathcal{F}_{m,2}(B)$ with $B\leq\frac{M}{4}$, we have

$$ $\log\mathcal{N}(\bm{\mathcal{F}},d,\epsilon)\leq cmd\log(\frac{b}{\epsilon})$ $$

with a universal constant $c$.

(2) For $\mathcal{F}=\Phi(L,W,S,B;H)$ with $L=\mathcal{O}(1),W=\mathcal{O}(K^{d}),S=\mathcal{O}(K^{d}),B=1,H=\mathcal{O}(1)$, we have

$$ $\log\mathcal{N}(\bm{\mathcal{F}},d,\epsilon)\leq CK^{d}\log(\frac{K}{\epsilon}),$ $$

where $C$ is a constant independent of $K$.

###### Proof.

Recall that

$$ $(Lu-f)^{2}=(-\sum\limits_{i,j=1}^{d}a_{ij}(x)\partial_{i,j}u(x)+\sum\limits_{i =1}^{d}b_{i}(x)\partial_{i}u(x)+c(x)u(x)-f(x))^{2}$ $$

and

$$ $\bm{\mathcal{F}}=\{\bm{u}=(|\Omega|(Lu(x)-f(x))^{2},|\partial\Omega|(u(y)-g(y) )^{2}):\ u\in\mathcal{F}\}.$ $$

(1)
For the two functions $\bm{u}=(|\Omega|(Lu-f)^{2},|\partial\Omega|(u-g)^{2}),\bar{\bm{u}}=(|\Omega|(L
\bar{u}-f)^{2},|\partial\Omega|(\bar{u}-g)^{2})\in\bm{\mathcal{F}}$, where $u,\bar{u}$ belong to $\mathcal{F}_{m}(B)$ and are of the form

$$ $u(x)=\sum\limits_{k=1}^{m}\gamma_{k}\sigma_{2}(\omega_{k}\cdot x+t_{k}),\bar{u }(x)=\sum\limits_{k=1}^{m}\bar{\gamma}_{k}\sigma_{2}(\bar{\omega}_{k}\cdot x+ \bar{t}_{k})$ $$

respectively, we write $\bm{u}$, $\bar{\bm{u}}$ as $(u_{1},u_{2})$ and $(\bar{u}_{1},\bar{u}_{2})$ for simplicity.

As for the samples from $\Omega$ and $\partial\Omega$, we denote their empirical measure as

$$ $P_{N_{1}}:=\frac{1}{N_{1}}\sum\limits_{i=1}^{N_{1}}\delta_{X_{i}^{1}}\ and\ P_ {N_{2}}:=\frac{1}{N_{2}}\sum\limits_{i=1}^{N_{2}}\delta_{X_{i}^{2}},$ $$

respectively.

Now, we are ready to estimate $d(\bm{u},\bar{\bm{u}})$, recall that

$$ $\displaystyle d(\bm{u},\bar{\bm{u}})$ $\displaystyle=\sqrt{\frac{1}{2}}\sqrt{\|u_{1}-\bar{u}_{1}\|_{L^{2}(P_{N_{1}})} ^{2}+\|u_{2}-\bar{u}_{2}\|_{L^{2}(P_{N_{2}})}^{2}}$ $\displaystyle\leq\sqrt{\frac{1}{2}}(\|u_{1}-\bar{u}_{1}\|_{L^{2}(P_{N_{1}})}+ \|u_{2}-\bar{u}_{2}\|_{L^{2}(P_{N_{2}})}),$ $$

which allows us to estimate these two terms separately.

From the boundedness of related functions, we have

$$ $\displaystyle\|u_{1}-\bar{u}_{1}\|_{L^{2}(P_{N_{1}})}$ $\displaystyle=\||\Omega|(Lu-f)^{2}-|\Omega|(L\bar{u}-f)^{2}\|_{L^{2}(P_{N_{1}})}$ $\displaystyle\leq cd^{2}M^{2}|\Omega|\|L(u-\bar{u})\|_{L^{2}(P_{N_{1}})}$ $$

and

$$ $\displaystyle\|u_{2}-\bar{u}_{2}\|_{L^{2}(P_{N_{2}})}$ $\displaystyle=\||\partial\Omega|(u-g)^{2}-|\partial\Omega|(\bar{u}-g)^{2}\|_{L ^{2}(P_{N_{2}})}$ $\displaystyle\leq cM|\partial\Omega|\|u-\bar{u}\|_{L^{2}(P_{N_{2}})}.$ $$

It turns to bound $\|L(u-\bar{u})\|_{L^{2}(P_{N_{1}})}$ and $\|u-\bar{u}\|_{L^{2}(P_{N_{2}})}$.

For $\|L(u-\bar{u})\|_{L^{2}(P_{N_{1}})}$, applying the triangle inequality yields

$$ $\displaystyle\|L(u-\bar{u})\|_{L^{2}(P_{N_{1}})}$ $\displaystyle=\|-\sum\limits_{i,j=1}^{d}a_{ij}\partial_{i,j}(u-\bar{u})+\sum \limits_{i=1}^{d}b_{i}\partial_{i}(u-\bar{u})+c(u-\bar{u})\|_{L^{2}(P_{N_{1}})}$ $\displaystyle\leq\|\sum\limits_{i,j=1}^{d}a_{ij}\partial_{i,j}(u-\bar{u})\|_{L ^{2}(P_{N_{1}})}+\|\sum\limits_{i=1}^{d}b_{i}\partial_{i}(u-\bar{u})\|_{L^{2}( P_{N_{1}})}+\|c(u-\bar{u})\|_{L^{2}(P_{N_{1}})}$ $\displaystyle:=A_{1}+A_{2}+A_{3}.$ $$

Note that $\partial_{i}u,u$ are Lipschitz continuous with respect to the parameters, thus for $A_{2}$, we have

$$ $\displaystyle A_{2}$ $\displaystyle=\|\sum\limits_{i=1}^{d}b_{i}\partial_{i}(u-\bar{u})\|_{L^{2}(P_{ N_{1}})}$ $\displaystyle\leq\|\sum\limits_{i=1}^{d}b_{i}\partial_{i}(u-\bar{u})\|_{L^{ \infty}(\Omega)}$ $\displaystyle=\|\sum\limits_{i=1}^{d}2b_{i}(\sum\limits_{k=1}^{m}\gamma_{k} \omega_{k}^{i}\sigma(\omega_{k}\cdot x+t_{k})-\bar{\gamma}_{k}\bar{\omega}_{k} ^{i}\sigma(\bar{\omega}_{k}\cdot x+\bar{t}_{k}))\|_{L^{\infty}(\Omega)}$ $\displaystyle=\|\sum\limits_{i=1}^{d}2b_{i}(\sum\limits_{k=1}^{m}(\gamma_{k}- \bar{\gamma}_{k})\omega_{k}^{i}\sigma(\omega_{k}\cdot x+t_{k})+\bar{\gamma}_{k }\omega_{k}^{i}\sigma(\omega_{k}\cdot x+t_{k})-\bar{\gamma}_{k}\bar{\omega}_{k }^{i}\sigma(\bar{\omega}_{k}\cdot x+\bar{t}_{k})\|_{L^{\infty}(\Omega)}$ $\displaystyle\leq 4M\sum\limits_{k=1}^{m}|\gamma_{k}-\bar{\gamma}_{k}|+2M\sum \limits_{i=1}^{d}\|\sum\limits_{k=1}^{m}\bar{\gamma}_{k}\omega_{k}^{i}\sigma( \omega_{k}\cdot x+t_{k})-\bar{\gamma}_{k}\bar{\omega}_{k}^{i}\sigma(\bar{ \omega}_{k}\cdot x+\bar{t}_{k})\|_{L^{\infty}(\Omega)},$ $$

where the last inequality follows from the facts that $|b_{i}|\leq M,1\leq i\leq d$ and $\omega_{k}=(\omega_{k}^{1},\cdots,\omega_{k}^{d})$, $\sum\limits_{i=1}^{d}|\omega_{k}^{i}|=1$. And we denote the second term by $A_{22}$,
then

$$ $\displaystyle A_{22}$ $\displaystyle=2M\sum\limits_{i=1}^{d}\|\sum\limits_{k=1}^{m}\bar{\gamma}_{k} \omega_{k}^{i}\sigma(\omega_{k}\cdot x+t_{k})-\bar{\gamma}_{k}\bar{\omega}_{k} ^{i}\sigma(\bar{\omega}_{k}\cdot x+\bar{t}_{k})\|_{L^{\infty}(\Omega)}$ $\displaystyle=2M\sum\limits_{i=1}^{d}\|\sum\limits_{k=1}^{m}\bar{\gamma}_{k}( \omega_{k}^{i}-\bar{\omega}_{k}^{i})\sigma(\omega_{k}\cdot x+t_{k})+\bar{ \gamma}_{k}\bar{\omega}_{k}^{i}(\sigma(\omega_{k}\cdot x+t_{k})-\sigma(\bar{ \omega}_{k}\cdot x+\bar{t}_{k}))\|_{L^{\infty}(\Omega)}$ $\displaystyle\leq 4M\sum\limits_{i=1}^{d}\sum\limits_{k=1}^{m}|\bar{\gamma}_{k }||\omega_{k}^{i}-\bar{\omega}_{k}^{i}|+2M\sum\limits_{i=1}^{d}\sum\limits_{k= 1}^{m}|\bar{\gamma}_{k}||\bar{\omega}_{k}^{i}|(|\omega_{k}-\bar{\omega}_{k}|_{ 1}+|t_{k}-\bar{t}_{k}|)$ $\displaystyle=4M\sum\limits_{k=1}^{m}|\bar{\gamma}_{k}||\omega_{k}-\bar{\omega }_{k}|_{1}+2M\sum\limits_{k=1}^{m}|\bar{\gamma}_{k}|(|\omega_{k}-\bar{\omega}_ {k}|_{1}+|t_{k}-\bar{t}_{k}|),$ $$

where the inequality follows from the triangle inequality and the facts that $\sigma$ is $1$-Lipschitz continuous and $\|\sigma\|_{L^{\infty}([-2,2])}\leq 2$.

Combining the results for $A_{2}$, we have

$$ $A_{2}\leq 4M\sum\limits_{k=1}^{m}|\gamma_{k}-\bar{\gamma}_{k}|+4M\sum\limits_{ k=1}^{m}|\bar{\gamma}_{k}||\omega_{k}-\bar{\omega}_{k}|_{1}+2M\sum\limits_{k=1 }^{m}|\bar{\gamma}_{k}|(|\omega_{k}-\bar{\omega}_{k}|_{1}+|t_{k}-\bar{t}_{k}|).$ $$

Similarly, we have

$$ $\displaystyle A_{3}$ $\displaystyle=\|c(u-\bar{u})\|_{L^{2}(P_{N_{1}})}$ $\displaystyle\leq 4M\sum\limits_{k=1}^{m}|\gamma_{k}-\bar{\gamma}_{k}|+4M\sum \limits_{k=1}^{m}|\bar{\gamma}_{k}|(|\omega_{k}-\bar{\omega}_{k}|_{1}+|t_{k}- \bar{t}_{k}|)$ $$

and

$$ $\|u-\bar{u}\|_{L^{2}(P_{N_{2}})}\leq 4\sum\limits_{k=1}^{m}|\gamma_{k}-\bar{ \gamma}_{k}|+4\sum\limits_{k=1}^{m}|\bar{\gamma}_{k}|(|\omega_{k}-\bar{\omega} _{k}|_{1}+|t_{k}-\bar{t}_{k}|).$ $$

As $A_{1}$ involves the second derivative of $\sigma_{2}$, the method described above cannot be applied. However, we can borrow the idea from the proof of Proposition 6.

$$ $\displaystyle A_{1}$ $\displaystyle=\|\sum\limits_{i,j=1}^{d}a_{ij}\partial_{i,j}(u-\bar{u})\|_{L^{2 }(P_{N_{1}})}$ $\displaystyle=2\|\sum\limits_{k=1}^{m}\gamma_{k}{\omega_{k}}^{T}A\omega_{k}I_{ \{\omega_{k}\cdot x+t_{k}\geq 0\}}-\bar{\gamma}_{k}{\bar{\omega}_{k}}^{T}A\bar {\omega}_{k}I_{\{\bar{\omega}_{k}\cdot x+\bar{t}_{k}\geq 0\}}\|_{L^{2}(P_{N_{1 }})}$ $\displaystyle=2\|\sum\limits_{k=1}^{m}(\gamma_{k}{\omega_{k}}^{T}A\omega_{k}- \bar{\gamma}_{k}{\bar{\omega}_{k}}^{T}A\bar{\omega}_{k})I_{\{\omega_{k}\cdot x +t_{k}\geq 0\}}+\bar{\gamma}_{k}{\bar{\omega}_{k}}^{T}A\bar{\omega}_{k}(I_{\{ \omega_{k}\cdot x+t_{k}\geq 0\}}-I_{\{\bar{\omega}_{k}\cdot x+\bar{t}_{k}\geq 0 \}})\|_{L^{2}(P_{N_{1}})}$ $\displaystyle\leq 2\sum\limits_{k=1}^{m}|\gamma_{k}{\omega_{k}}^{T}A\omega_{k} -\bar{\gamma}_{k}{\bar{\omega}_{k}}^{T}A\bar{\omega}_{k}|+2\sum\limits_{k=1}^{ m}|\bar{\gamma}_{k}{\bar{\omega}_{k}}^{T}A\bar{\omega}_{k}|\|I_{\{\omega_{k} \cdot x+t_{k}\geq 0\}}-I_{\{\bar{\omega}_{k}\cdot x+\bar{t}_{k}\geq 0\}}\|_{L^ {2}(P_{N_{1}})}.$ $$

For the first term, we have

$$ $\displaystyle\sum\limits_{k=1}^{m}|\gamma_{k}{\omega_{k}}^{T}A\omega_{k}-\bar{ \gamma}_{k}{\bar{\omega}_{k}}^{T}A\bar{\omega}_{k}|$ $\displaystyle\leq\sum\limits_{k=1}^{m}|(\gamma_{k}-\bar{\gamma}_{k}){\omega_{k }}^{T}A\omega_{k}|+|\bar{\gamma}_{k}({\omega_{k}}^{T}A\omega_{k}-{\bar{\omega} _{k}}^{T}A\bar{\omega}_{k})|$ $\displaystyle\leq\sum\limits_{k=1}^{m}M|\gamma_{k}-\bar{\gamma}_{k}|+|\bar{ \gamma}_{k}||{\omega_{k}}^{T}A(\omega_{k}-\bar{\omega}_{k})+{\bar{\omega}_{k}} ^{T}A(\omega_{k}-\bar{\omega}_{k})|$ $\displaystyle\leq M\sum\limits_{k=1}^{m}|\gamma_{k}-\bar{\gamma}_{k}|+2|\bar{ \gamma}_{k}||\omega_{k}-\bar{\omega}_{k}|_{1},$ $$

where the ineqaulities follow from the triangle inequality and the fact that for any $x\in\partial B_{1}^{d}(1),y\in\mathbb{R}^{d}$ and matrix $A\in\mathbb{R}^{d\times d}$ with $|A(i,j)|\leq M(1\leq i,j\leq d)$,
we have $|x^{T}Ay|=|(A^{T}x)^{T}y|\leq|A^{T}x|_{\infty}|y|_{1}\leq M|y|_{1}$.

Thus we obtain the final upper bound for $A_{1}$.

$$ $A_{1}\leq 2M\sum\limits_{k=1}^{m}(|\gamma_{k}-\bar{\gamma}_{k}|+2|\bar{\gamma} _{k}||\omega_{k}-\bar{\omega}_{k}|_{1})+2M\sum\limits_{k=1}^{m}|\bar{\gamma}_{ k}|\|I_{\{\omega_{k}\cdot x+t_{k}\geq 0\}}-I_{\{\bar{\omega}_{k}\cdot x+\bar{t }_{k}\geq 0\}}\|_{L^{2}(P_{N_{1}})}.$ $$

Combining all results above, we can deduce that

$$ $\displaystyle d(\bm{u},\hat{\bm{u}})$ $\displaystyle\leq c(d^{2}M^{3}|\Omega|+M|\partial\Omega|)(\sum\limits_{k=1}^{m }(|\gamma_{k}-\bar{\gamma}_{k}|+|\bar{\gamma}_{k}||\omega_{k}-\bar{\omega}_{k} |_{1})$ $\displaystyle\ \ \ +\sum\limits_{k=1}^{m}|\bar{\gamma}_{k}|\|I_{\{\omega_{k} \cdot x+t_{k}\geq 0\}}-I_{\{\bar{\omega}_{k}\cdot x+\bar{t}_{k}\geq 0\}}\|_{L^ {2}(P_{N_{1}})}).$ $$

Similar to bound the empirical covering number of $\mathcal{G}$ for the two-layer neural networks in Lemma 14(1), the covering number of $\bm{\mathcal{F}}$ under $d$ is

$$ $(\frac{c(d^{2}M^{3}|\Omega|+M|\partial\Omega|)B}{\epsilon})^{cmd}\leq(\frac{c( d^{2}M^{4}|\Omega|+M^{2}|\partial\Omega|)}{\epsilon})^{cmd}\leq(\frac{cb}{ \epsilon})^{cmd},$ $$

where $c$ is a universal constant.

(2) Note that $d(\bm{u},\bar{\bm{u}})\leq C\|u-\bar{u}\|_{C^{2}(\bar{\Omega})}$, then proposition 1 [] implies that

$$ $\log\mathcal{N}(\bm{\mathcal{F}},\|\cdot\|_{C^{2}(\bar{\Omega})},\epsilon)\leq CK ^{d}\log\frac{K}{\epsilon},$ $$

where $C$ is a constant independent of $K$.

Therefore, the conclusion holds.
∎

###### Lemma 16 ( [ 65 ] ) .

For $u\in H^{\frac{1}{2}}(\Omega)\cap L^{2}(\partial\Omega)$,

$$ $\displaystyle\|u\|_{H^{\frac{1}{2}}(\Omega)}^{2}$ $\displaystyle\leq C\|-\sum\limits_{i,j=1}^{d}a_{ij}\partial_{ij}u+\sum\limits_ {i=1}^{d}b_{i}\partial_{i}u+cu\|_{H^{-\frac{3}{2}}(\Omega)}^{2}+C\|u\|_{L^{2}( \partial\Omega)}^{2}$ $\displaystyle\leq C_{\Omega}(\|-\sum\limits_{i,j=1}^{d}a_{ij}\partial_{ij}u+ \sum\limits_{i=1}^{d}b_{i}\partial_{i}u+cu\|_{L^{2}(\Omega)}^{2}+\|u\|_{L^{2}( \partial\Omega)}^{2}),$ $$

where $C_{\Omega}$ is a constant that depends only on $\Omega$.

###### Lemma 7 (Bernstein inequality) .

###### Lemma 8 (Hoeffding inequality) .

###### Lemma 9 (Bounded difference inequality) .

###### Lemma 10 (Covering number of ∂ B 1 d ⁢ ( 1 ) superscript subscript 𝐵 1 𝑑 1 \partial B_{1}^{d}(1) ∂ italic_B start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_d end_POSTSUPERSCRIPT ( 1 ) in the L 1 superscript 𝐿 1 L^{1} italic_L start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT norm) .

###### Proof.

###### Lemma 11 (Equivalence between metric entropy and covering number) .

###### Proof.

###### Lemma 12 .

###### Proof.

###### Lemma 13 (The remaining part of the proof of Theorem 1) .

###### Proof.

###### Lemma 14 .

###### Proof.

###### Lemma 15 (Estimation of covering numbers for the PINNs) .

###### Proof.

###### Lemma 16 ( [ 65 ] ) .

## 8. Discussions

### 8.1. Over-parametrization

In the context of over-parametrization, the generalization bounds for two-layer neural networks may become less meaningful due to the term $m/n$. However, fortunately, the function class of two-layer neural networks forms a convex hull of a function class with a covering number similar to that of VC-classes. Consequently, we can extend the convex hull entropy theorem (Theorem 2.6.9 in []) to the $H^{1}$ norm, allowing us to derive generalization bounds that are independent of the network’s width. Theorem 5 is a modification of Theorem 2.6.9 in [] to obtain explicit dependence on the dimension.

###### Lemma 17 .

Let $\mathcal{F}$ be arbitrary set consisting of $n$ measurable function $f:\Omega\rightarrow\mathbb{R}$ of finite $H^{1}(Q)$-diameter $\text{diam}(\mathcal{F})$. Then for every $\epsilon>0$, we have

$$ $\mathcal{N}(\epsilon\text{diam}(\mathcal{F}),\text{conv}(\mathcal{F}),H^{1}(Q) )\leq(e+\frac{en\epsilon^{2}}{2})^{\frac{2}{\epsilon^{2}}}.$ $$

###### Proof.

Assume that $\mathcal{F}=\{f_{1},\cdots,f_{n}\}$. For given $\lambda$ in the $n$-dimensional simplex. Let $Y_{1},\cdots,Y_{k}$ be i.i.d. random elements such that $P(Y_{1}=f_{j})=\lambda_{i}$ for $j=1,\cdots,k$ and $k$ is natural number to be determined. Then we have

$$ $\mathbb{E}Y_{i}=\sum\limits_{j=1}^{n}\lambda_{j}f_{j}\ and\ \nabla\mathbb{E}Y_ {i}=\mathbb{E}\nabla Y_{i}=\sum\limits_{j=1}^{n}\lambda_{j}\nabla f_{j}.$ $$

Let $\bar{Y}_{k}=\frac{1}{k}\sum_{i=1}^{k}Y_{i}$, then the independence implies

$$ $\mathbb{E}\|\bar{Y}_{k}-\mathbb{E}Y_{1}\|_{H^{1}(Q)}^{2}=\frac{1}{k^{2}}\sum \limits_{i=1}^{k}\mathbb{E}\|Y_{i}-\mathbb{E}Y_{1}\|_{H^{1}(Q)}^{2}\leq\frac{1 }{k}(\text{diam}(\mathcal{F}))^{2}.$ $$

Therefore, Markov inequality implies that there is at least one realization of $\bar{Y}_{k}$ that have $H^{1}(Q)$-distance at most $k^{-1/2}\text{diam}(\mathcal{F})$ to the convex combination $\sum_{j=1}^{n}\lambda_{j}f_{j}$. Note that every realization has the form $k^{-1}\sum_{i=1}^{k}f_{i_{k}}$, where some functions $f_{j}$ in the set $\mathcal{F}$ may be used multiple times. As such forms are at most $C_{n+k-1}^{k}$, we can deduce that

$$ $\mathcal{N}(k^{-1/2}\text{diam}(\mathcal{F}),\text{conv}(\mathcal{F}),H^{1}(Q) )\leq C_{n+k-1}^{k}\leq e^{k}(1+\frac{n}{k})^{k},$ $$

where the last inequality follows from Stirling’s inequality.

For $0<\epsilon<1$, we can take $k=\lceil\frac{1}{\epsilon^{2}}\rceil$, then the monotonicity of the function $e^{k}(1+\frac{n}{k})^{k}$ and the fact $k\leq\frac{1}{\epsilon^{2}}+1\leq\frac{2}{\epsilon^{2}}$ imply that

$$ $e^{k}(1+\frac{n}{k})^{k}\leq(e+\frac{en\epsilon^{2}}{2})^{\frac{2}{\epsilon^{2 }}}.$ $$

For $\epsilon>1$, the right term is larger than $1$, thus the conclusion holds directly. ∎

###### Theorem 5 .

Let $Q$ be a probability on $\Omega$, and let $\mathcal{F}$ be a class of measurable functions with $\|F\|_{Q,2}:=\sup\limits_{f\in\mathcal{F}}\|f\|_{H^{1}(Q)}<\infty$ and

$$ $\mathcal{N}(\epsilon\|F\|_{Q,2},\mathcal{F},H^{1}(Q))\leq C(\frac{1}{\epsilon} )^{V},\ 0<\epsilon<1$ $$

for some $V\geq 1$. Then we have

$$ $\log\mathcal{N}(\epsilon\|F\|_{Q,2},\text{conv}(\mathcal{F}),H^{1}(Q))\leq KV( C^{\frac{1}{V}}+2)^{\frac{2V}{V+2}}(\frac{1}{\epsilon})^{\frac{2V}{V+2}},$ $$

where $K$ is a universal constant.

###### Proof.

Note that every element in the convex hull of $\mathcal{F}$ has distance $\epsilon$ to the convex hull of an $\epsilon$-net over $\mathcal{F}$. Accordingly, given a fixed $\epsilon$, it suffices to consider scenarios where the set $\mathcal{F}$ is finite.

Set $W=\frac{1}{2}+\frac{1}{V}$ and $L=C^{1/V}\|F\|_{Q,2}$. Then the assumption implies that $\mathcal{F}$ can be covered by $n$ balls of radius at most $Ln^{-1/V}$ for every natural number $n$. Form sets $\mathcal{F}_{1}\subset\mathcal{F}_{2}\subset\cdots\subset\mathcal{F}$ such that for each $n$, the set $\mathcal{F}_{n}$ is a maximal, $Ln^{-1/V}$-separated net over $\mathcal{F}$. Thus $\mathcal{F}_{n}$ has at most $n$ elements. We will show by induction that there exist constant $C_{k}$ and $D_{k}$ depending only on $C$ and $V$ such that $\sup_{k}C_{j}\lor D_{k}<\infty$ and for $q\geq 3V$,

$$ $\log\mathcal{N}(C_{k}Ln^{-W},\text{conv}(\mathcal{F}_{nk^{q}}),H^{1}(Q))\leq D _{k}n,\ n,k\geq 1.$ $$

The proof consists of a nested induction argument. The outer layer is induction on $k$ and the inner layer is induction on $n$.

First, we apply induction for $n$, i.e., for $k=1$, we will prove the conclusion for each $n$. For fixed $n_{0}=10$, it suffices to choose $C_{1}L{n_{0}}^{-W}=C_{1}L{10}^{-W}\geq\|F\|_{Q,2}$ so that the statement is trivially ture for $n\leq n_{0}=10$, i.e., $C_{1}\geq 10^{W}C^{-1/V}$. For $10<n\leq 100$, set $m=\lfloor\frac{n}{10}\rfloor$, thus $1\leq m\leq 10$. By the definition of $\mathcal{F}_{m}$, each $f\in\mathcal{F}_{n}-\mathcal{F}_{m}$ has distance at most $Lm^{-1/V}$ of some element $\pi_{m}f$ of $\mathcal{F}_{m}$. Thus each element of $\text{conv}(\mathcal{F})$ can be written as

$$ $\sum\limits_{f\in\mathcal{F}_{n}}\lambda_{f}f=\sum\limits_{f\in\mathcal{F}_{m} }\mu_{f}f+\sum\limits_{f\in\mathcal{F}_{n}-\mathcal{F}_{m}}\lambda_{f}(f-\pi_{ m}f),$ $$

where $\mu_{f}\geq 0$ and $\sum\mu_{f}=\sum\lambda_{f}=1$. Taking $\mathcal{G}$ as the set of function $f-\pi_{m}f$ with $f$ ranging over $\mathcal{F}_{n}-\mathcal{F}_{m}$, thus $\text{conv}(\mathcal{F}_{n})\subset\text{conv}(\mathcal{F}_{m})+\text{conv}(
\mathcal{G}_{n})$ for a set $\mathcal{G}_{n}$ consisting of at most $n$ elements, each of norm smaller than $Lm^{-1/V}$, then $\text{diam}(\mathcal{G}_{n})\leq 2Lm^{-1/V}$. Applying Lemma 17 for $\mathcal{G}_{n}$ with $\epsilon$ defined by $m^{-1/V}\epsilon=\frac{1}{4}C_{1}n^{-W}$, i.e., $\epsilon\text{diam}(\mathcal{G}_{n})\leq\frac{1}{2}C_{1}Ln^{-W}$, we can find a $\frac{1}{2}C_{1}Ln^{-W}$-net over $\text{conv}(\mathcal{G}_{n})$ consisting of at most

$$ $(e+\frac{en\epsilon^{2}}{2})^{2/\epsilon^{2}}=\left(e+\frac{eC_{1}^{2}}{32}( \frac{m}{n})^{\frac{2}{V}}\right)^{\frac{32n}{C_{1}^{2}}(\frac{n}{m})^{\frac{2 }{V}}}\leq\left(e+\frac{eC_{1}^{2}}{32}(\frac{1}{20})^{\frac{2}{V}}\right)^{ \frac{32n}{C_{1}^{2}}20^{\frac{2}{V}}}$ $$

elements, where the inequality follows from the facts that $(e+enx)^{\frac{1}{x}}$ is increasing with respect to $x>0$ and $\lfloor\frac{n}{10}\rfloor\geq\frac{1}{2}\frac{n}{10}$ for $n\geq 10$. Applying the induction hypothesis to $\mathcal{F}_{m}$ to find a $C_{1}Lm^{-W}$-net over $\text{conv}(\mathcal{F}_{m})$ consisting of at most $e^{m}$ elements, where we choose $D_{1}=1$. This defines a partition of $\text{conv}(\mathcal{F}_{m})$ into $m$-dimensional sets of radius at most $C_{1}Lm^{-W}$. Without loss of generality, we can assume that $\mathcal{F}_{m}=\{f_{i_{1}},f_{i_{2}},\cdots,f_{i_{m}}\}$. For any fixed element $h$ in the $C_{1}Lm^{-W}$-net over $\text{conv}(\mathcal{F}_{m})$, assume that $h=\lambda_{1}f_{i_{1}}+\cdots\lambda_{m}f_{i_{m}}$ for $\lambda=(\lambda_{1},\cdots,\lambda_{m})\in\mathbb{R}^{m}$. And we denote the ball centered at $h$ with $H^{1}(Q)$ radius $C_{1}Lm^{-W}$ by

$$ $H:=\{\bar{\lambda}=(\bar{\lambda}_{1},\cdots,\bar{\lambda}_{m})\in\mathcal{A}: \bar{h}=\bar{\lambda}_{1}f_{i_{1}}+\cdots+\bar{\lambda}_{m}f_{i_{m}},\|\bar{h} -h\|_{H^{1}(Q)}\leq C_{1}Lm^{-W}\},$ $$

where $\mathcal{A}$ is a subset of $\mathbb{R}^{m}$.

Note that

$$ $\displaystyle\|h-\bar{h}\|_{H^{1}(Q)}$ $\displaystyle=\|\lambda_{1}f_{i_{1}}+\cdots\lambda_{m}f_{i_{m}}-\bar{\lambda}_ {1}f_{i_{1}}-\cdots-\bar{\lambda}_{m}f_{i_{m}}\|_{H^{1}(Q)}$ $\displaystyle\leq|\lambda_{1}-\bar{\lambda}_{1}|\|f_{i_{1}}\|_{H^{1}(Q)}+ \cdots+|\lambda_{m}-\bar{\lambda}_{m}|\|f_{i_{m}}\|_{H^{1}(Q)}$ $\displaystyle\leq(|\lambda_{1}-\bar{\lambda}_{1}|+\cdots+|\lambda_{m}-\bar{ \lambda}_{m}|)\|F\|_{Q,2}.$ $$

Thus if $\|\lambda-\bar{\lambda}\|_{1}\leq C_{1}C^{1/V}m^{-W}$, then
$\|h-\bar{h}\|_{H^{1}(Q)}\leq C_{1}Lm^{-W}$. Therefore, $\mathcal{A}\subset\{\bar{\lambda}\in\mathbb{R}^{m}:\|\bar{\lambda}-\lambda\|_{
1}\leq C_{1}C^{1/V}m^{-W}\}$. By Lemma 5.7 in [], we can find a $\frac{1}{2}C_{1}C^{1/V}n^{-W}$-net of $\mathcal{A}$ under the distance $\|\cdot\|_{1}$ consisting of at most

$$ $\left(\frac{6C_{1}C^{1/V}m^{-W}}{\frac{1}{2}C_{1}C^{1/V}n^{-W}}\right)^{m}=(12 (\frac{n}{m})^{W})^{m}\leq(12(20)^{W})^{\frac{n}{10}}$ $$

elements. Moreover, it yields a $\frac{1}{2}C_{1}Ln^{-W}$-net of $H$ under $H^{1}(Q)$. Select a function from each of the given sets. Then, construct all possible combinations of the sums $f+g$ by preceding procedure, where $f$ is associated with $\text{conv}(\mathcal{F}_{m})$ and $g$ is associated with $\text{conv}(\mathcal{G}_{n})$. These form a $C_{1}Ln^{-W}$-net over $\text{conv}(\mathcal{F}_{n})$ of cardinality bounded by

$$ $e^{n/10}(12(20)^{W})^{n/10}(e+\frac{eC_{1}^{2}}{32}(\frac{1}{20})^{\frac{2}{V} })^{\frac{32(20)^{\frac{2}{V}}n}{C_{1}^{2}}}.$ $$

This is bounded by $e^{n}$ for some suitable choice of $C_{1}$. Specifically, note that for $V\geq 1$, the term attains the maximum at $V=1$, thus it is bounded by

$$ $e^{n/10}(12(20)^{\frac{3}{2}})^{n/10}(e+\frac{eC_{1}^{2}}{32\cdot 400})^{\frac {32\cdot 400n}{C_{1}^{2}}}.$ $$

We can just take $C_{1}=1000$. This concludes the proof for $k=1$ and $10<n\leq 100$. Proceeding in the same way yields that the conclusion holds for every $n$.

We continue by induction on $k$. By a similar construction as before, $\text{conv}(\mathcal{F}_{nk^{q}})\subset\text{conv}(\mathcal{F}_{n(k-1)^{q}})+
\text{conv}(\mathcal{G}_{n,k})$ for a set $\text{conv}(\mathcal{G}_{n,k})$ containing at most $nk^{q}$ elements, each of norm smaller than $L(n(k-1)^{q})^{-1/V}$, so that $\text{conv}(\mathcal{G}_{n,k})\leq 2Ln^{-1/V}k^{-q/V}2^{q/V}$. Applying Lemma 17 to $\text{conv}(\mathcal{G}_{n,k})$ with $\epsilon=2^{-1}k^{q/V-2}2^{-q/V}n^{-1/2}$, we can find an $Lk^{-2}n^{-W}$-net over $\text{conv}(\mathcal{G}_{n,k})$ consisting of at most

$$ $(e+\frac{enk^{q}\epsilon^{2}}{2})^{\frac{2}{\epsilon^{2}}}=\left(e+\frac{ek^{q +\frac{2q}{V}-4}}{2^{\frac{2q}{V}+3}}\right)^{n2^{\frac{2q}{V}+3}k^{4-\frac{2q }{V}}}$ $$

elements. Apply the induction hypothesis to obatin a $C_{k-1}Ln^{-W}$-net over the set $\text{conv}(\mathcal{F}_{n(k-1)^{q}})$ with respect to $H^{1}(Q)$ consisting at most $e^{D_{k-1}n}$ elements. Combine the nets as before to obtain a $C_{k-1}Ln^{-W}$-net over $\text{conv}(\mathcal{F}_{nk^{q}})$ consisting of at most $e^{D_{k}n}$ elements, for

$$ $\displaystyle C_{k}$ $\displaystyle=C_{k-1}+\frac{1}{k^{2}},$ $\displaystyle D_{k}$ $\displaystyle=D_{k-1}+2^{\frac{2q}{V}+3}\frac{1+\log(1+2^{-\frac{2q}{V}-3}k^{q +\frac{2q}{V}-4})}{k^{2(\frac{q}{V}-2)}}.$ $$

For $2(\frac{q}{V}-2)\geq 2$, the resulting sequences $C_{k}$ and $D_{k}$ are bounded. By setting $q=3V$, i.e., $2(\frac{q}{V}-2)=2$, we have

$$ $D_{k}=D_{k-1}+2^{9}\frac{1+\log(1+2^{-9}k^{3V+2})}{k^{2}}.$ $$

Therefore, for any $k$, we can deduce that $C_{k}\leq C_{1}+2$ and
$D_{k}\leq D_{1}+KV$, where $K$ is a universal constant. Recall that $C_{1}=\max(10^{W}C^{-1/V},1000)$, thus $\sup_{k}C_{k}\leq\max(10^{W}C^{-1/V},1000)+2$.

Finally,

$$ $\log\mathcal{N}(\epsilon\|F\|_{Q,2},\text{conv}(\mathcal{F}),H^{1}(Q))\leq\sup \limits_{k}D_{k}\left(\frac{C_{k}C^{\frac{1}{V}}}{\epsilon}\right)^{\frac{2V}{ V+2}}\leq KV(C^{\frac{1}{V}}+2)^{\frac{2V}{V+2}}(\frac{1}{\epsilon})^{\frac{2V }{V+2}},$ $$

where $K$ is a universal constant.
∎

For the function class of two-layer neural networks considered in the DRM, $\mathcal{F}=\{\sigma(\omega\cdot x+t),-\sigma(\omega\cdot x+t),0:|\omega|_{1}=
1,t\in[-1,1)\}$, thus
for any probability measure $Q$ on $[0,1]^{d}$, we have $\|F\|_{Q,2}\leq 3$ and

$$ $\mathcal{N}(\epsilon\|F\|_{Q,2},\mathcal{F},H^{1}(Q))\leq C(d+1)(4e)^{d+1}( \frac{C}{\epsilon})^{3d},$ $$

where $C$ is a universal constant. Thus, Theorem 5 implies that

$$ $\log\mathcal{N}(\epsilon\|F\|_{Q,2},\text{conv}(\mathcal{F}),H^{1}(Q))\leq Kd( \frac{1}{\epsilon})^{\frac{6d}{3d+2}},$ $$

where $K$ is a universal constant.

Applying Theorem 5 for deriving the generalization error for the static Schrödinger equation, we can deduce that the fixed point $r^{*}$ satisfies

$$ $r^{*}\lesssim d^{\frac{3}{2}}(\frac{1}{n})^{\frac{1}{2}+\frac{1}{2(3d+1)}},$ $$

which yields a meaningful generalization bound in the setting of over-parametrization.

###### Lemma 17 .

###### Proof.

###### Theorem 5 .

###### Proof.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See the weaknesses.
soundness: 3
strengths: * The proposed theoretical results show that the function $\mathcal{B}^2$ can be well approximated in the $H^1$ norm by MLP (two hidden layers with ReLU as the activation function) network with controlled weights.  
* For the DRM, authors obtain more precise generalization bounds for the Poisson and Schrödinger equations with Neumann boundary condition, irrespective of whether the solutions are in Barron spaces or Sobolev spaces.
* For the PINNs, authors provide a generalization error for PINN loss of the linear second order elliptic equation in the $H^{1/2}$ norm. 
* Theoretical estimations for both methods with neural networks are correct for both cases if the PDE solution is in the Barron space or the Sobolev space.
summary: Authors utilize the localized analysis to refine generalization bounds for both the Deep Ritz Method (DRM) and Physics-Informed Neural Networks (PINNs). For the DRM, authors provide theoretical results for the Poisson and Schrödinger equations. For PINN, the theoretical results obtained for the linear second order elliptic equation.
weaknesses: * Considering PDEs(the Poisson equation and the static Schrödinger equation with Neumann boundary condition) defined on d-dimensional cube are simple and not interested in community. Is it possible to extend the list of PDEs where proposed theory can be used?
* It will be good for understanding to add some numerical experiments.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 1
questions: See weakness above.
soundness: 2
strengths: The authors have performed a mathematical analysis of the Deep Ritz method and PINNs, which continue to attract considerable interest. This mathematical analysis can aid in understanding the characteristics of the models.
summary: The paper theoretically analyzes the generalization error of two prominent deep learning methods for solving PDEs: the Deep Ritz method and PINNs. The analysis is conducted on simple second-order linear PDEs, and under certain assumptions, they obtain tighter bounds compared to previous studies. However, the assumptions made are somewhat impractical, and the results raise questions about the significance of this research beyond its mathematical implications and what it contributes to the field of deep learning.
weaknesses: A robust mathematical analysis can be more powerful than empirical observations. However, there are several areas of concern in this study:

1.	What is the intention behind studying the two methods using different mathematical tools for different PDEs? There is no connection drawn between the two methods within the paper, which leads to confusion about the main message of the study.

2.	Due to the complexity of deep learning models, it is a natural approach to start the analysis with simpler scenarios. However, the assumption that the solutions of the PDEs lie within the Barron space is too strong. The Barron space is specifically designed for functions that are best approximated by two-layer MLPs and is suitable for analyzing two-layer neural network structures, but it is too narrow to represent the space of PDE solutions.

3.	The primary goal of Deep Ritz and PINNs is not just to minimize the loss but to find the solution to PDEs. Thus, instead of focusing on how much a minimizer of the empirical loss reduces the expectation loss, the analysis should focus on how close the empirical loss minimizer is to the true solution. It is crucial because both methods include unbounded derivative operators in their loss functions, meaning that a small difference in loss values does not necessarily imply that the functions are close. Although the paper briefly addresses this in equations (16) and (30), it does so under very restrictive assumptions, casting doubt on the implications of the results.

4.	While the theoretical analysis uses the $ReLU^k$ activation function (also known as RePU) for convenience, this also has limitations. The theory is developed in the context of simple two-layer networks, but practical implementations often involve deeper networks, where the power of the RePU increases with depth, leading to floating-point precision issues. This limitation should be acknowledged in the paper.
5.	The PDEs considered in this study are too simple. Such simple PDEs can be solved much faster and more accurately using classical methods such as FEM, FDM, Discrete Galerkin, and FVM. Deep learning-based methods should focus on more complex PDEs or inverse problems. While it is important to start from a mathematical understanding of simpler problems, it is also important to recognize that these methods are not just mathematical objects. Research in this area must integrate PDEs, classical numerical methods, and deep learning, rather than excluding any one aspect. Furthermore, the significance of this research in the field of deep learning remains unclear beyond its mathematical implications.

6.	Eq (16) combines the results of Prop 1 and Thm 3, but it appears to omit the Poincaré constant from Prop 1, which is dimension-dependent.


7.	Limitations are not addressed in the paper.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. One thing the authors emphasized as a contribution is that this manuscript derived a generalization bound for the Poisson equation compared to [2]. However, it seems to the reviewer that Proposition 1, which provides a variational formulation of Poisson equation, actually requires extra assumption ($\int_{\Omega}f dx = 0$) compared to [2]. Hence, the review wonders whether this is a fair comparison and would appreciates it if the authors can elaborate on why they think the main reason is actually because the expectation of empirical loss is not equal to the population loss under the variational formulation (line 111-112, line 129-130). 

2. In Remark 2, the authors state that the convergence rate associated with the DRM gets improved from $n^{-\frac{2k-2}{d+2k-4}}$ to $n^{-\frac{2k-2}{d+2k-2}}$, which seems to be incorrect as the second rate is larger than the first one. Also, the same rate (up to $\log n$ factors) has also been achieved in [2] via the peeling method. Therefore, the reviewer would appreciate it if the author could specify the major novelty/contribution of this paper for the DRM. 

3. For the generalization bounds on PINN, the authors claimed that results presented in Section 4 are more general compared to [2] as this paper doesn't require strong convexity of the objective function. However, it seems to the reviewer that this cannot be claimed as a major novelty as Lemma 17 serves a similar role as Theorem B.2 in [2] (Also, here the authors still need the strict elliptic condition). Furthermore, even though the convergence rate $n^{-\frac{2k-4}{d+2k-4}}$ of PINN attained here is the same as that of [2], the norm used for measuring the error seems to be different - (30) implies that the norm used here is the $H^{\frac{1}{2}}$ norm, while [2] uses the $H^1$ norm. This will for sure influence the convergence rate and statistical optimality, so the reviewer would appreciate it if the authors could provide some intuition on the change of norms here. 

4. Given that [2] provides not only upper bounds but also informational theoretical lower bounds on the expected estimation error, which certifies statistical optimality under certain regimes, would it be possible for the authors to provide some intuition on the lower bounds for the cases when the true solution is in some Barron spaces? Essentially speaking, are the bounds presented in (13) and (27) statistically optimal? 

5. In the abstract and introduction (line 15-16 and line 125-127), the authors claimed that sharper generalization bounds are derived for solving both the Poisson equation and the Schrödinger equation via the DRM. However, it seems that Theorem 3 in Section 2 only contains results for the Poisson equation?

References: 

[1] Lu, Y., Lu, J. and Wang, M., 2021, July. A priori generalization analysis of the deep Ritz method for solving high dimensional elliptic partial differential equations. In Conference on learning theory (pp. 3196-3241). PMLR.

[2] Lu, Y., Chen, H., Lu, J., Ying, L. and Blanchet, J., 2021. Machine learning for elliptic pdes: Fast rate generalization bound, neural scaling law and minimax optimality. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=mhYUBYNoGz.

[3] Sirignano, J. and Spiliopoulos, K., 2018. DGM: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375, pp.1339-1364.

[4] Han, J., Jentzen, A. and E, W., 2018. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34), pp.8505-8510.

[5] Khoo, Y., Lu, J. and Ying, L., 2021. Solving parametric PDE problems with artificial neural networks. European Journal of Applied Mathematics, 32(3), pp.421-435.

[6] Zang, Y., Bao, G., Ye, X. and Zhou, H., 2020. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics, 411, p.109409.

[7] Chen, Y., Hosseini, B., Owhadi, H. and Stuart, A.M., 2021. Solving and learning nonlinear PDEs with Gaussian processes. Journal of Computational Physics, 447, p.110668.

[8] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A. and Anandkumar, A., 2020. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895.

[9] Lu, L., Jin, P., Pang, G., Zhang, Z. and Karniadakis, G.E., 2021. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3), pp.218-229.

[10] Lu, Y., Blanchet, J. and Ying, L., 2022. Sobolev acceleration and statistical optimality for learning elliptic equations via gradient descent. Advances in Neural Information Processing Systems, 35, pp.33233-33247.
soundness: 3
strengths: 1. This paper is presented in a clear way for readers to follow. Also, a thorough review of related work on variants of Rademacher complexity done by researchers from the statistical learning theory community is included. Proofs of essential lemmas are also presented to ensure the rigorousness of the entire manuscript. 

2. In terms of specific contribution, this paper obtained finer bounds for approximating certain Barron functions via two layer neural networks, which lead to better generalization bounds on solving elliptic PDEs via DRM and PINN under the circumstances when the true solution is in some Barron space compared to existing work [1].
summary: This paper derived new generalization bounds for solving elliptic Partial Differential Equations (PDEs) via two deep learning based methods - the Deep Ritz Method (DRM) and Physics Informed Neural Network (PINN). For DRM, this paper obtained generalization bounds for the Poisson equation with Neuman boundary condition when the solution is in either some Barron space or some Sobolev space. For PINN, this paper considered linear second order elliptic PDEs with Dirichlet boundary condition. By utilizing results from statistical learning theory under the Multi-Task Learning (MTL) framework, the authors also obtained generalization bounds for PINN when the solution is in some Barron space or some Sobolev space. As a side product of the main results, this paper also obtained better rates of approximating functions in certain Barron spaces via two-layer neural networks.
weaknesses: 1. The reviewer's main concern is that some contributions mentioned in this article are not described in a clear way. Specifically, the reviewer thinks that it might be worthwhile for the authors to include a separate paragraph to compare the contributions made in this manuscript and existing work [2]. For specific questions about the main contributions and novelty of this paper compared to [2], the authors may refer to the first four bulletin points in the "Questions" section below. 

2. Regarding presentation of this paper, there are a few grammatic issues that can be potentially resolved. For instance, the sentence from line 40 to 41 can be possibly rephrased as "The Deep Ritz method, on the other hand, incorporates the variational formulation into training the neural networks due to the widespread use of the variational formulation in traditional methods".

3. Given that solving PDEs via machine learning based methods is now a popular field, the authors might consider performing a brief literature review by citing a few important work (other than DRM and PINN) [3-9] in the first part of subsection 1.1 (Related Works) for the sake of completeness. One may refer to the related works section in [2] and [10] as possible examples.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: + How does the analysis differ (substantially) from the work of Weinan E (paper above) and subsequent papers? In particular, when comparing the author's theoretical framework or results to Weinan's section 2 or the CMAME Section 3, can you highlight how your results/conclusions are different and what changes your work makes in terms of the results.

+ Clarify how the author's work is distinct from the CMAME paper, as an 'extension' of the Lu and Weinan E ideas?  Although not identical, can the authorrs provide details on how their work is novel and how it leads to improved or different results over those of Weinan E and/or the CMAME paper.

+ Please compare the framework with the multi-head (multitask) work of Karniadakis :  https://www.semanticscholar.org/paper/L-HYDRA%3A-Multi-Head-Physics-Informed-Neural-Zou-Karniadakis/ec7289f0cb03f0987a0f84391de278f83654bb09   (either argue how they are different or show numerical results).
soundness: 3
strengths: The strength of the paper is that the authors present mathematical analysis of physics-informed neural networks and similar (related) methods. The author's work fits within the current push for developing and growing the mathematical foundations of PINNs and related ideas.
summary: In this paper, the authors derive refined generalization bounds for the Deep Ritz Method (DRM) and Physics-Informed Neural Networks (PINNs), building on the work of Lu et al. (2021c in the manuscript).  The authors derive sharper bounds for the Poisson and Scrodinger's equations, and then present their modified framework within a multi-task learning perspective.
weaknesses: The reviewer did not find any mathematical errors in the work, but was surprised that the authors did not point to the other mathematical works that exist (in the field).  The most well-known on the mathematical side would be Weinan E and collaborators:

https://link.springer.com/article/10.1007/s00365-021-09549-y

Weinan and others (some of which the authors reference part of their work, like: 
https://arxiv.org/abs/2106.07539).

People have since built on Weinan's ideas:  A Deep Double Ritz Method (D
RM) for solving Partial Differential Equations using Neural Networks by Uriiarte et al., CMAME 2023.

Meta-review: The manuscript studies the generalization error of machine learning based solvers for partial differential equations. While the work seems solid, overall the reviewers find the novelty marginal compared with existing literature. After reading the manuscript, the metareviewer agrees with the majority of the reviewer comments. The manuscript is probably better suited for a venue more focused on numerical solutions to PDEs.

Decision: Reject