Title: Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives
Venue: NeurIPS 2023
Accepted: yes
Keywords: VC-dimension

pseudo-dimension

Sobolev space

generalization error

nearly optimal approximation
Abstract: This paper addresses the problem of  nearly optimal Vapnik--Chervonenkis dimension (VC-dimension) and pseudo-dimension estimations of the derivative functions of deep neural networks (DNNs). Two important applications of these estimations include: 1) Establishing a nearly tight approximation result of DNNs in the Sobolev space; 2)  Characterizing the generalization error of machine learning methods with loss functions involving function derivatives. This theoretical investigation fills the gap of learning error estimations for a wide range of physics-informed machine learning models and applications including generative models, solving partial differential equations, operator learning, network compression, distillation, regularization, etc.

Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives

Abstract

This paper addresses the problem of nearly optimal Vapnik-Chervonenkis dimension (VC-dimension) and pseudo-dimension estimations of the derivative functions of deep neural networks (DNNs). Two important applications of these estimations include: 1) Establishing a nearly tight approximation result of DNNs in the Sobolev space; 2) Characterizing the generalization error of machine learning methods with loss functions involving function derivatives. This theoretical investigation fills the gap of learning error estimations for a wide range of physics-informed machine learning models and applications including generative models, solving partial differential equations, operator learning, network compression, distillation, regularization, etc.

1 Introduction

The Sobolev training [8, 40, 46, 45, 22, 42] of deep neural networks (DNNs) has had a significant impact on scientific and engineering fields, including solving partial differential equations [25, 12,

33, 10], operator learning [29, 26], network compression [35], distillation [21, 34], regularization [8], and dynamic programming [15, 48], etc. For example, Sobolev (semi) norms have been applied to penalize function gradients in loss functions [2, 17, 15, 30] to control the Lipschitz constant of DNNs. Moreover, Sobolev norms and equivalent formulas are commonly used to define loss functions in various applications such as dynamic programming [15, 48], solving partial differential equations [25, 12, 33], and distillation [21, 34, 35]. These loss functions enable models to learn DNNs that can approximate the target function with small discrepancies in both magnitude and derivative. For example, when utilizing the Deep Ritz method [12] to solve PDEs such as the following one:

the corresponding loss function can be expressed as:

where θ represents all the parameters in the neural network. Here, Ω denotes the domain (0 , 1) d . Proposition 1 in [27] establishes the equivalence between the loss function E D ( θ ) and ∥ ϕ ( x ; θ ) -u ∗ ( x ) ∥ H 1 ((0 , 1) d ) , where u ∗ ( x ) denotes the exact solution of the PDEs in equation (1), and ∥ f ∥ H 1 ((0 , 1) d ) := ( ∑ 0 ≤| α |≤ 1 ∥ D α f ∥ p L 2 ((0 , 1) d ) ) 1 / 2 . Thus, the Sobolev norm H 1 ((0 , 1) d ) serves as a measure of the loss function, and Sobolev training is employed to solve PDEs within the Deep Ritz method.

Two natural questions that arise are: 1) What is the optimal approximation error of DNNs described by a Sobolev norm? 2) What is the generalization error of the loss function defined by a Sobolev norm? The key step to address these questions is to estimate the optimal Vapnik-Chervonenkis dimension (VC-dimension) and pseudo-dimension [3, 46, 1, 32] of DNNs and their derivatives. Intuitively, these concepts characterize the complexity or richness of a function set and, hence, they can be applied to establish the best possible approximation and generalization power of DNNs.

Definition 1 (VC-dimension [1]) . Let H denote a class of functions from X to { 0 , 1 } . For any non-negative integer m , define the growth function of H as

The Vapnik-Chervonenkis dimension (VC-dimension) of H , denoted by VCdim ( H ) , is the largest m such that Π H ( m ) = 2 m . For a class G of real-valued functions, define VCdim ( G ) := VCdim (sgn( G )) , where sgn( G ) := { sgn( f ) : f ∈ G} and sgn( x ) = 1[ x > 0] .

Definition 2 (pseudo-dimension [32]) . Let F be a class of functions from X to R . The pseudo-dimension of F , denoted by Pdim ( F ) , is the largest integer m for which there exists ( x 1 , x 2 , . . . , x m , y 1 , y 2 , . . . , y m ) ∈ X m × R m such that for any ( b 1 , . . . , b m ) ∈ { 0 , 1 } m there is f ∈ F such that ∀ i : f ( x i ) > y i ⇐⇒ b i = 1 .

The main contribution of this paper is to estimate nearly optimal bounds of the VC-dimension and pseudo-dimension of DNN derivatives. Based on these bounds, we can prove the optimality of our DNN approximation, as measured by Sobolev norms (Theorem 3), and obtain a tighter generalization error of loss functions defined by Sobolev norms. Our results facilitate the understanding of Sobolev training and the performance of DNNs in Sobolev spaces.

Bounds for the VC-dimension and pseudo-dimension of DNNs have been established in [16, 5, 3, 4, 6, 41, 20]. However, these approaches and findings cannot be applied to Sobolev training, as they do not account for the derivatives of DNNs, which represent a key difference between Sobolev training and classical methods. Obtaining such bounds for DNN derivatives is much more difficult due to their complex compositional structures. DNN derivatives consist of a series of interdependent parts that are multiplied together via the chain rule, rendering existing methods for estimating bounds inapplicable. Estimating the VC-dimension and pseudo-dimension of DNN derivatives is the most crucial and challenging problem addressed in this paper. In [11], the VC-dimension and pseudo-dimension of DNN derivatives were analyzed, but the results were suboptimal due to a lack of consideration for the relationships between the multiplied terms in a DNN derivative. As a result, their findings do not provide the optimal approximation of DNNs in Sobolev spaces and can only give a generalization

error that is much larger than the actual error that may arise from Sobolev training. In this paper, we introduce a novel method that investigates these relationships, resulting in a simplified complexity of DNN derivatives. This, in turn, allows us to obtain nearly optimal bounds on their VC-dimension and pseudo-dimension.

The paper is divided into two parts. In the first part, we establish a nearly optimal bound on the VC-dimension of DNN derivatives with the ReLU activation function σ 1 ( x ) := max { 0 , x } :

Theorem 1. For any N,L,d ∈ N + , there exists a constant ¯ C independent with N,L such that

where Φ := { ϕ : ϕ is a σ 1 -NN in R d with width ≤ N and depth ≤ L } , and D i is the weak derivative in the i -th variable.

By utilizing Theorem 1, we prove that our DNN approximation rate for approximating functions in Sobolev spaces W n, ∞ ((0 , 1) d ) using Sobolev norms in W 1 , ∞ ((0 , 1) d ) is nearly optimal. We present our construction of DNNs for this approximation in Theorem 3, and we demonstrate the optimality of such approximation in Theorem 4. Furthermore, we generalize our method to approximate DNNs in Sobolev spaces measured by Sobolev norms W m, ∞ ((0 , 1) d ) for m ≥ 2 . The details of this generalization are presented in Corollaries 1 and 2. The Sobolev spaces, equipped with Sobolev (semi) norms, are defined as follows:

Definition 3 (Sobolev Spaces [13]) . Denote Ω as (0 , 1) d , D as the weak derivative of a single variable function and D α = D α 1 1 D α 2 2 . . . D α d d as the partial derivative where α = [ α 1 , α 2 , . . . , α d ] T and D i is the derivative in the i -th variable. Let n ∈ N and 1 ≤ p ≤ ∞ . Then we define Sobolev spaces

with a norm ∥ f ∥ W n,p (Ω) := ( ∑ 0 ≤| α |≤ n ∥ D α f ∥ p L p (Ω) ) 1 /p , if p < ∞ , and ∥ f ∥ W n, ∞ (Ω) := max 0 ≤| α |≤ n ∥ D α f ∥ L ∞ (Ω) . Furthermore, for f = ( f 1 , f 2 , . . . , f d ) , f ∈ W 1 , ∞ (Ω , R d ) if and only if f i ∈ W 1 , ∞ (Ω) for each i = 1 , 2 , . . . , d and ∥ f ∥ W 1 , ∞ (Ω , R d ) := max i =1 ,...,d {∥ f i ∥ W 1 , ∞ (Ω) } . Definition 4 (Sobolev semi-norm [13]) . Let n ∈ N + and 1 ≤ p ≤ ∞ . Then we define Sobolev semi-norm | f | W n,p (Ω) := ( ∑ | α | = n ∥ D α f ∥ p L p (Ω) ) 1 /p , if p < ∞ , and | f | W n, ∞ (Ω) := max | α | = n ∥ D α f ∥ L ∞ (Ω) . Furthermore, for f ∈ W 1 , ∞ (Ω , R d ) , we define | f | W 1 , ∞ (Ω , R d ) := max i =1 ,...,d {| f i | W 1 , ∞ (Ω) } .

In the second part of our paper, we utilize our previous work on estimating the VC-dimension of DNN derivatives to obtain an upper bound on the pseudo-dimension of DNN derivatives:

Theorem 2. For any N,L,d ∈ N + , there exists a constant ̂ C independent with N,L such that

where D Φ is defined in Theorem 1.

Based on Theorem 2, we can estimate the generalization error of loss functions defined by Sobolev norms, as demonstrated in Theorem 5. Specifically, the error is bounded by O ( NL (log 2 N log 2 L ) 1 / 2 ) with respect to the width N and depth L of DNNs. This bound is significantly smaller than the previously reported bound of O ( NL 5 / 2 (log 2 N log 2 L ) 1 / 2 ) in [11]. We attribute this improvement to our more accurate estimation of the pseudo-dimension of DNN derivatives. Our findings indicate that learning target functions with loss functions defined by Sobolev norms does not require substantially more sample points than those defined by L 2 -norms [14], as their generalization error orders are equivalent with respect to the width N and depth L of DNNs.

Our main contributions are:

We propose a method to achieve nearly optimal estimations of the VC-dimension and pseudodimension of DNN derivatives.

for

By utilizing our estimation of the VC-dimension of DNN derivatives, we demonstrate the optimality of our DNN approximation, as measured by Sobolev norms.

By applying our estimation of the pseudo-dimension of DNN derivatives, we obtain a bound for the generalization error measured by the Sobolev norm. Importantly, our results demonstrate that the degree of generalization error defined by Sobolev norms is equivalent to that defined by L 2 -norms, corresponding to the width N and depth L of DNNs.

2 Preliminaries

Let us summarize all basic notations used in the DNNs as follows:

1 . Matrices are denoted by bold uppercase letters. For example, A ∈ R m × n is a real matrix of size m × n and A ⊺ denotes the transpose of A .

2 . Vectors are denoted by bold lowercase letters. For example, v ∈ R n is a column vector of size n . Furthermore, denote v ( i ) as the i -th elements of v .

3 . For a d -dimensional multi-index α = [ α 1 , α 2 , · · · α d ] ∈ N d , we denote several related notations as follows: ( a ) | α | = | α 1 | + | α 2 | + · · · + | α d | ; ( b ) x α = x α 1 1 x α 2 2 · · · x α d d , x = [ x 1 , x 2 , · · · , x d ] ⊺ ; ( c ) α ! = α 1 ! α 2 ! · · · α d ! .

4 . Let B r, |·| ( x ) ⊂ R d be the closed ball with a center x ∈ R d and a radius r measured by the Euclidean distance. Similarly, B r, ∥·∥ ℓ ∞ ( x ) ⊂ R d be the closed ball with a center x ∈ R d and a radius r measured by the ℓ ∞ -norm.

5 . Assume n ∈ N n + , then f ( n ) = O ( g ( n )) means that there exists positive C independent of n , f, g such that f ( n ) ≤ Cg ( n ) when all entries of n go to + ∞ .

6 . Define σ 1 ( x ) := σ ( x ) = max { 0 , x } and σ 2 := σ 2 ( x ) . We call the neural networks with activation function σ t with t ≤ i as σ i neural networks ( σ i -NNs). With the abuse of notations, we

7 . Define L, N ∈ N + , N 0 = d and N L +1 = 1 , N i ∈ N + for i = 1 , 2 , . . . , L , then a σ i -NN ϕ with the width N and depth L can be described as follows:

where W i ∈ R N i × N i -1 and b i ∈ R N i are the weight matrix and the bias vector in the i -th linear transform in ϕ , respectively, i.e., h i := W i ˜ h i -1 + b i , for i = 1 , . . . , L +1 and ˜ h i = σ i ( h i ) , for i = 1 , . . . , L. In this paper, an DNN with the width N and depth L , means (a) The maximum width of this DNN for all hidden layers less than or equal to N . (b) The number of hidden layers of this DNN less than or equal to L .

3 Nearly Optimal Approximation Results of DNNs in Sobolev Spaces Measured by Sobolev Norms

3.1 Approximation of functions in W n, ∞ with W 1 , ∞ norm by ReLU neural networks

In this subsection, we construct deep neural networks (DNNs) with a width of O ( N log N ) and a depth of O ( L log L ) to approximate functions in the Sobolev space W n, ∞ , as measured by Sobolev norms in W 1 , ∞ . The approximation rate achieved by these networks is O ( N -2( n -1) /d L -2( n -1) /d ) .

Theorem 3. For any f ∈ W n, ∞ ((0 , 1) d ) with n ≥ 2 and ∥ f ∥ W n, ∞ ((0 , 1) d ) ≤ 1 , any N,L ∈ N + , there is a σ 1 -NN ϕ with the width (34 + d )2 d n d +1 ( N +1)log 2 (8 N ) and depth 56 d 2 n 2 ( L + 1) log 2 (4 L ) such that

where C 9 is the constant independent with N,L .

The proof of Theorem 3 can be outlined in five parts, and the complete proof is provided in Appendix 7.2:

(i) : First of all, define a sequence of subsets of Ω :

Then we define a partition of unity { g m } m ∈{ 1 , 2 } d on (0 , 1) d with supp g m ∩ (0 , 1) d ⊂ Ω m for each m ∈ { 1 , 2 } d :

Definition 6. Given K,d ∈ N + , we define

for i ∈ Z . For any m = ( m 1 , m 2 , . . . , m d ) ∈ { 1 , 2 } d , define g m ( x ) = ∏ d j =1 g m j ( x j ) , x = ( x 1 , x 2 , . . . , x d ) .

91

92

0

4K

1

2K

1

4K

3

K

1

4K

7

1

(ii) : Then we use the following proposition to approximate { g m } m ∈{ 1 , 2 } d by σ 1 -NNs and construct a sequence of σ 1 -NNs { ϕ m } m ∈{ 1 , 2 } d :

Proposition 1. Given any N,L,n ∈ N + for K = ⌊ N 1 /d ⌋ 2 ⌊ L 2 /d ⌋ , then for any m = ( m 1 , m 2 , . . . , m d ) ∈ { 1 , 2 } d , there is a σ 1 -NN with the width smaller than (9+ d )( N +1)+ d -1 and depth smaller than 15 d ( d -1) nL such as ∥ ϕ m ( x ) -g m ( x ) ∥ W 1 , ∞ ((0 , 1) d ) ≤ 50 d 5 2 ( N +1) -4 dnL .

The proof of Proposition 1 is presented in Appendix 7.2.1.

(iii) : For each Ω m ⊂ [0 , 1] d , where m ∈ { 1 , 2 } d , we find a function f K, m satisfying

where C 1 is a constant independent of K . Moreover, each f K, m can be expressed as f K, m = ∑ | α |≤ n -1 g f, α , m ( x ) x α , where g f, α , m ( x ) is a piecewise constant function on Ω m . The proof of this result is based on the Bramble-Hilbert Lemma [7, Lemma 4.3.8], and the details are provided in Appendix 7.2.2.

(iv) : The fourth step involves approximating f K, m using neural networks ψ m , following the approach outlined in [28]. This method is suitable for our work because g f, α , m ( x ) is a piecewise constant function on Ω m , and the weak derivative of g f, α , m ( x ) on Ω m is zero. This property allows for the use of the L ∞ norm approximation method presented in [28]. Thus, we obtain a neural network ψ m with width O ( N log N ) and depth O ( L log L ) such that

where C 5 is a constant independent of N and L .

By combining (iii) and (iv) and setting K = ⌊ N 1 /d ⌋ 2 ⌊ L 2 /d ⌋ , we obtain that for each m ∈ { 1 , 2 } d , there exists a neural network ψ m with width O ( N log N ) and depth O ( L log L ) such that

where C 6 is a constant independent of N and L . Further details are provided in Appendix 7.2.3.

(v) : The final step is to combine the sequences { ϕ m } m ∈{ 1 , 2 } d and { ψ m } m ∈{ 1 , 2 } d to construct a network that can approximate f over the entire space [0 , 1] d . We define the sequence { ϕ m } m ∈{ 1 , 2 } d because ψ m may not accurately approximate f on [0 , 1] d \ Ω m . The purpose of ϕ m is to remove this portion of the domain and allow other networks to approximate f on [0 , 1] d \ Ω m . Further details on this step are provided in Appendix 7.2.4.

While recent works [28, 23, 39, 18, 31, 9, 19] have studied the approximation of smooth functions or functions in Sobolev spaces by DNNs measured in the norm of L p (Ω) or W s,p (Ω) , they typically present results that are not optimal or are measured in L p -norms. For example, in [28], they applies Taylor's expansion to approximate smooth functions but cannot be applied directly in Sobolev spaces. In [39], they improve on this by using the Bramble-Hilbert Lemma to approximate functions in Sobolev spaces, but their error is still measured in L p -norms. In [18], the authors show that there exists a ReLU neural network that can approximate f ∈ W 1 ,p (Ω) , but their approximation rate is not optimal and is the same as that in traditional methods such as the finite element theory. Our work provides a superior approximation rate. Later, a rigorous proof of optimality of Theorem 3 is discussed in Appendix 7.2.4 and Subsection 3.3.

3.2 Approximation of functions in W n, ∞ measured by W m, ∞ norm with m> 1 by neural networks (sketches of the proofs of the Corollaries 1 and 2)

In this subsection, we utilize neural networks to approximate functions in W n, ∞ measured by W m, ∞ , where m > 1 . The proof strategy is similar to the approximation measured in the norm of W 1 , ∞ . However, we cannot rely on ReLU neural networks alone to achieve this goal, as ReLU neural networks are piece-wise linear functions that do not belong to W m, ∞ with m > 1 . Note that the Bramble-Hilbert Lemma is still applicable in higher-order approximation. Therefore, we need to approximate the function f K, m = ∑ | α |≤ n -1 g f, α , m ( x ) x α within each domain Ω m using DNNs, where g f, α , m ( x ) represents a piece-wise constant function within Ω m . Consequently, ReLU-based DNNs can effectively approximate g f, α , m ( x ) within Ω m when measured by higher-order Sobolev spaces, since both the higher-order derivatives of g f, α , m ( x ) and ReLU-based DNNs are zero. The parts of ReLU-based DNNs that do not have high-order derivatives appear in the domain Ω \ Ω m , and we will use the partition of unity to ensure that this part disappears in the final presentation, this is the reason why it is still acceptable to have ReLU activations appearing in the network. However, when it comes to approximating x α for | α | > 1 based on high-order Sobolev norms, ReLU-based DNNs fail to provide accurate results. Therefore, we require DNNs that utilize the square of ReLU activation in this specific scenario. This is how we construct our approach and the reason why we employ both ReLU and the square of ReLU activation functions.

Instead, we examine the use of σ 2 neural networks for approximating functions measured in the norm of W 2 , ∞ . As per Corollary 1, a neural network with O ( N log N ) width and O ( L log L ) depth can achieve a nonasymptotic approximation rate of O ( N -2( n -2) /d L -2( n -2) /d ) with respect to the W 2 , ∞ ((0 , 1) d ) norm. Moreover, our method can be extended to approximations measured in the norm of W m, ∞ with m > 2 , as shown in Corollary 2. The proof strategy is similar to that used in Subsection 3.1, except that we need to construct a smoother partition of unity rather than { g m } m ∈{ 1 , 2 } d . The corollaries are presented below, and further details are provided in Appendix 7.3.

Corollary 1. For any f ∈ W n, ∞ ((0 , 1) d ) with ∥ f ∥ W n, ∞ ((0 , 1) d ) ≤ 1 , any N,L ∈ N + with NL + 2 ⌊ log 2 N ⌋ ≥ max { d, n } and L ≥ ⌈ log 2 N ⌉ , there is a σ 2 -NN γ ( x ) with the width 2 d +6 n d +1 ( N + d ) log 2 (8 N ) and depth 15 n 2 ( L +2)log 2 (4 L ) such that

where C 10 is the constant independent with N,L .

Corollary 2. For any f ∈ W n, ∞ ((0 , 1) d ) with ∥ f ∥ W n, ∞ ((0 , 1) d ) ≤ 1 , any N,L,m ∈ N + with NL +2 ⌊ log 2 N ⌋ ≥ max { d, n } and L ≥ ⌈ log 2 N ⌉ , there is a σ 2 -NN φ ( x ) with the width O ( N log N ) and depth O ( L log L ) such that

where C 11 is the constant independent with N,L .

3.3 Optimality of Theorem 3 via estimation of VC-dimension of DNN derivatives (Theorem 1)

In this section, we demonstrate that the approximation rate presented in Theorem 3 is nearly asymptotically optimal:

Theorem 4. Given any ρ, C 1 , C 2 , C 3 , J 0 > 0 and n, d ∈ N + , there exist N,L ∈ N with NL ≥ J 0 and f with ∥ f ∥ W n, ∞ (((0 , 1) d )) ≤ 1 , satisfying for any σ 1 -NN ϕ with the width smaller than C 1 N log N and depth smaller than C 2 L log L , we have

In other words, the approximation rate of O ( N -2( n -1) /d -ρ K -2( n -1) /d -ρ ) cannot be achieved asymptotically when ReLU σ 1 -NNs with width O ( N log N ) and depth O ( L log L ) to approximate functions in F n,d := { f ∈ W n, ∞ ((0 , 1) d ) : ∥ f ∥ W n, ∞ ((0 , 1) d ) ≤ 1 } . The proof of Theorem 4 is based on the estimation of the VC-dimension of DNN derivatives, which is provided in Theorem 1.

Theorem 1 plays a crucial role in our proof of Theorem 4, which is established through a proof by contradiction following the approach outlined in Ref. [28]. Further details on the proof can be found in Appendix 7.4. The main idea behind the proof is that Theorem 1 characterizes the complexity of DNN derivatives, which in turn limits the ability of DNNs to approximate functions in Sobolev spaces.

In this paper, we focus on the optimality of approximation rate with respect to width N and depth L of DNNs. The dimensionality d is not the focus that we consider in our research. Addressing the question about mitigating the exponential dependence of width on dimensionality, we have observed that this arises from the utilization of methods like Taylor's expansion or average Taylor polynomials in our approximation techniques. It remains an open question for future research to explore alternative approaches to address the challenge of getting the dependence of d in the lower bounds.

4 Generalization Analysis in Sobolev Spaces via Estimation of Pseudo-dimension of DNN Derivatives (Theorem 2)

In a typical supervised learning algorithm, the objective is to learn a high-dimensional target function f ( x ) defined on (0 , 1) d with ∥ f ∥ W n, ∞ ((0 , 1) d ) ≤ 1 from a finite set of data samples { ( x i , f ( x i )) } i M =1 . When training a DNN, we aim to identify a DNN ϕ ( x ; θ S ) that approximates f ( x ) based on random data samples { ( x i , f ( x i )) } i M =1 . We assume that { x i } i M =1 is an i.i.d. sequence of random variables uniformly distributed on (0 , 1) d in this section. Denote

The overall inference error is E R D ( θ S ) , which can be divided into two parts:

where the last inequality is due to E R S ( θ S ) ≤ E R S ( θ D ) by the definition of θ S .

Due to Theorem 3, we know that the approximation error R D ( θ D ) is a O ( N -4( n -1) /d L -4( n -1) /d ) term since ∥ f ( x ) -ϕ ( x ) ∥ H 1 ((0 , 1) d ) ≤ ∥ f ( x ) -ϕ ( x ) ∥ W 1 , ∞ ((0 , 1) d ) . In this section, we bound generalization error in the H 1 ((0 , 1) d ) sense:

Theorem 5. For any N,L,d,B,C 1 , C 2 , if ϕ ( x ; θ D ) , ϕ ( x ; θ S ) ∈ ˜ Φ , we will have that there are constants C 5 = C 5 ( B,d,C 1 , C 2 ) and J = J ( d, N, L, C 1 , C 2 ) such that for any M ≥ J , we have

where ˜ Φ := { ϕ : ϕ with the width ≤ C 1 N log N and depth ≤ C 2 L log L, ∥ ϕ ∥ W 1 , ∞ ((0 , 1) d ) ≤ B } , and R S , R D , θ S , θ D are defined in Eqs. (10,11).

The proof of Theorem 5 is based on the works of [3, 11, 26]. We begin by bounding the generalization error using the Rademacher Complexity and then bound the Rademacher Complexity by the uniform covering number. We further bound the uniform covering number by the pseudo-dimension. Finally, we estimate the pseudo-dimension by Theorem 2. The proof of Theorem 5 is presented in Appendix 7.5

Theorem 2 helps to control the degree of the generalization error with respect to N and L in Theorem 5. In [11], the generalization error is bounded by O ( NL 5 2 ) . In [24], the authors estimate the covering number using the Lipschitz condition of DNNs instead of the pseudo-dimension, leading to a generalization error that is exponentially dependent on the depth of the DNNs. Our result is much better than them due to the optimal estimation of pseudo-dimension of DNN derivatives (Theorem 2).

5 Proof Sketches for Theorems 1 and 2

As Theorems 1 and 2 address the estimation of VC-dimension and pseudo-dimension of DNN derivatives, which is the main contribution of this paper, we provide the proofs for these theorems in this section. The distinction in our approach compared to that of [4] lies in the fact that the application of the chain rule requires the consideration of correlations among distinct segments of the deep neural networks, as opposed to treating them as independent components multiplied together.

In the proof of Theorem 1, we use the following lemmas:

Lemma 1 ([4, Lemma 17],[3, Theorem 8.3]) . Suppose W ≤ M and let P 1 , . . . , P M be polynomials of degree at most D in W variables. Define K := ∣ ∣ { (sgn( P 1 ( a )) , . . . , sgn( P M ( a ))) : a ∈ R W } ∣ ∣ , then we have K ≤ 2(2 eMD/W ) W .

Lemma 2 ([4, Lemma 18]) . Suppose that 2 m ≤ 2 t ( mr/w ) w for some r ≥ 16 and m ≥ w ≥ t ≥ 0 . Then, m ≤ t + w log 2 (2 r log 2 r ) .

As the proof of Theorem 1 represents the most critical and challenging question in our work, we present a sketch of it below.

Proof Sketch of Theorem 1. An element in Φ can be represented as ϕ = W L +1 σ 1 ( W L σ 1 ( . . . σ 1 ( W 1 x + b 1 ) . . . ) + b L ) + b L +1 . Therefore, an element in D Φ can be represented as

where W i ∈ R N i × N i -1 ( ( W ) i is i -th column of W ) and b i ∈ R N i are the weight matrix and the bias vector in the i -th linear transform in ϕ , and σ 0 ( x ) = sgn( x ) = 1[ x > 0] , which is the derivative of the ReLU function and σ 0 ( x ) = diag( σ 0 ( x i )) .

Let x ∈ R d be an input and θ ∈ R W be a parameter vector in ψ . We denote the output of ψ with input x and parameter vector θ as f ( x , θ ) . For fixed x 1 , x 2 , . . . , x m in R d , we aim to bound

The proof is inspired by [4, Theorem 7]. For any partition S = { P 1 , P 2 , . . . , P T } of the parameter domain R W , we have K ≤ ∑ T i =1 |{ (sgn( f ( x 1 , θ )) , . . . , sgn( f ( x m , θ ))) : θ ∈ P i }| . We choose the partition such that within each region P i , the functions f ( x j , · ) are all fixed polynomials of bounded degree. This allows us to bound each term in the sum using Lemma 1.

The partition of R W is constructed layer by layer through successive refinements denoted by S 0 , S 1 , . . . , S L . These refinements possess the following properties:

1 . We have |S 0 | = 1 , and for each n = 1 , . . . , L , we have |S n | |S n -1 | ≤ 2 ( 2 emnN k ∑ n i =1 W i ) ∑ n i =1 W i .

2 . For each n = 0 , . . . , L -1 , each element S of S n , when θ varies in S , the output of each term in F n is a fixed polynomial function in ∑ n i =1 W i variables of θ , with a total degree no more than n +1 .

3 . For each element S of S L , when θ varies in S , the h -th term in F L for h ∈ { 1 , 2 , . . . , L +1 } is a fixed polynomial function in W h variables of θ , with a total degree no more than 1 . The sequence of sets of functions { F j } L j =0 with respect to parameters θ ∈ R W is defined as:

The details of the refinements required to obtain the partitions are presented in the appendix. By leveraging Property 3 and Lemma 1, we can deduce that K ≤ 2 L +1 ( 2 em ( L +2)( L +1) N 2 U ) U where U = O ( N 2 L 2 ) , N is the width of the network, and the last inequality is due to weighted AM-GM. For the definition of the VC-dimension, we have 2 VCdim ( D Φ) ≤ 2 L +1 ( e VCdim ( D Φ)( L +1)( L +2) N U ) U . Due to Lemma 2, we obtain that VCdim ( D Φ) ≤ L +1+ U log 2 [2( L +1)( L +2)log 2 ( L +1)( L +2)] = O ( N 2 L 2 log 2 L log 2 N ) since U = O ( N 2 L 2 ) .

Note that the VC-dimension estimation achieved in Theorem 1 is nearly optimal, as demonstrated in Corollary 3 combined with the upper bound in Theorem 1. If the polynomial degree in the VC-dimension bound as a function of N and L were any smaller, it would contradict Theorem 3, which is based on our proof of Theorem 4.

where D Φ is defined in Theorem 1.

We discuss the proof of Corollary 3 at the end of Section 7.4. Next we now present the proof for Theorem 2.

The proof of Theorem 2 can be found in the appendix. The key idea behind the proof is to reformulate the pseudo-dimension of DNN derivatives as the VC-dimension of a larger set. Based on the definition of pseudo-dimensions, it follows that Pdim ( D Φ) ≥ VCdim ( D Φ) . Consequently, we can derive the following corollary, utilizing Corollary 3, to demonstrate the near optimality of the pseudo-dimension estimate presented in Theorem 2:

Corollary 4. For any d ∈ N + , C, J 0 , ε > 0 , there exists N,L ∈ N with NL ≥ J 0 such that

where D Φ is defined in Theorem 1.

6 Conclusions and Discussions

In this paper, we establish nearly optimal bounds for the VC-dimension and pseudo-dimension of DNNderivatives. Based on these bounds, two contributions to Sobolev training [8, 40, 46] are made in

this paper. Firstly, we show that the optimal approximation rate of DNNs with a width of O ( N log N ) and a depth of O ( L log L ) is O ( N -2( n -1) /d L -2( n -1) /d ) in Sobolev spaces. This demonstrates the ability of DNNs to learn target functions well in Sobolev training. Secondly, we find that the degree of the pseudo-dimension of DNN derivatives is the same as that for DNNs corresponding to the width N and depth L of DNNs. This result suggests that despite the apparent complexity of DNN derivatives, the degree of generalization error of loss functions containing derivatives of DNNs is equivalent to that without derivatives, corresponding to the width N and depth L of DNNs. As a result, we do not need to use a significantly larger number of sample points to learn the target function in Sobolev training compared to regular training.

The estimations of the VC-dimension and pseudo-dimension of DNN derivatives have broad applications in deep learning research. For example, in classification tasks, the VC-dimension characterizes the uniform convergence of misclassification frequencies to probabilities and asymptotically determines the sample complexity of PAC learning [4, 44, 6]. These applications can be explored in the further work. Our focus in this paper is on the Sobolev training with loss functions containing first-order derivatives, and we also obtain the approximation rate of σ 2 -NNs described by higher-order Sobolev norms (Corollaries 1 and 2). The optimality of these results and the generalization error of Sobolev training with loss functions containing higher-order derivatives of DNNs remain open problems, as estimating the VC-dimension and pseudo-dimension of higher-order derivatives of σ 2 -NNs requires further investigation.

Acknowledgments and Disclosure of Funding

This work was done during Y.Y.'s visit under the supervision of Prof. H.Y., in the Department of Mathematics, University of Maryland College Park. The work of H. Y. was partially supported by the US National Science Foundation under award DMS-2244988, DMS-2206333, and the Office of Naval Research Award N00014-23-1-2007. The work of Y.X. was supported by the Project of Hetao Shenzhen-HKUST Innovation Cooperation Zone HZQB-KCZYB-2020083.

References

Y. Abu-Mostafa. The Vapnik-Chervonenkis dimension: Information versus complexity in learning. Neural Computation , 1(3):312-317, 1989.

J. Adler and S. Lunz. Banach wasserstein Gan. Advances in neural information processing systems , 31, 2018.

M. Anthony, P. Bartlett, et al. Neural network learning: Theoretical foundations , volume 9. cambridge university press Cambridge, 1999.

P. Bartlett, N. Harvey, C. Liaw, and A. Mehrabian. Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research , 20(1):2285-2301, 2019.

P. Bartlett, V. Maiorov, and R. Meir. Almost linear VC dimension bounds for piecewise polynomial networks. Advances in neural information processing systems , 11, 1998.

A. Blumer, A. Ehrenfeucht, D. Haussler, and M. Warmuth. Learnability and the VapnikChervonenkis dimension. Journal of the ACM (JACM) , 36(4):929-965, 1989.

S. Brenner, L. Scott, and L. Scott. The mathematical theory of finite element methods , volume 3. Springer, 2008.

W. Czarnecki, S. Osindero, M. Jaderberg, G. Swirszcz, and R. Pascanu. Sobolev training for neural networks. Advances in neural information processing systems , 30, 2017.

T. De Ryck, S. Lanthaler, and S. Mishra. On the approximation of functions by tanh neural networks. Neural Networks , 143:732-750, 2021.

T. De Ryck and S. Mishra. Error analysis for physics-informed neural networks (PINNs) approximating Kolmogorov PDEs. Advances in Computational Mathematics , 48(6):1-40, 2022.

C. Duan, Y. Jiao, Y. Lai, X. Lu, and Z. Yang. Convergence rate analysis for Deep Ritz method. arXiv preprint arXiv:2103.13330 , 2021.

W. E, J. Han, and A. Jentzen. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in Mathematics and Statistics , 5(4):349-380, 2017.

L. Evans. Partial differential equations , volume 19. American Mathematical Society, 2022.

M. Farrell, T. Liang, and S. Misra. Deep neural networks for estimation and inference. Econometrica , 89(1):181-213, 2021.

C. Finlay, J. Calder, B. Abbasi, and A. Oberman. Lipschitz regularized deep neural networks generalize and are adversarially robust. arXiv preprint arXiv:1808.09540 , 2018.

P. Goldberg and M. Jerrum. Bounding the Vapnik-Chervonenkis dimension of concept classes parameterized by real numbers. In Proceedings of the sixth annual conference on Computational learning theory , pages 361-369, 1993.

S. Gu and L. Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5068 , 2014.

I. Gühring, G. Kutyniok, and P. Petersen. Error bounds for approximations with deep ReLU neural networks in W s,p norms. Analysis and Applications , 18(05):803-859, 2020.

I. Gühring and M. Raslan. Approximation rates for neural networks with encodable weights in smoothness spaces. Neural Networks , 134:107-130, 2021.

N. Harvey, C. Liaw, and A. Mehrabian. Nearly-tight vc-dimension bounds for piecewise linear neural networks. In Conference on learning theory , pages 1064-1068. PMLR, 2017.

G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

J. Hoffman, D. Roberts, and S. Yaida. Robust learning with jacobian regularization. arXiv preprint arXiv:1908.02729 , 2019.

S. Hon and H. Yang. Simultaneous neural network approximation for smooth functions. Neural Networks , 154:152-164, 2022.

Y. Jiao, Y. Lai, Y . Lo, Y . Wang, and Y . Yang. Error analysis of Deep Ritz methods for elliptic equations. arXiv preprint arXiv:2107.14478 , 2021.

I. Lagaris, A. Likas, and D. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks , 9(5):987-1000, 1998.

H. Liu, H. Yang, M. Chen, T. Zhao, and W. Liao. Deep nonparametric estimation of operators between infinite dimensional spaces. arXiv preprint arXiv:2201.00217 , 2022.

J. Lu, Y. Lu, and M. Wang. A priori generalization analysis of the Deep Ritz method for solving high dimensional elliptic partial differential equations. In Conference on Learning Theory , pages 3196-3241. PMLR, 2021.

J. Lu, Z. Shen, H. Yang, and S. Zhang. Deep network approximation for smooth functions. SIAM Journal on Mathematical Analysis , 53(5):5465-5506, 2021.

L. Lu, P. Jin, G. Pang, Z. Zhang, and G. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature machine intelligence , 3(3):218-229, 2021.

Y. Mroueh, C. Li, T. Sercu, A. Raj, and Y. Cheng. Sobolev Gan. In International Conference on Learning Representations . International Conference on Learning Representations, ICLR, 2018.

J. Müller and M. Zeinhofer. Error estimates for the Deep Ritz method with boundary penalty. In Mathematical and Scientific Machine Learning , pages 215-230. PMLR, 2022.

D. Pollard. Empirical processes: theory and applications. Ims, 1990.

M. Raissi, P. Perdikaris, and G. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics , 378:686-707, 2019.

A. A Rusu, S. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295 , 2015.

B. Sau and V. Balasubramanian. Deep model compression: Distilling knowledge from noisy teachers. arXiv preprint arXiv:1610.09650 , 2016.

Z. Shen, H. Yang, and S. Zhang. Nonlinear approximation via compositions. Neural Networks , 119:74-84, 2019.

Z. Shen, H. Yang, and S. Zhang. Deep network approximation characterized by number of neurons. Communications in Computational Physics , 28(5), 2020.

Z. Shen, H. Yang, and S. Zhang. Optimal approximation rate of ReLU networks in terms of width and depth. Journal de Mathématiques Pures et Appliquées , 157:101-135, 2022.

J. Siegel. Optimal approximation rates for deep ReLU neural networks on Sobolev spaces. arXiv preprint arXiv:2211.14400 , 2022.

H. Son, J. Jang, W. Han, and H. Hwang. Sobolev training for the neural network solutions of PDEs. arXiv preprint arXiv:2101.08932 , 2021.

E. Sontag et al. Vc dimension of neural networks. NATO ASI Series F Computer and Systems Sciences , 168:69-96, 1998.

S. Srinivas and F. Fleuret. Knowledge transfer with jacobian matching. In International Conference on Machine Learning , pages 4723-4731. PMLR, 2018.

E. Stein. Singular integrals and differentiability properties of functions , volume 2. Princeton university press, 1970.

V. Vapnik and A. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Measures of complexity: festschrift for alexey chervonenkis , pages 11-30, 2015.

N. Vlassis, R. Ma, and W. Sun. Geometric deep learning for computational mechanics part i: Anisotropic hyperelasticity. Computer Methods in Applied Mechanics and Engineering , 371:113299, 2020.

N. Vlassis and W. Sun. Sobolev training of thermodynamic-informed neural networks for interpretable elasto-plasticity models with level set hardening. Computer Methods in Applied Mechanics and Engineering , 377:113695, 2021.

M. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

P. Werbos. Approximate dynamic programming for real-time control and neural modeling. Handbook of intelligent control , 1992.

7 Supplementary Material

7.1 Proofs of Theorems 1 and 2.

Proof of Theorem 1. An element in Φ can be represented as ϕ = W L +1 σ 1 ( W L σ 1 ( . . . σ 1 ( W 1 x + b 1 ) . . . ) + b L ) + b L +1 . Therefore, an element in D Φ can be represented as

where W i ∈ R N i × N i -1 ( ( W ) i is i -th column of W ) and b i ∈ R N i are the weight matrix and the bias vector in the i -th linear transform in ϕ , and σ 0 ( x ) = sgn( x ) = 1[ x > 0] , which is the derivative of the ReLU function and σ 0 ( x ) = diag( σ 0 ( x i )) . Denote W i as the number of parameters in W i , b i , i.e., W i = N i N i -1 + N i .

Let x ∈ R d be an input and θ ∈ R W be a parameter vector in ψ . We denote the output of ψ with input x and parameter vector θ as f ( x , θ ) . For fixed x 1 , x 2 , . . . , x m in R d , we aim to bound

The proof is inspired by [4, Theorem 7]. For any partition S = { P 1 , P 2 , . . . , P T } of the parameter domain R W , we have K ≤ ∑ T i =1 |{ (sgn( f ( x 1 , θ )) , . . . , sgn( f ( x m , θ ))) : θ ∈ P i }| . We choose the partition such that within each region P i , the functions f ( x j , · ) are all fixed polynomials of bounded degree. This allows us to bound each term in the sum using Lemma 1.

We define a sequence of sets of functions { F j } L j =0 with respect to parameters θ ∈ R W :

The partition of R W is constructed layer by layer through successive refinements denoted by S 0 , S 1 , . . . , S L . These refinements possess the following properties:

1 . We have |S 0 | = 1 , and for each n = 1 , . . . , L , we have |S n | |S n -1 | ≤ 2 ( 2 emnN k ∑ n i =1 W i ) ∑ n i =1 W i .

2 . For each n = 0 , . . . , L -1 , each element S of S n , when θ varies in S , the output of each term in F n is a fixed polynomial function in ∑ n i =1 W i variables of θ , with a total degree no more than n +1 .

3 . For each element S of S L , when θ varies in S , the h -th term in F L for h ∈ { 1 , 2 , . . . , L +1 } is a fixed polynomial function in W h variables of θ , with a total degree no more than 1 .

We define S 0 = { R W } , which satisfies properties 1,2 above, since W 1 x j + b 1 and ( W 1 ) i are affine functions of W 1 , b 1 .

To define S n , we use the last term of F n -1 as inputs for the last two terms in F n . Assuming that S 0 , S 1 , . . . , S n -1 have already been defined, we observe that the last two terms are new additions to F n when comparing it to F n -1 . Therefore, all elements in F n except the last two are fixed polynomial functions in W n variables of θ , with a total degree no greater than n when θ varies in S ∈ S n . This is because S n is a finer partition than S n -1 .

We denote p x j ,n -1 ,S,k ( θ ) as the output of the k -th node in the last term of F n -1 in response to x j when θ ∈ S . The collection of polynomials

can attain at most 2 ( 2 emnN n ∑ n i =1 W i ) ∑ n i =1 W i distinct sign patterns when θ ∈ S due to Lemma 1 for n sufficiently large m . Therefore, we can divide S into 2 ( 2 emnN n ∑ n i =1 W i ) ∑ i =1 W i parts, each having the

property that p x j ,n -1 ,S,k ( θ ) does not change sign within the subregion. By performing this for all S ∈ S n -1 , we obtain the desired partition S n . This division ensures that the required property 1 is satisfied.

Additionally, since the input to the last two terms in F n is p x j ,n -1 ,S,k ( θ ) , and we have shown that the sign of this input will not change in each region of S n , it follows that the output of the last two terms in F n is also a polynomial without breakpoints in each element of S n . Therefore, the required property 2 is satisfied.

In the context of DNNs, the last layer is characterized by all terms containing the activation function σ 0 . Consequently, for any element S of the partition S L , when the vector of parameters θ varies within S , the h -th term in F L for h ∈ { 1 , 2 , . . . , L +1 } can be expressed as a polynomial function of at most degree 1 , which depends on at most W h variables of θ . Hence, the required property 3 is satisfied.

Due to property 3, we multiply all the terms in F L and obtain a term in D Φ . Hence, the output of each term in D Φ is a polynomial function in ∑ L +1 i =1 W i variables of θ ∈ S ∈ S L , of total degree no more than L +1 . Therefore, for each S ∈ S L we have |{ (sgn( f ( x 1 , θ )) , . . . , sgn( f ( x m , θ ))) : θ ∈ S }| ≤

where U := ∑ L +1 n =1 ∑ n i =1 W i = O ( N 2 L 2 ) , N is the width of the network, and the last inequality is due to weighted AM-GM. For the definition of the VC-dimension, we have

Due to Lemma 2, we obtain that

Proof of Theorem 2. Denote D Φ N := { η ( x , y ) : η ( x , y ) = ψ ( x ) -y, ψ ∈ D Φ , ( x , y ) ∈ R d +1 } . Based on the definition of VC-dimension and pseudo-dimension, we have that

For the VCdim ( D Φ N ) , it can be bounded by O ( N 2 L 2 log 2 L log 2 N ) . The proof is similar to that for the estimate of VCdim ( D Φ) as given in Theorem 1.

We establish that Pdim ( D Φ) ≤ VCdim ( D Φ N ) , where Φ N represents DNNs with N + 1 width and L +1 depth. This implies that Pdim ( D Φ) is upper bounded by ¯ C ( N +1) 2 ( L +1) 2 log 2 ( L + 1) log 2 ( N +1) ≤ 64 ¯ CN 2 L 2 log 2 L log 2 N . Therefore, we conclude that 64 ¯ C ≥ ˆ C .

7.2 Proof of Theorem 3

7.2.1 Propositions of Sobolev spaces and ReLU neural networks

The following two lemmas estimate the Sobolev norms and Sobolev semi-norms for the composition and product, which will be used in later proof.

Lemma 3 ([18, Corollary B.5]) . Let d, m ∈ N + and Ω 1 ⊂ R d and Ω 2 ⊂ R m both be open, bounded, and convex. Then for f ∈ W 1 , ∞ (Ω 1 , R m ) and g ∈ W 1 , ∞ (Ω 2 ) with ran f ⊂ Ω 2 , we have

Lemma 4 ([18, Corollary B.6]) . Let d ∈ N and Ω ⊂ R d . Then for f, g ∈ W 1 , ∞ (Ω) , we have

Then we collect and establish some propositions for ReLU neural networks.

Proposition 2 ([28, Proposition 4.3]) . Given any N,L ∈ N + and δ ∈ ( 0 , 1 3 K ] for K = ⌊ N 1 /d ⌋ 2 ⌊ L 2 /d ⌋ , there exists a σ 1 -NN ϕ with the width 4 N +5 and depth 4 L +4 such that

Proposition 3. [28, Proposition 4.4] Given any N,L,s ∈ N + and ξ i ∈ [0 , 1] for i = 0 , 1 , . . . N 2 L 2 -1 , there exists a σ 1 -NN ϕ with the width 16 s ( N +1)log 2 (8 N ) and depth (5 L +2)log 2 (4 L ) such that

2. 0 ≤ ϕ ( x ) ≤ 1 , x ∈ R .

Proposition 4. For any N,L ∈ N + and a > 0 , there is a σ 1 -NN ϕ with the width 15 N and depth 2 L such that ∥ ϕ ∥ W 1 , ∞ (( -a,a ) 2 ) ≤ 12 a 2 and

Proof. We first need to construct a neural network to approximate x 2 on ( -1 , 1) , and the idea is similar with [23, Lemma 3.2] and [28, Lemma 5.1]. The reason we do not use [23, Lemma 3.4] and [28, Lemma 4.2] directly is that constructing ϕ ( x, y ) by translating a neural network in W 1 , ∞ [0 , 1] will lose the proposition of ϕ (0 .y ) = 0 . Here we need to define teeth functions T i on ˜ x ∈ [ -1 , 1] :

and

Define

According to [23, Lemma 3.2] and [28, Lemma 5.1], we know ψ is a neural network with the width 5 N and depth 2 L such that ∥ ˜ ψ ( ˜ x ) ∥ W 1 , ∞ (( -1 , 1)) ≤ 2 , ∥ ˜ ψ ( ˜ x ) -˜ x 2 ∥ W 1 , ∞ (( -1 , 1)) ≤ N -L and ψ (0) = 0 .

By setting x = a ˜ x ∈ ( -a, a ) for ˜ x ∈ ( -1 , 1) , we define

Furthermore,

Note that x 2 = a 2 ( x a ) 2 , we have

and ψ (0) = 0 , which will be used to prove Eq. (27).

Then we can construct ϕ ( x, y ) as

where ϕ ( x ) is a neural network with the width 15 N and depth 2 L such that ∥ ϕ ∥ W 1 , ∞ (( -a,a ) 2 ) ≤ 12 a 2 and

For the last equation Eq. (27) is due to ϕ ( x, y ) in the proof can be read as Eq. (28) with ψ (0) = 0 .

Proposition 5. For any N,L,s ∈ N + with s ≥ 2 , there exists a σ 1 -NN ϕ with the width 9( N +1) + s -1 and depth 14 s ( s -1) L such that ∥ ϕ ∥ W 1 , ∞ ((0 , 1) s ) ≤ 18 and

Furthermore, for any i = 1 , 2 , . . . , s , if x i = 0 , we will have

̸

Proof. The proof of the first inequality Eq. (30) can be found in [23, Lemma 3.5]. The proof of Eq. (31) can be obtained via induction. For s = 2 , based on Proposition 4, we know there is a neural network ϕ 2 satisfied Eq. (31).

Now assume that for any i ≤ n -1 , there is a neural network ϕ i satisfied Eq. (31). ϕ n in [23] is constructed as

which satisfies Eq. (30). Then ϕ n ( x 1 , x 2 , . . . , x i -1 , 0 , x i +1 , . . . , x n ) = 0 for any i = 1 , 2 , . . . , n . For i = n , we have

For i < n and j < n , we have

For i < n and j = n , we have

Therefore, Eq. (31) is valid.

Proposition 6 ([23, Propositiion 3.6]) . For any N,L,s ∈ N + and | α | ≤ s , there is a σ 1 -NN ϕ with the width 9( N +1) + s -1 and depth 14 s 2 L such that ∥ ϕ ∥ W 1 , ∞ ((0 , 1) d ) ≤ 18 and

Proposition 7 ([39, Proposition 1]) . Given a sequence of the neural network { p i } i M =1 , and each p i is a σ 1 -NN from R → R with the width N and depth L i , then ∑ M i =1 p i is a σ 1 -NN with the width N +4 and depth ∑ M i =1 L i .

We present the proof of Proposition 1 below.

Proof of Proposition 1. First, we construct g 1 and g 2 by neural networks in [0 , 1] . Note that ⌊ L 2 /d ⌋ ≤ L 2 /d ≤ ( ⌊ L 1 /d ⌋ +1 ) 2 . We first construct a σ 1 -NN in the small set [ 0 , ⌊ N 1 /d ⌋⌊ L 2 /d ⌋ ] . It is easy to check there is a neural network ˆ ψ with the width 4 and one layer such as

0

8K

1

8K

3

8K

5

8K

7

[N1/d] [L2/d]

Hence, we have a network ψ 1 with the width 4 ⌊ N 1 /d ⌋ and one layer such as

Next, we construct ψ i for i = 2 , 3 , 4 based on the symmetry and periodicity of g i . ψ 2 is the function with period 2 ⌊ N 1 /d ⌋⌊ L 2 /d ⌋ in [ 0 , 1 ⌊ L 2 /d ⌋ ] , and each period is a hat function with gradient 1. ψ 3 is the function with period 2 ⌊ L 2 /d ⌋ in [ 0 , ⌊ L 1 /d ⌋ +1 ⌊ L 2 /d ⌋ ] , and each period is a hat function with gradient 1. ψ 4 is the function with period 2 ( ⌊ L 1 /d ⌋ +1 ) ⌊ L 2 /d ⌋ in [ 0 , ( ⌊ L 1 /d ⌋ +1 ) 2 ⌊ L 2 /d ⌋ ] , and each period is a hat function with gradient 1. The schematic diagram is in Fig. 3 (The diagram is shown the case for ⌊ N 1 /d ⌋ and ⌊ L 1 /d ⌋ +1 is a even integer.).

Note that ψ 2 ◦ ψ 3 ◦ ψ 4 ( x ) is the function with period 2 ⌊ N 1 /d ⌋⌊ L 2 /d ⌋ in [0 , 1] ⊂ [ 0 , ( ⌊ L 1 /d ⌋ +1 ) 2 ⌊ L 2 /d ⌋ ] , and each period is a hat function with gradient 1. Then function ψ 1 ◦ ψ 2 ◦ ψ 3 ◦ ψ 4 ( x ) is obtained by repeating reflection ψ 1 in [ 0 , ( ⌊ L 1 /d ⌋ +1 ) 2 ⌊ L 2 /d ⌋ ] , which is the function we want.

Similar with ψ 1 , ψ 2 is a network with 4 ⌊ N 1 /d ⌋ width and one layer. Due to Proposition 7, we know that ψ 3 and ψ 4 is a network with 7 width and ⌊ L 1 /d ⌋ +1 depth. Hence

is a network with 4 ⌊ N 1 /d ⌋ width and 2 ⌊ L 1 /d ⌋ + 4 depth and g 1 = ψ ( x + 1 8 K ) and g 1 = ψ ( x + 5 8 K ) .

Now we can construct g m for m ∈ { 1 , 2 } d based on Proposition 5: There is a neural network ϕ prod with the width 9( N +1) + d -1 and depth 14 d ( d -1) nL such that ∥ ϕ prod ∥ W 1 , ∞ ((0 , 1) d ) ≤ 18 and

Y2

43

44

1

2

0

[p/zT] [p/N] [p/zT] [p//N]

[L2/d]

1

2

3

[L1/] + 1

0

[L2/d]

[L2/d]

[L2/d]

[L2/d]

[L1/d] + 1  2([L1/d] + 1)

(LL1/d] + 1)²

0

[L2/d]

[L2/]

[L2/d]

Then denote ϕ m ( x ) := ϕ prod ( g m 1 , g m 2 , . . . , g m d ) which is a neural network with the width smaller than (9 + d )( N +1) + d -1 and depth smaller than 15 d ( d -1) nL . Furthermore, due to Lemma 3, we have

where the last inequality is due to

In the final of this subsection, we establish three lemmas for { Ω m } m ∈{ 1 , 2 } d , { g m } m ∈{ 1 , 2 } d and { ϕ m } m ∈{ 1 , 2 } d defined in Subsection 3.1.

Lemma 5. For { Ω m } m ∈{ 1 , 2 } d defined in Definition 5, we have

Proof. We prove this lemma via induction. d = 1 is valid due to Ω 1 ∪ Ω 2 = [0 , 1] . Assume that the lemma is true for d -1 , then

hence the case of d is valid, and we finish the proof of the lemma.

Lemma 6. { g m } m ∈{ 1 , 2 } d defined in Definition 6 satisfies:

(ii): supp g m ∩ [0 , 1] d ⊂ Ω m , where Ω m is defined in Definition 5.

Proof. (i) can be proved via induction as Lemma 5, and we leave it to readers.

As for (ii) and (iii), without loss of generality, we show the proof for m ∗ := (1 , 1 , . . . , 1) . For any x ∈ [0 , 1] d \ Ω m ∗ , there is x j ∈ [0 , 1] \ Ω 1 . Then g 1 ( x j ) = 0 and g m ∗ ( x ) = ∏ d j =1 g 1 ( x j ) = 0 , therefore supp g m ∗ ∩ [0 , 1] d ⊂ Ω m ∗ . Furthermore, d g mj ( x j ) d x j = 0 for x j ∈ [0 , 1] ∈ Ω 1 due to the definition of g 1 (Definition 6), then we finish this proof.

The following lemma demonstrates that ϕ m , as defined in Proposition 1, can restrict the Sobolev norm of the entire space to Ω m .

Lemma 7. For any χ ( x ) ∈ W 1 , ∞ ((0 , 1) d ) , denote

then we have

for any m ∈ { 1 , 2 } d , where ϕ m ( x ) and Ω m is defined in Proposition 1 and Definition. 5, and ϕ M is from Proposition 4 (choosing a = M in the proposition).

Proof. For the first equality, we only need to show that

According to the Proposition 1, we have ϕ m ( x ) = ϕ prod ( g m 1 , g m 2 , . . . , g m d ) , and for any x = ( x 1 , x 2 , . . . , x d ) ∈ (0 , 1) d \ Ω m , there is m j such as g m j ( x j ) = 0 and d g mj ( x j ) d x j = 0 due to Lemma 6. Based on Eq. (31) in Proposition 5, we have

̸

Hence we have

for all x ∈ (0 , 1) d \ Ω m .

Similarly, for the second equality in this lemma, we have

Furthermore,

for all x ∈ (0 , 1) d \ Ω m based on

and ∂ϕ m ( x ) ∂x q = 0 . Hence we finish our proof.

7.2.2 An approximation of functions in Sobolev spaces based on the Bramble-Hilbert Lemma [7, Lemma 4.3.8]

In this subsection, we establish { f K, m } m ∈{ 1 , 2 } d as mentioned in Subsection 3.1, which is presented in Theorem 6. To prove this result, we build upon the work of [18], which leverages the average Taylor polynomials and the Bramble-Hilbert Lemma to approximate functions in Sobolev spaces.

Before we show Theorem 6, we define subsets of Ω m for simplicity notations.

For any m ∈ { 1 , 2 } d , we define

i = ( i 1 , i 2 , . . . , i d ) ∈ { 0 , 1 . . . , K } d , and it is easy to check ⋃ i ∈{ 0 , 1 ...,K } d Ω m , i = Ω m .

Theorem 6. Let K ∈ N + and n ≥ 2 . Then for any f ∈ W n, ∞ ((0 , 1) d ) with ∥ f ∥ W n, ∞ ((0 , 1) d ) ≤ 1 and m ∈ { 1 , 2 } d , there exist piece-wise polynomials function f K, m = ∑ | α |≤ n -1 g f, α , m ( x ) x α on Ω m (Definition 5) with the following properties:

Furthermore, g f, α , m ( x ) : Ω m → R is a constant function with on each Ω m , i for i ∈ { 0 , 1 . . . , K } d . And

for all x ∈ Ω m , where C 1 and C 2 are constants independent with K .

This proof is similar to that of [18, Lemma C.4.], but we provide detailed proof as follows for readability. Before the proof, we must introduce the partition of unity, average Taylor polynomials, and a lemma.

Definition 7 (The partition of unity) . Let d, K ∈ N + , then

with h i : R d → R for all i ∈ { 0 , 1 , . . . , K } d is called the partition of unity [0 , 1] d if it satisfies (i): 0 ≤ h i ( x ) ≤ 1 for every h i ∈ Ψ .

Definition 8. Let n ≥ 1 and f ∈ W n, ∞ ((0 , 1) d ) , x 0 ∈ ((0 , 1) d ) and r > 0 such that for the ball B ( x 0 ) := B ( x 0 ) r, |·| which is a compact subset of ((0 , 1) d ) . The corresponding Taylor polynomial of order n of f averaged over B is defined for

where

Lemma 8. Let n ≥ 1 and f ∈ W n, ∞ ((0 , 1) d ) , x 0 ∈ Ω and r > 0 such that for the ball B ( x 0 ) := B r, |·| ( x 0 ) which is a compact subset of ((0 , 1) d ) . The corresponding Taylor polynomial of order n of f averaged over B can be read as

where C 2 ( n, d ) = ∑ | α + β |≤ n -1 1 α ! β ! .

Proof. Based on [18, Lemma B.9.], Q n f ( x ) can be read as

where for a β + α ≤ ( α + β )! α ! β ! . Note that

Then

The proof of Theorem 6 is based on average Taylor polynomials and the Bramble-Hilbert Lemma [7, Lemma 4.3.8].

Definition 9. Let Ω , B ∈ R d . Then Ω is called stared-shaped with respect to B if

Definition 10. Let Ω ∈ R d be bounded, and define

Then we define

the chunkiness parameter of Ω if R̸ = ∅ .

Lemma 9 (Bramble-Hilbert Lemma [7, Lemma 4.3.8]) . Let Ω ∈ R d be open and bounded, x 0 ∈ Ω and r > 0 such that Ω is the stared-shaped with respect to B := B r, |·| ( x 0 ) , and r ≥ 1 2 r ⋆ max . Moreover, let n ∈ N + , 1 ≤ p ≤ ∞ and denote by γ by the chunkiness parameter of Ω . Then there is a constant C ( n, d, γ ) > 0 such that for all f ∈ W n,p (Ω)

where Q n f denotes the Taylor polynomial of order n of f averaged over B and h = diam(Ω) .

Proof of Theorem 6. Without loss of generalization, we prove the case for m = (1 , 1 , . . . , 1) =: m ∗ .

Denote E : W n, ∞ ((0 , 1) d ) → W n, ∞ ( R d ) be an extension operator [43] and set ˜ f := Ef and C E is the norm of the extension operator.

Furthermore,

Define p f, i as the average Taylor polynomial Definition 8 in B i ,K := B 1 4 K , |·| ( 8 i +3 8 K ) i.e.

Based on Lemma 8, p f, i can be read as

The reason to define average Taylor polynomial on B i ,K is to use the Bramble-Hilbert Lemma 9 on

Note that

Therefore we can apply the Bramble-Hilbert Lemma 9 and have

where C BH ( n, d ) = |{| α | = n }| 1 d ∫ 1 0 x d -1 e -( 1 -x 2 ) -1 d x ( 2 + 4 √ d ) d C E by following the proof of Lemma [7, Lemma 4.3.8]. Therefore,

where C 1 ( n, d ) = 2 C BH ( n, d ) .

Now we construct a partition of unity that we use in this theorem. First of all, given any integer K , define { h i } K i =0 from R → R :

It is easy to check that { h i } K i =0 is a partition of unity of [0 , 1] and h i ( x ) = 1 for x ∈ [ i K , 3+4 i 4 K ] . Hence we can define h i ( x ) for i = ( i 1 , i 2 , . . . , i d ) ∈ { 0 , 1 , . . . , K } d and x = ( x 1 , x 2 , . . . , x d ) ∈ R d :

and { h i : i ∈ { 0 , 1 , . . . , K } d } is a partition of unity of [0 , 1] d and h i ( x ) = 1 for x ∈ ∏ d j =1 [ i j K , 3+4 i j 4 K ] = Ω m ∗ , i and i = ( i 1 , i 2 , . . . , i d ) ∈ { 0 , 1 , . . . , K } d .

Furthermore,

and

which is due to h i = 1 on Ω m ∗ , i .

Then

where

Finally,

which is due to ∪ i ∈{ 0 , 1 ,...,K } d Ω m ∗ , i = Ω m ∗ and supp h i ∩ Ω m ∗ = Ω m ∗ , i . Similarly,

Last of all,

with | g f, α , m ∗ ( x ) | ≤ C 2 ( n, d ) for x ∈ Ω m ∗ . Note that g f, α , m ∗ ( x ) is a step function from Ω m ∗ → R :

7.2.3 Approximation of functions in W n, ∞ with W 1 , ∞ norm by ReLU neural networks in the whole space except a small set

Theorem 7. For any f ∈ W n, ∞ ((0 , 1) d ) with ∥ f ∥ W n, ∞ ((0 , 1) d ) ≤ 1 , any N,L ∈ N + , and m = ( m 1 , m 2 , . . . , m d ) ∈ { 1 , 2 } d , there is a neural network ψ m with the width 25 n d +1 ( N +1)log 2 (8 N ) and depth 27 n 2 ( L +2)log 2 (4 L ) such that

where C 6 is the constant independent with N,L .

Proof. Without loss of the generalization, we consider the case for m ∗ = (1 , 1 , . . . , 1) . Due to Theorem 6 and setting K = ⌊ N 1 /d ⌋ 2 ⌊ L 2 /d ⌋ , we have

where f K, m ∗ = ∑ | α |≤ n -1 g f, α , m ∗ ( x ) x α for x ∈ Ω m ∗ . Note that g f, α , m ∗ ( x ) is a constant function for x ∈ ∏ d j =1 [ i j K , 3+4 i j 4 K ] and i = ( i 1 , i 2 , . . . , i d ) ∈ { 0 , 1 , . . . , K -1 } d . The remaining part is to approximate f K, m ∗ by neural networks.

The way to approximate g f, α , m ∗ ( x ) is similar with [23, Theorem 3.1]. First of all, due to Proposition 2, there is a neural network ϕ 1 ( x ) with the width 4 N +5 and depth 4 L +4 such that

Note that we choose δ = 1 4 K ≤ 1 3 K in Proposition 2. Then define

For each p = 0 , 1 , . . . , K d -1 , there is a bijection

such that ∑ d j =1 η j K j -1 = p . Then define

where C 2 ( n, d ) is the bounded of g f, α , m ∗ defined in Theorem 6. Therefore, based on Proposition 3, there is a neural network ˜ ϕ α ( x ) with the width 16 n ( N +1)log 2 (8 N ) and depth (5 L +2)log 2 (4 L ) such that | ˜ ϕ α ( p ) -ξ α ,p | ≤ N -2 n L -2 n for p = 0 , 1 , . . . K d -1 . Denote

and obtain that

Then we obtain that

which is due to ϕ α ( ϕ 2 ( x )) -g f, α , m ∗ ( x ) is a step function, and the first order weak derivative is 0 in Ω m ∗ .

Due to Proposition 6, there is a neural network ϕ 3 , α with the width 9( N +1) + n -1 and depth 14 n 2 L such that ∥ ϕ 3 , α ∥ W 1 , ∞ ((0 , 1) d ) ≤ 18 and

Due to Proposition 4, there is a neural network ϕ 4 with the width 15( N +1) and depth 4 n ( L +1) such that ∥ ϕ 4 ∥ W 1 , ∞ ( -C 3 ,C 3 ) 2 ≤ 12( C 2 ( n, d )) 2 and

where C 3 ( n, d ) = max { 3 C 2 ( n, d ) , 18 } .

Now we define the neural network ϕ m ∗ ( x ) to approximate f K, m ∗ ( x ) in Ω m ∗ :

The remaining question is to find the error E :

As for E 1 , due to Lemma 3, we have

where C 4 ( n, d ) = 12 dn d [ C 3 ( n, d ) + 1] ( C 2 ( n, d )) 2 .

As for E 2 , due to Lemma 4, we have

The estimation of E 3 is similar with that of E 2 which is

Therefore, using the total error is

where C 5 ( n, d ) = C 4 ( n, d ) + 72 n d C 2 ( n, d ) + 10 n d C 2 ( n, d ) n .

At last, we finish the proof by estimating the network's width and depth, implementing ψ m ∗ ( x ) . From Eq. (74), we know that ψ m ∗ ( x ) consists of the following subnetworks:

1. ϕ 3 , α ( x ) with the width 9( N +1) + n -1 and depth 14 n 2 L .

ϕ 2 ( x ) with the width 4 N +5 and depth 4 L +4 .

ϕ α with the width 16 n ( N +1)log 2 (8 N ) and depth (5 L +2)log 2 (4 L ) .

ϕ 4 ( x, y ) with the width 15( N +1) and depth 4 n ( L +1) .

Therefore ϕ ( x ) is a neural network with the width 25 n d +1 ( N +1)log 2 (8 N ) and depth 27 n 2 ( L + 2) log 2 (4 L ) .

Combining Eqs. (69) and (79), we have that there is a neural network ψ m ∗ with the width 25 n d +1 ( N + 1) log 2 (8 N ) and depth 27 n 2 ( L +2)log 2 (4 L ) such that

where C 6 = C 1 + C 5 is the constant independent with N,L .

Similarly, we can construct a neural network ψ m with the width 25 n d +1 ( N +1)log 2 (8 N ) and depth 27 n 2 ( L +2)log 2 (4 L ) which can approximate f on Ω m with same order of Eq. (80).

7.2.4 Proof of Theorem 3

Now we can prove Theorem 3 based on Theorem 7 and Proposition 1.

Proof of Theorem 3. Based on Theorem 7, there is a sequence of the neural network { ψ m ( x ) } m ∈{ 1 , 2 } d such that

where C 6 = C 1 + C 5 is the constant independent with N,L , and each ψ m is a neural network with the width 25 n d +1 ( N +1)log 2 (8 N ) and depth 27 n 2 ( L +2)log 2 (4 L ) . According to Proposition 1, there is a sequence of the neural network { ϕ m ( x ) } m ∈{ 1 , 2 } d such that

where { g m } m ∈{ 1 , 2 } d is defined in Definition 6 with ∑ m ∈{ 1 , 2 } d g m ( x ) = 1 and supp g m ∩ [0 , 1] d = Ω m . For each ϕ m , it is a neural network with the width smaller than (9 + d )( N +1) + d -1 and depth smaller than 15 d ( d -1) nL .

Due to Proposition 4, there is a neural network ̂ ϕ with the width 15( N +1) and depth 14 n 2 L such that ∥ ̂ ϕ ∥ W 1 , ∞ ( -C 7 ,C 7 ) 2 ≤ 12( C 7 ( n, d )) 2 and

where C 7 = C 6 +50 d 5 2 +1 .

Now we define

Note that

As for the first part,

where the last equality is due to Lemma 7. Based on Lemma 4 and ∥ f ∥ W 1 , ∞ ((0 , 1) d ) ≤ 1 , we have

And

where the second inequality is due to

Therefore

due to ( N +1) -4 dnL ≤ N -2 n L -2 n .

For the second part, due to Lemma 7, we have

Similarly with the estimation of E 1 (76), we have that

Combining (89) and (91), we have that there is a σ 1 -NN ϕ with the width (34 + d )2 d n d +1 ( N + 1) log 2 (8 N ) and depth 56 d 2 n 2 ( L +1)log 2 (4 L ) such that

where C 9 is the constant independent with N,L .

The method proposed in [28, 23, 39, 38, 37] may not be applied to prove Theorems 3. These works approximate the target function f using a deep neural network ϕ in the unit cube except for an arbitrarily small region Ω δ , as per [36, Lemma 2.2]. Since ∥ ϕ ∥ L ∞ (Ω) can be bounded and is independent of the size of Ω δ , ∥ f -ϕ ∥ L p (Ω) can be well estimated across the entire space for p ∈ [1 , + ∞ ) . For approximations measured in the L ∞ (Ω) norm, [28] translates the deep neural network ϕ , while [39] constructs different neural networks in the unit cube away from various negligible regions. Both methods aim to find neural networks { ϕ i ( x ) } i = 1 N that approximate the target function f well in different regions. They then observe that the middle value of { ϕ i ( x ) } N i =1 is close to f ( x ) for all x ∗ ∈ Ω , and construct the middle-value function using a ReLU neural network. However, these methods may not be generalized to prove the theorems presented in this paper.

Neither of the methods previously proposed can be applied to the approximation measured in Sobolev space. In the first method, ∥ ϕ ∥ W 1 , ∞ (Ω) depends on the length of Ω δ , and the derivative is substantial in the negligible region, as shown in [36, Lemma 2.2]. Thus, ∥ f -ϕ ∥ W 1 ,p (Ω) will be excessively large. In the second method, median value functions can only identify the median values, not the median values of functions and their derivatives simultaneously. In this paper, we overcome this difficulty using a partition of unity. We construct a partition of unity of Ω and approximate them using ReLU DNNs denoted as { ϕ m } m ∈{ 1 , 2 } d . For each ϕ m , its support set is the unit cube away from a small region, and we can construct a deep neural network ψ m that approximates the target function f well on supp ϕ m . We then combine { ϕ m } m ∈{ 1 , 2 } d and { ψ m } m ∈{ 1 , 2 } d to obtain a deep neural network that can approximate the target function f well across the entire space. This approach resolves the issue of simultaneous approximation of both functions and their derivatives in Sobolev spaces.

7.3 Proofs of Corollaries 1 and 2

7.3.1 Preliminaries

First, we list a few basic lemmas of σ neural networks repeatedly applied in our main analysis.

2 Lemma 10 ([23, Lemma 3.7]) . The following basic lemmas of σ 2 neural networks hold:

(i) σ 1 neural networks are σ 2 neural networks.

(ii) Any identity map in R d can be realized exactly by a σ 2 neural network with one hidden layer and 2 d neurons.

(iii) f ( x ) = x 2 can be realized exactly by a σ 2 neural network with one hidden layer and two neurons.

(iv) f ( x, y ) = xy = ( x + y ) 2 -( x -y ) 2 4 can be realized exactly by a σ 2 neural network with one hidden layer and four neurons.

(v) Assume x α = x α 1 1 x α 2 2 · · · x α d d for α ∈ N d . For any N,L ∈ N + such that NL +2 ⌊ log 2 N ⌋ ≥| α | , there exists a σ 2 neural network ϕ ( x ) with the width 4 N +2 d and depth L + ⌈ log 2 N ⌉ such that

for any x ∈ R d .

(vi) Assume P ( x ) = ∑ J j =1 c j x α j for α j ∈ N d . For any N,L,a,b ∈ N + such that ab ≥ J and ( L -2 b -b log 2 N ) N ≥ b max j | α j | , there exists a σ 2 neural network ϕ ( x ) with the width 4 Na +2 d +2 and depth L such that

Next, we define a function which will be repeatly used in the proof of Corollary 1 in this section.

Definition 11. Define s ( x ) from R → [0 , 1] as

0.8

0.6

°0

0.2

-0.5

D

0.5

1.5

2

2.5

3

3.5

Definition 12. Given K ∈ N + , then we define two functions in R :

Then for any m = ( m 1 , m 2 , . . . , m d ) ∈ { 1 , 2 } d , we define

for any x = ( x 1 , x 2 , . . . , x d ) ∈ R d .

Proposition 8. Given N,L,d ∈ N + with NL + 2 ⌊ log 2 N ⌋ ≥ d and L ≥ ⌈ log 2 N ⌉ , and setting K = ⌊ N 1 /d ⌋⌊ L 2 /d ⌋ , { s m ( x ) } m ∈{ 1 , 2 } d defined in Definition 12 satisfies:

(i): ∥ s m ( x ) ∥ L ∞ ((0 , 1) d ) ≤ 1 , ∥ s m ( x ) ∥ W 1 , ∞ ((0 , 1) d ) ≤ 8 K and ∥ s m ( x ) ∥ W 1 , ∞ ((0 , 1) d ) ≤ 64 K 2 for any m ∈ { 1 , 2 } d .

(ii): { s m ( x ) } m ∈{ 1 , 2 } d is a partition of the unity [0 , 1] d with supp s m ( x ) ∩ [0 , 1] d = Ω m defined in Definition 5.

(iii):For any m ∈ { 1 , 2 } d , there is a σ 2 neural network λ m ( x ) with the width 16 N +2 d and depth 4 L +5 such as

Proof. (i) and (ii) are proved by direct calculation. The proof of (iii) follows:

First, we architect s ( x ) by a σ 2 neural network. The is a σ 1 neural network g ( x ) with 3 the width and one layer such that:

Based on (iii) in Lemma 10, g 2 ( x ) is a σ 2 neural network with 3 the width and two layers. Then by direct calculation, we notice that

which is a σ 2 neural network with 12 the width and two layers. The ˜ g ( x ) defined as

is a σ 2 neural network with 12( ⌊ N 1 /d ⌋ ) the width and two layers.

Similar with Lemma 1, we know that

is a σ 2 neural network with 12( ⌊ N 1 /d ⌋ ) the width and 5 + 2 ⌊ L 1 /d ⌋ , and

Based on (v) in Lemma 10, we have there is a σ 2 neural network λ m ( x ) with the width 16 N +2 d and depth 4 L +5 such as

7.3.2 Proof of Corollaries 1 and 2

The proof is comprised of three parts, which include Theorem 8 and 9, followed by the combination of these results. Theorem 8 is to apply the Bramble-Hilbert Lemma 9 measured in the norm of W 2 , ∞ :

Theorem 8. Let K ∈ N + and n ≥ 2 . Then for any f ∈ W n, ∞ ((0 , 1) d ) with ∥ f ∥ W n, ∞ ((0 , 1) d ) ≤ 1 and m ∈ { 1 , 2 } d , there exist piece-wise polynomials function f K, m = ∑ | α |≤ n -1 g f, α , m ( x ) x α on Ω m (Definition 5) with the following properties:

Furthermore, g f, α , m ( x ) : Ω m → R is a constant function with on each Ω m , i for i ∈ { 0 , 1 . . . , K } d . And

for all x ∈ Ω m , where C 1 and C 2 are constants independent with K .

The proof is the same as that of Theorem 6. Note that { f K, m } m ∈{ 1 , 2 } d will be same in two theorems if f ∈ W n, ∞ ((0 , 1) d ) in two theorem are same.

Theorem 9 is to establish σ 2 neural networks { γ m } { 1 , 2 } d , and each γ m can approximate f well on Ω m .

Theorem 9. For any f ∈ W n, ∞ ((0 , 1) d ) with ∥ f ∥ W n, ∞ ((0 , 1) d ) ≤ 1 , any N,L ∈ N + with NL + 2 ⌊ log 2 N ⌋ ≥ n and L ≥ ⌈ log 2 N ⌉ , and m = ( m 1 , m 2 , . . . , m d ) ∈ { 1 , 2 } d , there is a σ 2 neural network γ m with the width 28 n d +1 ( N + d ) log 2 (8 N ) and depth 11 n 2 ( L +2)log 2 (4 L ) such that

where C 10 is the constant independent with N,L .

Proof. The proof is similar to that of Theorem 7; the difference is that xy and x α can be architected precisely by σ 2 neural networks.

Without loss of the generalization, we consider the case for m ∗ = (1 , 1 , . . . , 1) . Due to Theorem 8 and setting K = ⌊ N 1 /d ⌋ 2 ⌊ L 2 /d ⌋ , we have

where f K, m ∗ = ∑ | α |≤ n -1 g f, α , m ∗ ( x ) x α for x ∈ Ω m ∗ . Note that g f, α , m ∗ ( x ) is a constant function for x ∈ ∏ d j =1 [ i j K , 3+4 i j 4 K ] and i = ( i 1 , i 2 , . . . , i d ) ∈ { 0 , 1 , . . . , K -1 } d . The remaining part is to approximate f K, m ∗ by neural networks.

The way to approximate g f, α , m ∗ ( x ) is same with Theorem 7, and we have that

which is due to ϕ α ( ϕ 2 ( x )) -g f, α , m ∗ ( x ) is a step function, and the first order weak derivative is 0 in Ω m ∗ .

Due to (v) in Lemma 10, there is a σ 2 neural network ϕ 5 , α ( x ) with the width 4 N +2 d and depth L + ⌈ log 2 N ⌉ such that

Due to (iv) in Lemma 10, there is a σ 2 neural network ϕ 6 ( x ) with the width 4 and depth 1 such that

Now we define the neural network γ m ∗ ( x ) to approximate f K, m ∗ ( x ) in Ω m ∗ :

The remaining question is to find the error E :

At last, we finish the proof by estimating the network's the width and depth, implementing γ m ∗ ( x ) . From Eq. (106), we know that γ m ∗ ( x ) consists of the following subnetworks:

ϕ 5 , α ( x ) with the width 4 N +2 d and depth L + ⌈ log 2 N ⌉ .

ϕ 2 ( x ) with the width 4 N +5 and depth 4 L +4 .

ϕ α with the width 16 n ( N +1)log 2 (8 N ) and depth (5 L +2)log 2 (4 L ) .

ϕ 6 ( x, y ) with the width 4 and depth 1 .

Therefore ϕ ( x ) is a neural network with the width 28 n d +1 ( N + d ) log 2 (8 N ) and depth 11 n 2 ( L + 2) log 2 (4 L ) .

Combining Eqs. (102) and (107), we have that there is a neural network γ m ∗ with the width 28 n d +1 ( N + d ) log 2 (8 N ) and depth 11 n 2 ( L +2)log 2 (4 L ) such that

where C 10 = C 1 +2 n d +2 C 2 is the constant independent with N,L .

Similarly, we can construct a neural network γ m with the width 28 n d +1 ( N + d ) log 2 (8 N ) and depth 11 n 2 ( L +2)log 2 (4 L ) which can approximate f on Ω m with same order of Eq. (108).

The last part is to combine { λ m } m ∈{ 1 , 2 } d and { γ m } m ∈{ 1 , 2 } d in [0 , 1] d and obtain a σ 2 neural network to approximate f measured in the norm of W 2 .

Proof of Corollary 1. Based on Theorem 9, there is a sequence of the neural network { γ m ( x ) } m ∈{ 1 , 2 } d such that

where C 10 is the constant independent with N,L , and each γ m is a neural network with the width 28 n d +1 ( N + d ) log 2 (8 N ) and depth 11 n 2 ( L +2)log 2 (4 L ) . According to Proposition 8, there is a sequence of the neural network { s m ( x ) } m ∈{ 1 , 2 } d satisfies:

(i): ∥ s m ( x ) ∥ L ∞ ((0 , 1) d ) ≤ 1 , ∥ s m ( x ) ∥ W 1 , ∞ ((0 , 1) d ) ≤ 8 K and ∥ s m ( x ) ∥ W 1 , ∞ ((0 , 1) d ) ≤ 64 K 2 for any m ∈ { 1 , 2 } d .

(ii): { s m ( x ) } m ∈{ 1 , 2 } d is a partition of the unity [0 , 1] d with supp s m ( x ) ∩ [0 , 1] d = Ω m defined in Definition 5.

For each s m , it is a σ 2 neural network with the width 16 N +2 d and depth 4 L +5 .

Due to (iv) in Lemma 10, there is a σ 2 neural network ϕ 6 ( x ) with the width 4 and depth 1 such that

Now we define

Note that

where the last equality is due to supp s m ( x ) ∩ [0 , 1] d = Ω m .

Then due to chain rule, for each m ∈ { 1 , 2 } d , we have

Hence

At last, we finish the proof by estimating the network's width and depth, implementing γ ( x ) . From Eq. (111), we know that γ ( x ) consists of the following subnetworks:

1. γ m ( x ) with the width 28 n d +1 ( N + d ) log 2 (8 N ) and depth 11 n 2 ( L +2)log 2 (4 L ) .

2. s m ( x ) with the width 16 N +2 d and depth 4 L +5 .

3. ϕ 6 ( x, y ) with the width 4 and depth 1 .

Therefore γ ( x ) is a neural network with the width 2 d +6 n d +1 ( N + d ) log 2 (8 N ) and depth 15 n 2 ( L + 2) log 2 (4 L ) .

Our method can easily extend to approximations measured by the norm of W m, ∞ . The primary difference in the proof lies in the need to establish a differential { s m ( x ) } { 1 , 2 } d , which can be achieved by constructing architected s m ( x ) as piece-wise m -degree polynomial functions. By extanding this approach, we can obtain Corollary 2 using our method.

7.4 Proof of Theorem 4

Proof. The Theorem 4 will be proved by contradiction. The idea of the proof is inspired by Ref. [28].

Claim 1. There exist ρ, C 1 , C 2 , C 3 , J 0 > 0 and s, d ∈ N + such that, for any f ∈ F n,d , we have

for all NL ≥ J 0 , where

̂ Φ := { ϕ : ReLU FNNs ϕ with the width ≤ C 1 N log N and depth ≤ C 2 L log L } .

The remaining question is to show Claim 1 is invalid.

Denote

Due to Theorem 1, we obtain

Now we will use Claim 1 to estimate a lower bound

of VCDim( D ̂ Φ) . In other words, we will construct { ψ β ( x ) : ψ β ( x ) ∈ D ̂ Φ , β ∈ B} to scatter b l points. B will be defined later.

First, fix i = 1 , . . . , d , and there exists ˜ g ∈ C ∞ (0 , 1) d such that ∂ ˜ g ( 0 ) ∂x i = 1 and ˜ g ( x ) = 0 for ∥ x ∥ 2 ≥ 1 / 3 . And we can find a constant C 5 > 0 such that g := ˜ g/C 5 ∈ F n,d .

Denote M = ⌊ ( NL ) 2 d + ρ 2( n -1) ⌋ . Divide [0 , 1] d into M d non-overlapping sub-cubes { Q θ } θ as follows:

for any index vector θ = [ θ 1 , θ 2 , · · · , θ d ] T ∈ { 1 , 2 , · · · , M } d . Denote the center of Q θ by x θ for all θ ∈ { 1 , 2 , · · · , M } d . Define

For each β ∈ B , we define, for any x ∈ R d ,

Due to | supp ˜ g ( x ) | ≤ 2 3 and | D α h β ( x ) | ≤ M -n + | α | ∥ g ∥ W n, ∞ ≤ 1 , we obtain that

for any | α | ≤ n Therefore, f β ∈ F n,d . And it is easy to check { D i h β = h β : β ∈ B} can scatters b l points since ∂ ˜ g ( 0 ) ∂x i = 1 and ˜ g ( x ) = 0 for ∥ x ∥ 2 ≥ 1 / 3 .

Note that for any h β ∈ F n,d , there is a ϕ β ∈ ̂ Φ such that C 3 ( NL ) -2( n -1) d -ρ 2 ≥ | D i h β ( x θ ) -D i ϕ β ( x θ ) | for any J β ≤ NL due to Claim 1. Denote J 1 = max β ∈B { J β } . There is a constant J 2 such that M -n +1 C 5 ≥ C 3 ( NL ) -2( n -1) d -ρ for J 2 ≤ NL . Define J := max { J 1 , J 2 } , then for any J ≤ NL , we have

In other words, for any β ∈ B and θ ∈ { 1 , 2 , · · · , M } d , D i f β ( x θ ) and D i ϕ β ( x θ ) have the same sign. Then { D i ϕ β : β ∈ B} shatters { x θ : θ ∈ { 1 , 2 , · · · , M } d } since { D i h β : β ∈ B} shatters { x θ : θ ∈ { 1 , 2 , · · · , M } d } as discussed above. Hence,

for N,L ∈ N with NL ≥ J .

By Eqs. (115,117), for any N,L ∈ N with NL ≥ J , we have b l ≤ VCDim( { ϕ β : β ∈ B} ) ≤ VCDim( D ̂ Φ) ≤ b u , implying that

which is a contradiction for sufficiently large N,L ∈ N . So we finish the proof of Theorem 4.

Based on the proof of Theorem 4, we can easily check that the estimation of VC-dimension of DNN derivatives (Theorem 1) is nearly optimal and prove Corollary 3. Assume VCDim( D ̂ Φ) ≤ b u = O ( N 2 -ε L 2 -ε ) in Eq. (118) for ε > 0 , and b l must be larger than ⌊ ( NL ) 2 d ⌋ d according the construction in the proof of Theorem 4 and Theorem 3. Hence we still obtain a contradiction in Eq. (118), and the estimation in Theorem 1 is nearly optimal.

7.5 Proof of Theorem 5

7.5.1 Bounding generalization error by Rademacher complexity

Definition 13 (Rademacher complexity [3]) . Given a sample set S = { z 1 , z 2 , . . . , z M } on a domain Z , and a class F of real-valued functions defined on Z , the empirical Rademacher complexity of F in S is defined as

where Σ M := { σ 1 , σ 2 , . . . , σ M } are independent random variables drawn from the Rademacher distribution, i.e., P ( σ i = +1) = P ( σ i = -1) = 1 2 for i = 1 , 2 , . . . , M. For simplicity, if S = { z 1 , z 2 , . . . , z M } is an independent random variable set with the uniform distribution, denote

The following lemma will be used to bounded generalization error by Rademacher complexities:

Lemma 11 ([47], Proposition 4.11) . Let F be a set of functions. Then

where X := { x 1 , . . . , x M } is an independent random variable set with the uniform distribution.

Now we can show that generalization error can be bounded by Rademacher complexities of two function sets.

Lemma 12. Let d, N, L, M ∈ N + , B,C 1 , C 2 ∈ R + . For any f ∈ W 1 , ∞ ((0 , 1) d ) with ∥ f ∥ W 1 , ∞ ((0 , 1) d ) ≤ 1 , set

We have

where E is expected responding to X , and X := { x 1 , . . . , x M } is an independent random variables set uniformly distributed on (0 , 1) d .

Proof. For any ϕ ( x ; θ ) ∈ ˜ Φ , we have

where the last inequality is due to Lemma 12.

7.5.2 Bounding the Rademacher complexity and the proof of Theorem 5

In this subsection, we aim to estimate the Rademacher complexity using the covering number. We then estimate the covering number using the pseudo-dimension.

Definition 14 (covering number [3]) . Let ( V, ∥·∥ ) be a normed space, and Θ ∈ V . { V 1 , V 2 , . . . , V n } is an ε -covering of Θ if Θ ⊂ ∪ n i =1 B ε, ∥·∥ ( V i ) . The covering number N ( ε, Θ , ∥ · ∥ ) is defined as

Definition 15 (Uniform covering number [3]) . Suppose the F is a class of functions from F to R . Given n samples Z n = ( z 1 , . . . , z n ) ∈ X n , define

The uniform covering number N ( ε, F , n ) is defined as

where N ( ε, F| Z n , ∥ · ∥ ∞ ) denotes the ε -covering number of F| Z n w.r.t the L ∞ -norm.

Then we use a lemma to estimate the Rademacher complexity using the covering number.

Lemma 13 (Dudley's theorem [3]) . Let F be a function class such that sup f ∈F ∥ f ∥ ∞ ≤ B . Then the Rademacher complexity R n ( F ) satisfies that

To bound the Rademacher complexity, we employ Lemma 13, which bounds it by the uniform covering number. We estimate the uniform covering number by the pseudo-dimension based on the following lemma.

Lemma 14 ([3]) . Let F be a class of functions from X to [ -B,B ] . For any ε > 0 , we have

for n ≥ Pdim ( F ) .

The remaining problem is to bound Pdim ( ˜ Φ) and Pdim ( D ˜ Φ) . Based on [4], Pdim ( ˜ Φ) = O ( L 2 N 2 log 2 L log 2 N ) . For the Pdim ( D ˜ Φ) , we can estimate it by Theorem 2.

Now we can estimate generalization error based on Lemma 12.

Proof of Theorem 5. Let J = max { Pdim ( D ˜ Φ) , Pdim ( ˜ Φ) } . Due to Lemma 13, 14 and Theorem 2, for any M ≥ J , we have

By the direct calculation for the integral, we have

Then choosing δ = B ( Pdim ( D ˜ Φ) M ) 1 2 ≤ B , we have

Therefore, due to Theorem 2, there is a constant C 4 independent with L, N, M such as

R M ( ˜ Φ) can be estimate in the similar way. Due to Lemma 12, we have that there is a constant C 5 = C 5 ( B,d,C 1 , C 2 ) such that

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a method to estimate the Vapnik-Chervonenkis (VC) dimension and pseudo-dimension of deep neural network (DNN) derivatives with the ReLU activation function, which have important applications such as characterizing the generalization error of machine learning methods and establishing the optimal approximation of DNNs in Sobolev spaces.\n\nThe authors provided theoretical analysis and proofs for their proposed method, which fills a gap in learning error estimations for many physics-informed machine learning models and applications, including solving partial differential equations, operator learning, network compression, and regularization.\n\nAnother contribution of the paper is the demonstration of how DNNs can be used to approximate functions in Sobolev spaces using ReLU activation functions in a deep feedforward neural network architecture, with a nearly-optimal approximation rate.\n\nOverall, the study provides a framework for analyzing and optimizing DNNs for different applications while taking into account mathematical concepts such as VC-dimension and pseudo-dimension.\n\nSTRENGTHS:\n* The topic of the paper is highly relevant to the field of deep learning and offers an interesting approach to estimate VC-dimension and pseudo-dimension of derivatives of deep neural networks.\n* The paper is well-written and clearly presents the mathematical language and definitions used in the study. The proofs provided are detailed and structured in a logical manner.\n* The paper presents two important theorems that provide a solution to the approximation rate problem of DNNs in Sobolev spaces and the degree of generalization error in loss functions involving derivatives of DNNs.\n* The proposed approach has the potential to be applied in different areas of physics-informed machine learning such as solving partial differential equations, operator learning, and generative models.\n\nWEAKNESSES:\n- Some aspects of the paper could be clearer and more thoroughly explained. The introduction, for instance, could better demonstrate the main contribution of the paper and provide a more detailed overview of the state-of-the-art and the limitations of existing research.\n\n- The section on references could be more comprehensive, covering more related studies and presenting a more thorough overview of the existing literature.\n\nQUESTIONS:\n1. Pseudo dimension is a more general concept than VC dimension,  while the bounds in Theorem 1 and Theorem 2 seems similar to each other except two constants $\\hat{C}$ and $\\overline{C}$, is there any relationship between  $\\hat{C}$ and $\\overline{C}$ ?\n\n2. Are those results in this paper holds only for ReLU networks?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper facilitates the understandings of Sobolev training and performances of DNNs in Sobolev spaces by providing the near optimal VC-dimension and pseudo-dimension of DNN derivatives.\n\nSTRENGTHS:\nTechnically, they improve the bounds on VC and pseudo-dimensions of DNN derivatives in reference [10].\n\nWEAKNESSES:\nThough I think the theoretical contributions of this paper is great, but in terms of readability of the paper, there are some rooms for the improvements. \\\\\nFirst, this paper assumes that the readers are very familiar with the notions of Sobolev training. In my opinion, authors should defer some technical proofs in the appendix, and introduce the notion briefly in the main paper, and motivate readers why the Sobolev training is interesting problem to consider. \\\\\nSecond, some notations should be introduced first, before being stated. I realized the Sobolev Spaces $W^{n,\\infty}([0,1]^{d})$ first appeared in line 59, then introduced in line 114, formally.\\\\\nThird, some sentences are repeated quite often. For instance, line numbers 125-127 are same with line numbers 167-169.\n\nQUESTIONS:\n1.\tTo my knowledge, VC-dimension and Pseudo-dimension are essentially same notion. (Reference [4].) I am wondering why the results in Theorem 1 and Theorem 2 are surprising in a sense that they have the same bound. Is there any intuitive reason on why it is non-trivial to expect they should be same for the DNN derivatives?  \\\\\n2.\tWhat is the meaning of approximating functions in $W^{n, \\infty}([0,1]^{d})$ with Sobolev norm $W^{1,\\infty}([0,1]^{d})$? Why this is interesting?  \\\\\n3.\tHow do we know the bound is optimal? To my knowledge, we commonly refer that we have an optimal bound when we have the matching orders of lower bound and upper bounds. But the author only provides the upper bounds in the paper.\n\nLIMITATIONS:\nThis work has no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide estimates on two measures of statistical complexity, the VC-dimension and the pseudo-dimension, of derivatives of deep neural networks. The estimate of the VC-dimension is shown to be optimal up to logarithmic factors. They also propose a constructive method for approximating functions in Sobolev spaces by deep neural networks. The VC-dimension bound is used to show that the obtained approximation rate is optimal as a function of the width and depth of the network. Finally, they prove a generalization bound in terms of Sobolev norm by leveraging the pseudo-dimension upper bound.\n\nSTRENGTHS:\nThe paper is globally well-written and pleasant to read. It brings several new results regarding the statistical and approximation properties of deep neural networks in terms of Sobolev norms, which could be of broad use, in particular in the community of deep learning for PDEs. I like that most of the presented bounds have matching lower bounds. I have not checked the proofs in details, so I cannot provide evidence on their soundness, but the mathematical statements presented in the main paper are easy to understand and unambiguous.\n\nWEAKNESSES:\nI do not have any strong reservation, a few questions are list below. The only part of the paper that I found hard to follow is the proof of Theorem 1. I suggest that authors take advantage of the additional page to expand a bit on the proof. Perhaps a drawing would help?\n\nQUESTIONS:\n+ Line 77: you claim that the estimate of the pseudo-dimension is nearly optimal, but I do not see a lower bound in the paper. Could you provide a lower-bound or at least an argument on how to obtain one, or otherwise change the phrasing of this sentence (and similar ones elsewhere in the paper)?\n+ Line 101: I don’t understand the \\leq sign. Shouldn’t it be an equal sign? Otherwise, I feel the argument of lines 184-187 breaks down, since \\sigma_2 networks include ReLU networks.\n+ Line 129: the dependence of the width on the dimension d is exponential. Is this expected? Do you think that you could get a matching dependence in the lower boud?\n+ Line 224 and Theorem 5: I think it would be clearer to upper bound your generalization error term by 2 sup_{\\theta} |\\Esp(R_S(\\theta)) - R_D(\\theta)|. Otherwise it is a bit confusing since, without further clarification, the expectation applies both to the estimator \\theta_S and to the random function R_S. Similarly, in the proof of Lemma 12, I don’t think that the proof is correct as it is if you apply it for \\theta_S, since \\theta_S depends on the random sample. However, it is correct if you write it for any (deterministic) \\theta, thereby getting the sup over \\theta as in the LHS of Lemma 11, which you can then apply to get the same upper bound that you get with your proof.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main contribution of the paper are new VC dimension and pseudo-dimension bounds for derivatives of functions implemented by deep neural networks. The utility of these bounds is demonstrated by proving the tightness of approximation error bounds in the Sobolev norm for networks with ReLU and ReLU squared activations, and by giving an improved generalization bound in a similar setting.\n\nSTRENGTHS:\n**Contribution.** This is a good technical paper that improves state of the art in the theoretical studies of VC/pseudo-dimensions and approximation and generalization rates of DNNs. The focus of the paper is the setting where VC/pseudo-dimensions are estimated for model derivatives, and approximation/generalization is considered with respect to Sobolev norms. This setting is not so well-explored as the more common setting where the fitted functions are assumed to belong to a Sobolev space, but the error of fitting does not involve the derivatives. The main results claimed in the paper are new VC/pseudo-dimensions bounds. The other results are new approximation and generalization bounds. The VC/pseudo-dimension bounds naturally help to obtain the generalization bounds and show the tightness of approximation bounds. It appears that all the results established in the paper improve previous analogous results in terms of giving more accurate/tight rates. \n\n**Quality and clarity.** The paper is fairly well written. All the results are precisely stated, sketches of proofs are provided where appropriate (full proofs provided in the appendix), connections between the results are well-explained, previous work is duly mentioned.\n\nWEAKNESSES:\nI don't see any major issues in the paper, but my overall impression is that it is fairly technical and lacks significant new insights. Virtually all results in the paper rely very heavily on previous research, and most of them look like being assembled from ideas scattered across many previous publications. I find it hard to name new ideas that never appeared before. I would say that this paper is more suitable for a journal.\n\nI think that the claims of achievement in this paper are exaggerated. The main claim is connected with the new VC-dimension bound: \"obtaining such bounds for DNN derivatives is much more difficult\", \"DNN derivatives consist of a series of interdependent parts...rendering existing methods for estimating bounds inapplicable\". In fact, the bound for VC-dimension of derivatives given in Theorem 1 is very close to the bound for the original network function given in Theorem 7 of Bartlett et al (2019), and the proof of Theorem 1 is just a slight modification of the proof of Theorem 7 in Bartlett et al (2019). This is not surprising because the chain rule expression (13) for the derivatives of the network function is only slightly more complicated than the original function for the purpose of partitioning the parameter domain into piecewise polynomial components and estimating the degrees of the resulting polynomials, as required for the proof. The authors claim \"we propose a method to achieve nearly optimal estimations of the VC-dimension and pseudo-dimension of DNN derivatives\", but I don't see here any new method.\n\nQUESTIONS:\nWhat are the important non-technical takeaways from this paper?\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}