SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this subsection, we utilize neural networks to approximate functions in  $W^{n,\infty}$  measured by  $W^{m,\infty}$ , where  $m > 1$ . The proof strategy is similar to the approximation measured in the norm of  $W^{1,\infty}$ . However, we cannot rely on ReLU neural networks alone to achieve this goal, as ReLU neural networks are piece-wise linear functions that do not belong to  $W^{m,\infty}$  with  $m > 1$ . Note that the Bramble-Hilbert Lemma is still applicable in higher-order approximation. Therefore, we need to approximate the function  $f_{K,m} = \sum_{|\alpha| \leq n-1} g_{f,\alpha,m}(x) x^{\alpha}$  within each domain  $\Omega_m$  using DNNs, where  $g_{f,\alpha,m}(x)$  represents a piece-wise constant function within  $\Omega_m$ . Consequently, ReLU-based DNNs can effectively approximate  $g_{f,\alpha,m}(x)$  within  $\Omega_m$  when measured by higher-order Sobolev spaces, since both the higher-order derivatives of  $g_{f,\alpha,m}(x)$  and ReLU-based DNNs are zero. The parts of ReLU-based DNNs that do not have high-order derivatives appear in the domain  $\Omega \setminus \Omega_m$ , and we will use the partition of unity to ensure that this part disappears in the final presentation, this is the reason why it is still acceptable to have ReLU activations appearing in the network. However, when it comes to approximating  $x^{\alpha}$  for  $|\alpha| > 1$  based on high-order Sobolev norms, ReLU-based DNNs fail to provide accurate results. Therefore, we require DNNs that utilize the square of ReLU activation in this specific scenario. This is how we construct our approach and the reason why we employ both ReLU and the square of ReLU activation functions.
Instead, we examine the use of  $\sigma_{2}$  neural networks for approximating functions measured in the norm of  $W^{2,\infty}$ . As per Corollary 1, a neural network with  $O(N\log N)$  width and  $O(L\log L)$  depth can achieve a nonasymptotic approximation rate of  $O(N^{-2(n - 2) / d}L^{-2(n - 2) / d})$  with respect to the  $W^{2,\infty}((0,1)^{d})$  norm. Moreover, our method can be extended to approximations measured in the norm of  $W^{m,\infty}$  with  $m > 2$ , as shown in Corollary 2. The proof strategy is similar to that used in Subsection 3.1, except that we need to construct a smoother partition of unity rather than  $\{g_m\}_{m\in \{1,2\}^d}$ . The corollaries are presented below, and further details are provided in Appendix 7.3.
Corollary 1. For any  $f \in W^{n,\infty}((0,1)^d)$  with  $\|f\|_{W^{n,\infty}((0,1)^d)} \leq 1$ , any  $N, L \in \mathbb{N}_+$  with  $NL + 2^{\lfloor \log_2 N \rfloor} \geq \max\{d, n\}$  and  $L \geq \lceil \log_2 N \rceil$ , there is a  $\sigma_2$ -NN  $\gamma(\pmb{x})$  with the width  $2^{d+6} n^{d+1}(N + d) \log_2(8N)$  and depth  $15n^2 (L + 2) \log_2(4L)$  such that
$$
\| f (\boldsymbol {x}) - \gamma (\boldsymbol {x}) \| _ {W ^ {2, \infty} ((0, 1) ^ {d})} \leq 2 ^ {d + 7} C _ {1 0} (n, d) N ^ {- 2 (n - 2) / d} L ^ {- 2 (n - 2) / d},
$$
where  $C_{10}$  is the constant independent with  $N, L$ .
Corollary 2. For any  $f \in W^{n,\infty}((0,1)^d)$  with  $\| f\|_{W^{n,\infty}((0,1)^d)} \leq 1$ , any  $N,L,m \in \mathbb{N}_{+}$  with  $NL + 2^{\lfloor \log_2N\rfloor} \geq \max \{d,n\}$  and  $L \geq \lceil \log_2N\rceil$ , there is a  $\sigma_{2}\text{-NN}\varphi (\pmb {x})$  with the width  $O(N\log N)$  and depth  $O(L\log L)$  such that
$$
\left\| f (\boldsymbol {x}) - \varphi (\boldsymbol {x}) \right\| _ {W ^ {m, \infty} ((0, 1) ^ {d})} \leq C _ {1 1} (n, d, m) N ^ {- 2 (n - m) / d} L ^ {- 2 (n - m) / d},
$$
where  $C_{11}$  is the constant independent with  $N, L$ .
3.3 Optimality of Theorem 3 via estimation of VC-dimension of DNN derivatives (Theorem 1)
In this section, we demonstrate that the approximation rate presented in Theorem 3 is nearly asymptotically optimal:
Theorem 4. Given any  $\rho, C_1, C_2, C_3, J_0 > 0$  and  $n, d \in \mathbb{N}^+$ , there exist  $N, L \in \mathbb{N}$  with  $NL \geq J_0$  and  $f$  with  $\| f \|_{W^{n,\infty}((0,1)^d)} \leq 1$ , satisfying for any  $\sigma_1$ -NN  $\phi$  with the width smaller than  $C_1N\log N$  and depth smaller than  $C_2L\log L$ , we have
$$
\left| \phi - f \right| _ {W ^ {1, \infty} ((0, 1) ^ {d})} > C _ {3} L ^ {- 2 (n - 1) / d - \rho} N ^ {- 2 (n - 1) / d - \rho}. \tag {9}
$$
In other words, the approximation rate of  $O(N^{-2(n - 1) / d - \rho}K^{-2(n - 1) / d - \rho})$  cannot be achieved asymptotically when ReLU  $\sigma_{1}$ -NNs with width  $O(N\log N)$  and depth  $O(L\log L)$  to approximate functions in  $\mathcal{F}_{n,d} := \{f\in W^{n,\infty}((0,1)^d):\| f\|_{W^{n,\infty}((0,1)^d)}\leq 1\}$ . The proof of Theorem 4 is based on the estimation of the VC-dimension of DNN derivatives, which is provided in Theorem 1.
Theorem 1 plays a crucial role in our proof of Theorem 4, which is established through a proof by contradiction following the approach outlined in Ref. [28]. Further details on the proof can be found in Appendix 7.4. The main idea behind the proof is that Theorem 1 characterizes the complexity of DNN derivatives, which in turn limits the ability of DNNs to approximate functions in Sobolev spaces.
In this paper, we focus on the optimality of approximation rate with respect to width  $N$  and depth  $L$  of DNNs. The dimensionality  $d$  is not the focus that we consider in our research. Addressing the question about mitigating the exponential dependence of width on dimensionality, we have observed that this arises from the utilization of methods like Taylor's expansion or average Taylor polynomials in our approximation techniques. It remains an open question for future research to explore alternative approaches to address the challenge of getting the dependence of  $d$  in the lower bounds.
4 Generalization Analysis in Sobolev Spaces via Estimation of Pseudo-dimension of DNN Derivatives (Theorem 2)
In a typical supervised learning algorithm, the objective is to learn a high-dimensional target function  $f(\pmb{x})$  defined on  $(0,1)^d$  with  $\|f\|_{W^{n,\infty}((0,1)^d)} \leq 1$  from a finite set of data samples  $\{(\pmb{x}_i, f(\pmb{x}_i))\}_{i=1}^M$ . When training a DNN, we aim to identify a DNN  $\phi(\pmb{x}; \pmb{\theta}_S)$  that approximates  $f(\pmb{x})$  based on random data samples  $\{(\pmb{x}_i, f(\pmb{x}_i))\}_{i=1}^M$ . We assume that  $\{x_i\}_{i=1}^M$  is an i.i.d. sequence of random variables uniformly distributed on  $(0,1)^d$  in this section. Denote
$$
\boldsymbol {\theta} _ {D} := \arg \inf  _ {\boldsymbol {\theta}} \mathcal {R} _ {D} (\boldsymbol {\theta}) := \arg \inf  _ {\boldsymbol {\theta}} \int_ {(0, 1) ^ {d}} | \nabla (f (\boldsymbol {x}) - \phi (\boldsymbol {x}; \boldsymbol {\theta})) | ^ {2} + | f (\boldsymbol {x}) - \phi (\boldsymbol {x}; \boldsymbol {\theta}) | ^ {2} d \boldsymbol {x}, \tag {10}
$$
$$
\boldsymbol {\theta} _ {S} := \arg \inf  _ {\boldsymbol {\theta}} \mathcal {R} _ {S} (\boldsymbol {\theta}) := \arg \inf  _ {\boldsymbol {\theta}} \frac {1}{M} \sum_ {i = 1} ^ {M} \left[ | \nabla \left(f \left(\boldsymbol {x} _ {i}\right) - \phi \left(\boldsymbol {x} _ {i}; \boldsymbol {\theta}\right)\right) | ^ {2} + | f \left(\boldsymbol {x} _ {i}\right) - \phi \left(\boldsymbol {x} _ {i}; \boldsymbol {\theta}\right) | ^ {2} \right]. \tag {11}
$$
The overall inference error is  $\mathbf{E}\mathcal{R}_D(\pmb{\theta}_S)$ , which can be divided into two parts:
$$
\begin{array}{l} \mathbf {E} \mathcal {R} _ {D} (\boldsymbol {\theta} _ {S}) = \mathcal {R} _ {D} (\boldsymbol {\theta} _ {D}) + \mathbf {E} \mathcal {R} _ {S} (\boldsymbol {\theta} _ {D}) - \mathcal {R} _ {D} (\boldsymbol {\theta} _ {D}) + \mathbf {E} \mathcal {R} _ {S} (\boldsymbol {\theta} _ {S}) - \mathbf {E} \mathcal {R} _ {S} (\boldsymbol {\theta} _ {D}) + \mathbf {E} \mathcal {R} _ {D} (\boldsymbol {\theta} _ {S}) - \mathbf {E} \mathcal {R} _ {S} (\boldsymbol {\theta} _ {S}) \\ \leq \underbrace {\mathcal {R} _ {D} \left(\boldsymbol {\theta} _ {D}\right)} _ {\text {a p p r o x i m a t i o n e r r o r}} + \underbrace {\mathbf {E} \mathcal {R} _ {S} \left(\boldsymbol {\theta} _ {D}\right) - \mathcal {R} _ {D} \left(\boldsymbol {\theta} _ {D}\right) + \mathbf {E} \mathcal {R} _ {D} \left(\boldsymbol {\theta} _ {S}\right) - \mathbf {E} \mathcal {R} _ {S} \left(\boldsymbol {\theta} _ {S}\right)} _ {\text {g e n e r a l i z a t i o n e r r o r}}, \tag {12} \\ \end{array}
$$
where the last inequality is due to  $\mathbf{E}\mathcal{R}_S(\pmb {\theta}_S)\leq \mathbf{E}\mathcal{R}_S(\pmb {\theta}_D)$  by the definition of  $\pmb{\theta}_{S}$
Due to Theorem 3, we know that the approximation error  $\mathcal{R}_D(\pmb{\theta}_D)$  is a  $O(N^{-4(n-1)/d}L^{-4(n-1)/d})$  term since  $\|f(\pmb{x}) - \phi(\pmb{x})\|_{H^1((0,1)^d)} \leq \|f(\pmb{x}) - \phi(\pmb{x})\|_{W^{1,\infty}((0,1)^d)}$ . In this section, we bound generalization error in the  $H^1((0,1)^d)$  sense:
Theorem 5. For any  $N, L, d, B, C_1, C_2$ , if  $\phi(\boldsymbol{x}; \boldsymbol{\theta}_D), \phi(\boldsymbol{x}; \boldsymbol{\theta}_S) \in \tilde{\Phi}$ , we will have that there are constants  $C_5 = C_5(B, d, C_1, C_2)$  and  $J = J(d, N, L, C_1, C_2)$  such that for any  $M \geq J$ , we have
$$
\begin{array}{l} \mathbf {E} \mathcal {R} _ {S} (\boldsymbol {\theta} _ {D}) - \mathcal {R} _ {D} (\boldsymbol {\theta} _ {D}) + \mathbf {E} \mathcal {R} _ {D} (\boldsymbol {\theta} _ {S}) - \mathbf {E} \mathcal {R} _ {S} (\boldsymbol {\theta} _ {S}) \leq 2 \sup _ {\boldsymbol {\theta}, \phi (\boldsymbol {x}; \boldsymbol {\theta}) \in \widetilde {\Phi}} | \mathbf {E} (\mathcal {R} _ {S} (\boldsymbol {\theta})) - \mathcal {R} _ {D} (\boldsymbol {\theta}) | \\ \leq C _ {5} \frac {N L \left(\log_ {2} L \log_ {2} N\right) ^ {\frac {1}{2}}}{\sqrt {M}} \log M. \tag {13} \\ \end{array}
$$
where  $\widetilde{\Phi} := \{\phi : \phi$  with the width  $\leq C_1N\log N$  and depth  $\leq C_2L\log L, \| \phi \|_{W^{1,\infty}((0,1)^d)} \leq B\}$ , and  $\mathcal{R}_S, \mathcal{R}_D, \pmb{\theta}_S, \pmb{\theta}_D$  are defined in Eqs. (10,11).
The proof of Theorem 5 is based on the works of [3, 11, 26]. We begin by bounding the generalization error using the Rademacher Complexity and then bound the Rademacher Complexity by the uniform covering number. We further bound the uniform covering number by the pseudo-dimension. Finally, we estimate the pseudo-dimension by Theorem 2. The proof of Theorem 5 is presented in Appendix 7.5
Theorem 2 helps to control the degree of the generalization error with respect to  $N$  and  $L$  in Theorem 5. In [11], the generalization error is bounded by  $O(NL^{\frac{5}{2}})$ . In [24], the authors estimate the covering number using the Lipschitz condition of DNNs instead of the pseudo-dimension, leading to a generalization error that is exponentially dependent on the depth of the DNNs. Our result is much better than them due to the optimal estimation of pseudo-dimension of DNN derivatives (Theorem 2).
5 Proof Sketches for Theorems 1 and 2
As Theorems 1 and 2 address the estimation of VC-dimension and pseudo-dimension of DNN derivatives, which is the main contribution of this paper, we provide the proofs for these theorems in this section. The distinction in our approach compared to that of [4] lies in the fact that the application of the chain rule requires the consideration of correlations among distinct segments of the deep neural networks, as opposed to treating them as independent components multiplied together.
In the proof of Theorem 1, we use the following lemmas:
Lemma 1 ([4, Lemma 17],[3, Theorem 8.3]). Suppose  $W \leq M$  and let  $P_{1},\ldots ,P_{M}$  be polynomials of degree at most  $D$  in  $W$  variables. Define  $K \coloneqq \left|\{\left(\operatorname{sgn}(P_1(a)),\dots,\operatorname{sgn}(P_M(a))\right):a\in \mathbb{R}^W\}\right|$ , then we have  $K \leq 2(2eMD / W)^W$ .
Lemma 2 ([4, Lemma 18]). Suppose that  $2^m \leq 2^t (mr / w)^w$  for some  $r \geq 16$  and  $m \geq w \geq t \geq 0$ . Then,  $m \leq t + w\log_2(2r\log_2r)$ .
As the proof of Theorem 1 represents the most critical and challenging question in our work, we present a sketch of it below.
Proof Sketch of Theorem 1. An element in  $\Phi$  can be represented as  $\phi = W_{L + 1}\sigma_1(W_L\sigma_1(\ldots \sigma_1(W_1x + b_1)\ldots) + b_L) + b_{L + 1}$ . Therefore, an element in  $D\Phi$  can be represented as
$$
\begin{array}{l} \psi (\boldsymbol {x}) = D _ {i} \phi (\boldsymbol {x}) = \boldsymbol {W} _ {L + 1} \sigma_ {0} \left(\boldsymbol {W} _ {L} \sigma_ {1} \left(\dots \sigma_ {1} \left(\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}\right) \dots\right) + \boldsymbol {b} _ {L}\right) \\ \cdot \boldsymbol {W} _ {L} \sigma_ {0} (\dots \sigma_ {1} (\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}) \dots) \dots \boldsymbol {W} _ {2} \sigma_ {0} (\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}) (\boldsymbol {W} _ {1}) _ {i}, \tag {14} \\ \end{array}
$$
where  $\pmb{W}_i\in \mathbb{R}^{N_i\times N_{i - 1}}((\pmb {W})_i$  is  $i$ -th column of  $\pmb{W}$ ) and  $\pmb{b}_{i}\in \mathbb{R}^{N_{i}}$  are the weight matrix and the bias vector in the  $i$ -th linear transform in  $\phi$ , and  $\sigma_0(x) = \mathrm{sgn}(x) = 1[x > 0]$ , which is the derivative of the ReLU function and  $\sigma_0(\pmb {x}) = \mathrm{diag}(\sigma_0(x_i))$ .
Let  $\pmb{x} \in \mathbb{R}^d$  be an input and  $\pmb{\theta} \in \mathbb{R}^{W}$  be a parameter vector in  $\psi$ . We denote the output of  $\psi$  with input  $\pmb{x}$  and parameter vector  $\pmb{\theta}$  as  $f(\pmb{x}, \pmb{\theta})$ . For fixed  $\pmb{x}_1, \pmb{x}_2, \dots, \pmb{x}_m$  in  $\mathbb{R}^d$ , we aim to bound
$$
K := \left| \right.\left\{ \right.\left( \right.\operatorname {s g n} \left(f \left(\boldsymbol {x} _ {1}, \boldsymbol {\theta}\right)\right), \dots , \operatorname {s g n} \left(f \left(\boldsymbol {x} _ {m}, \boldsymbol {\theta}\right)\right): \boldsymbol {\theta} \in \mathbb {R} ^ {W} \left. \right\}\left. \right|. \tag {15}
$$
The proof is inspired by [4, Theorem 7]. For any partition  $\mathcal{S} = \{P_1, P_2, \ldots, P_T\}$  of the parameter domain  $\mathbb{R}^W$ , we have  $K \leq \sum_{i=1}^{T} |\{(\operatorname{sgn}(f(\boldsymbol{x}_1, \boldsymbol{\theta})), \ldots, \operatorname{sgn}(f(\boldsymbol{x}_m, \boldsymbol{\theta})): \boldsymbol{\theta} \in P_i\}|$ . We choose the partition such that within each region  $P_i$ , the functions  $f(\boldsymbol{x}_j, \cdot)$  are all fixed polynomials of bounded degree. This allows us to bound each term in the sum using Lemma 1.
The partition of  $\mathbb{R}^W$  is constructed layer by layer through successive refinements denoted by  $S_0, S_1, \ldots, S_L$ . These refinements possess the following properties:
1. We have  $|\mathcal{S}_0| = 1$ , and for each  $n = 1, \ldots, L$ , we have  $\frac{|\mathcal{S}_n|}{|\mathcal{S}_{n-1}|} \leq 2 \left( \frac{2 \text{emn} N_k}{\sum_{i=1}^n W_i} \right)^{\sum_{i=1}^n W_i}$ .
2. For each  $n = 0, \dots, L - 1$ , each element  $S$  of  $\mathcal{S}_n$ , when  $\pmb{\theta}$  varies in  $S$ , the output of each term in  $\mathbb{F}_n$  is a fixed polynomial function in  $\sum_{i=1}^{n} W_i$  variables of  $\pmb{\theta}$ , with a total degree no more than  $n + 1$ .
3. For each element  $S$  of  $S_{L}$ , when  $\theta$  varies in  $S$ , the  $h$ -th term in  $\mathbb{F}_L$  for  $h \in \{1, 2, \dots, L + 1\}$  is a fixed polynomial function in  $W_{h}$  variables of  $\theta$ , with a total degree no more than 1. The sequence of sets of functions  $\{\mathbb{F}_j\}_{j=0}^L$  with respect to parameters  $\theta \in \mathbb{R}^{W}$  is defined as:
$$
\begin{array}{l} \mathbb {F} _ {0} := \left\{\left(\boldsymbol {W} _ {1}\right) _ {i}, \boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1} \right\} \\ \mathbb {F} _ {1} := \left\{\left(\boldsymbol {W} _ {1}\right) _ {i}, \boldsymbol {W} _ {2} \sigma_ {0} \left(\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}\right), \boldsymbol {W} _ {2} \sigma_ {1} \left(\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}\right) + \boldsymbol {b} _ {2} \right\} \\ \mathbb {F} _ {2} := \left\{\left(\boldsymbol {W} _ {1}\right) _ {i}, \boldsymbol {W} _ {2} \sigma_ {0} \left(\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}\right), \boldsymbol {W} _ {3} \sigma_ {0} \left(\boldsymbol {W} _ {2} \sigma_ {1} \left(\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}\right) + \boldsymbol {b} _ {2}\right), \right. \\ \left. \boldsymbol {W} _ {3} \sigma_ {1} \left(\boldsymbol {W} _ {2} \sigma_ {1} \left(\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}\right) + \boldsymbol {b} _ {2}\right) + \boldsymbol {b} _ {3} \right\} \\ \end{array}
$$
$$
\mathbb {F} _ {L} := \left\{\left(\boldsymbol {W} _ {1}\right) _ {i}, \boldsymbol {W} _ {2} \sigma_ {0} \left(\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}\right), \dots , \boldsymbol {W} _ {L + 1} \sigma_ {0} \left(\boldsymbol {W} _ {L} \sigma_ {1} \left(\dots \sigma_ {1} \left(\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}\right) \dots\right) + \boldsymbol {b} _ {L}\right) \right\}. \tag {16}
$$
The details of the refinements required to obtain the partitions are presented in the appendix. By leveraging Property 3 and Lemma 1, we can deduce that  $K \leq 2^{L + 1}\left(\frac{2em(L + 2)(L + 1)N}{2U}\right)^U$  where  $U = O(N^2 L^2)$ ,  $N$  is the width of the network, and the last inequality is due to weighted AM-GM. For the definition of the VC-dimension, we have  $2^{\mathrm{VCdim}(D\Phi)} \leq 2^{L + 1}\left(\frac{e\mathrm{VCdim}(D\Phi)(L + 1)(L + 2)N}{U}\right)^U$ . Due to Lemma 2, we obtain that  $\mathrm{VCdim}(D\Phi) \leq L + 1 + U\log_2[2(L + 1)(L + 2)\log_2(L + 1)(L + 2)] = O(N^2 L^2\log_2L\log_2N)$  since  $U = O(N^2 L^2)$ .
Note that the VC-dimension estimation achieved in Theorem 1 is nearly optimal, as demonstrated in Corollary 3 combined with the upper bound in Theorem 1. If the polynomial degree in the VC-dimension bound as a function of  $N$  and  $L$  were any smaller, it would contradict Theorem 3, which is based on our proof of Theorem 4.
Corollary 3. For any  $d \in \mathbb{N}_+$ ,  $C, J_0, \varepsilon > 0$ , there exists  $N, L \in \mathbb{N}$  with  $NL \geq J_0$  such that
$$
V C d i m (D \Phi) > C N ^ {2 - \varepsilon} L ^ {2 - \varepsilon}, \tag {17}
$$
We discuss the proof of Corollary 3 at the end of Section 7.4. Next we now present the proof for Theorem 2.
The proof of Theorem 2 can be found in the appendix. The key idea behind the proof is to reformulate the pseudo-dimension of DNN derivatives as the VC-dimension of a larger set. Based on the definition of pseudo-dimensions, it follows that  $\mathrm{Pdim}(D\Phi) \geq \mathrm{VCdim}(D\Phi)$ . Consequently, we can derive the following corollary, utilizing Corollary 3, to demonstrate the near optimality of the pseudo-dimension estimate presented in Theorem 2:
Corollary 4. For any  $d \in \mathbb{N}_{+}$ ,  $C, J_{0}, \varepsilon > 0$ , there exists  $N, L \in \mathbb{N}$  with  $NL \geq J_{0}$  such that
$$
P d i m (D \Phi) > C N ^ {2 - \varepsilon} L ^ {2 - \varepsilon}, \tag {18}
$$
In this paper, we establish nearly optimal bounds for the VC-dimension and pseudo-dimension of DNN derivatives. Based on these bounds, two contributions to Sobolev training [8, 40, 46] are made in
this paper. Firstly, we show that the optimal approximation rate of DNNs with a width of  $O(N \log N)$  and a depth of  $O(L \log L)$  is  $O(N^{-2(n-1)/d} L^{-2(n-1)/d})$  in Sobolev spaces. This demonstrates the ability of DNNs to learn target functions well in Sobolev training. Secondly, we find that the degree of the pseudo-dimension of DNN derivatives is the same as that for DNNs corresponding to the width  $N$  and depth  $L$  of DNNs. This result suggests that despite the apparent complexity of DNN derivatives, the degree of generalization error of loss functions containing derivatives of DNNs is equivalent to that without derivatives, corresponding to the width  $N$  and depth  $L$  of DNNs. As a result, we do not need to use a significantly larger number of sample points to learn the target function in Sobolev training compared to regular training.
The estimations of the VC-dimension and pseudo-dimension of DNN derivatives have broad applications in deep learning research. For example, in classification tasks, the VC-dimension characterizes the uniform convergence of misclassification frequencies to probabilities and asymptotically determines the sample complexity of PAC learning [4, 44, 6]. These applications can be explored in the further work. Our focus in this paper is on the Sobolev training with loss functions containing first-order derivatives, and we also obtain the approximation rate of  $\sigma_{2}$ -NNs described by higher-order Sobolev norms (Corollaries 1 and 2). The optimality of these results and the generalization error of Sobolev training with loss functions containing higher-order derivatives of DNNs remain open problems, as estimating the VC-dimension and pseudo-dimension of higher-order derivatives of  $\sigma_{2}$ -NNs requires further investigation.
This work was done during Y.Y.'s visit under the supervision of Prof. H.Y., in the Department of Mathematics, University of Maryland College Park. The work of H. Y. was partially supported by the US National Science Foundation under award DMS-2244988, DMS-2206333, and the Office of Naval Research Award N00014-23-1-2007. The work of Y.X. was supported by the Project of Hetao Shenzhen-HKUST Innovation Cooperation Zone HZQB-KCZYB-2020083.
[1] Y. Abu-Mostafa. The Vapnik-Chervonenkis dimension: Information versus complexity in learning. Neural Computation, 1(3):312-317, 1989.
[2] J. Adler and S. Lunz. Banach wasserstein Gan. Advances in neural information processing systems, 31, 2018.
[3] M. Anthony, P. Bartlett, et al. Neural network learning: Theoretical foundations, volume 9. Cambridge university press Cambridge, 1999.
[4] P. Bartlett, N. Harvey, C. Liaw, and A. Mehrabian. Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, 20(1):2285-2301, 2019.
[5] P. Bartlett, V. Maiorov, and R. Meir. Almost linear VC dimension bounds for piecewise polynomial networks. Advances in neural information processing systems, 11, 1998.
[6] A. Blumer, A. Ehrenfeucht, D. Haussler, and M. Warmuth. Learnability and the Vapnik-Chervonenkis dimension. Journal of the ACM (JACM), 36(4):929-965, 1989.
[7] S. Brenner, L. Scott, and L. Scott. The mathematical theory of finite element methods, volume 3. Springer, 2008.
[8] W. Czarnecki, S. Osindero, M. Jaderberg, G. Swirszcz, and R. Pascanu. Sobolev training for neural networks. Advances in neural information processing systems, 30, 2017.
[9] T. De Ryck, S. Lanthaler, and S. Mishra. On the approximation of functions by tanh neural networks. Neural Networks, 143:732-750, 2021.
[10] T. De Ryck and S. Mishra. Error analysis for physics-informed neural networks (PINNs) approximating Kolmogorov PDEs. Advances in Computational Mathematics, 48(6):1-40, 2022.