SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yahong YANG
The Pennsylvania State University, University Park,
State College, PA, USA
Clear Water Bay, Hong Kong SAR, China
yxy5498@psu.edu
Haizhao YANG*
Department of Mathematics and Department of Computer Science
University of Maryland College Park
College Park, MD, USA
hzyang@umd.edu
Yang XIANG
Algorithms of Machine Learning and Autonomous Driving Research Lab
Futian, Shenzhen, China
maxiang@ust.hk
This paper addresses the problem of nearly optimal Vapnik-Chervonenkis dimension (VC-dimension) and pseudo-dimension estimations of the derivative functions of deep neural networks (DNNs). Two important applications of these estimations include: 1) Establishing a nearly tight approximation result of DNNs in the Sobolev space; 2) Characterizing the generalization error of machine learning methods with loss functions involving function derivatives. This theoretical investigation fills the gap of learning error estimations for a wide range of physics-informed machine learning models and applications including generative models, solving partial differential equations, operator learning, network compression, distillation, regularization, etc.
The Sobolev training [8, 40, 46, 45, 22, 42] of deep neural networks (DNNs) has had a significant impact on scientific and engineering fields, including solving partial differential equations [25, 12,
33, 10], operator learning [29, 26], network compression [35], distillation [21, 34], regularization [8], and dynamic programming [15, 48], etc. For example, Sobolev (semi) norms have been applied to penalize function gradients in loss functions [2, 17, 15, 30] to control the Lipschitz constant of DNNs. Moreover, Sobolev norms and equivalent formulas are commonly used to define loss functions in various applications such as dynamic programming [15, 48], solving partial differential equations [25, 12, 33], and distillation [21, 34, 35]. These loss functions enable models to learn DNNs that can approximate the target function with small discrepancies in both magnitude and derivative. For example, when utilizing the Deep Ritz method [12] to solve PDEs such as the following one:
$$
\left\{ \begin{array}{l l} - \Delta u = f & \text {i n} \Omega , \\ \frac {\partial u}{\partial \nu} = 0 & \text {o n} \partial \Omega , \end{array} \right. \tag {1}
$$
the corresponding loss function can be expressed as:
$$
\mathcal {E} _ {D} (\boldsymbol {\theta}) := \frac {1}{2} \int_ {\Omega} | \nabla \phi (\boldsymbol {x}; \boldsymbol {\theta}) | ^ {2} \mathrm {d} \boldsymbol {x} + \frac {1}{2} \left(\int_ {\Omega} \phi (\boldsymbol {x}; \boldsymbol {\theta}) \mathrm {d} \boldsymbol {x}\right) ^ {2} - \int_ {\Omega} f \phi (\boldsymbol {x}; \boldsymbol {\theta}) \mathrm {d} \boldsymbol {x},
$$
where  $\pmb{\theta}$  represents all the parameters in the neural network. Here,  $\Omega$  denotes the domain  $(0,1)^{d}$ . Proposition 1 in [27] establishes the equivalence between the loss function  $\mathcal{E}_D(\pmb{\theta})$  and  $\| \phi (\pmb {x};\pmb {\theta}) - u^{*}(\pmb {x})\|_{H^{1}((0,1)^{d})}$ , where  $u^{*}(\pmb {x})$  denotes the exact solution of the PDEs in equation (1), and  $\| f\|_{H^1 ((0,1)^d)}\coloneqq \left(\sum_{0\leq |\alpha |\leq 1}\| D^\alpha f\|_{L^2 ((0,1)^d)}^p\right)^{1 / 2}$ . Thus, the Sobolev norm  $H^{1}((0,1)^{d})$  serves as a measure of the loss function, and Sobolev training is employed to solve PDEs within the Deep Ritz method.
Two natural questions that arise are: 1) What is the optimal approximation error of DNNs described by a Sobolev norm? 2) What is the generalization error of the loss function defined by a Sobolev norm? The key step to address these questions is to estimate the optimal Vapnik-Chervonenkis dimension (VC-dimension) and pseudo-dimension [3, 46, 1, 32] of DNNs and their derivatives. Intuitively, these concepts characterize the complexity or richness of a function set and, hence, they can be applied to establish the best possible approximation and generalization power of DNNs.
Definition 1 (VC-dimension [1]). Let  $H$  denote a class of functions from  $\mathcal{X}$  to  $\{0,1\}$ . For any non-negative integer  $m$ , define the growth function of  $H$  as
$$
\Pi_ {H} (m) := \max  _ {x _ {1}, x _ {2}, \dots , x _ {m} \in \mathcal {X}} | \left\{\left(h \left(x _ {1}\right), h \left(x _ {2}\right), \dots , h \left(x _ {m}\right)\right): h \in H \right\} |.
$$
The Vapnik-Chervonenkis dimension (VC-dimension) of  $H$ , denoted by  $\operatorname{VCdim}(H)$ , is the largest  $m$  such that  $\Pi_H(m) = 2^m$ . For a class  $\mathcal{G}$  of real-valued functions, define  $\operatorname{VCdim}(\mathcal{G}) \coloneqq \operatorname{VCdim}(\operatorname{sgn}(\mathcal{G}))$ , where  $\operatorname{sgn}(\mathcal{G}) \coloneqq \{\operatorname{sgn}(f) : f \in \mathcal{G}\}$  and  $\operatorname{sgn}(x) = 1 [x > 0]$ .
Definition 2 (pseudo-dimension [32]). Let  $\mathcal{F}$  be a class of functions from  $\mathcal{X}$  to  $\mathbb{R}$ . The pseudo-dimension of  $\mathcal{F}$ , denoted by  $Pdim(\mathcal{F})$ , is the largest integer  $m$  for which there exists  $(x_{1}, x_{2}, \ldots, x_{m}, y_{1}, y_{2}, \ldots, y_{m}) \in \mathcal{X}^{m} \times \mathbb{R}^{m}$  such that for any  $(b_{1}, \ldots, b_{m}) \in \{0, 1\}^{m}$  there is  $f \in \mathcal{F}$  such that  $\forall i: f(x_{i}) > y_{i} \Longleftrightarrow b_{i} = 1$ .
The main contribution of this paper is to estimate nearly optimal bounds of the VC-dimension and pseudo-dimension of DNN derivatives. Based on these bounds, we can prove the optimality of our DNN approximation, as measured by Sobolev norms (Theorem 3), and obtain a tighter generalization error of loss functions defined by Sobolev norms. Our results facilitate the understanding of Sobolev training and the performance of DNNs in Sobolev spaces.
Bounds for the VC-dimension and pseudo-dimension of DNNs have been established in [16, 5, 3, 4, 6, 41, 20]. However, these approaches and findings cannot be applied to Sobolev training, as they do not account for the derivatives of DNNs, which represent a key difference between Sobolev training and classical methods. Obtaining such bounds for DNN derivatives is much more difficult due to their complex compositional structures. DNN derivatives consist of a series of interdependent parts that are multiplied together via the chain rule, rendering existing methods for estimating bounds inapplicable. Estimating the VC-dimension and pseudo-dimension of DNN derivatives is the most crucial and challenging problem addressed in this paper. In [11], the VC-dimension and pseudo-dimension of DNN derivatives were analyzed, but the results were suboptimal due to a lack of consideration for the relationships between the multiplied terms in a DNN derivative. As a result, their findings do not provide the optimal approximation of DNNs in Sobolev spaces and can only give a generalization
error that is much larger than the actual error that may arise from Sobolev training. In this paper, we introduce a novel method that investigates these relationships, resulting in a simplified complexity of DNN derivatives. This, in turn, allows us to obtain nearly optimal bounds on their VC-dimension and pseudo-dimension.
The paper is divided into two parts. In the first part, we establish a nearly optimal bound on the VC-dimension of DNN derivatives with the ReLU activation function  $\sigma_{1}(x) := \max \{0, x\}$ :
Theorem 1. For any  $N, L, d \in \mathbb{N}_{+}$ , there exists a constant  $\bar{C}$  independent with  $N, L$  such that
$$
V C d i m (D \Phi) \leq \bar {C} N ^ {2} L ^ {2} \log_ {2} L \log_ {2} N, \tag {2}
$$
$$
D \Phi := \left\{\psi = D _ {i} \phi : \phi \in \Phi , i = 1, 2, \dots , d \right\}, \tag {3}
$$
where  $\Phi := \{\phi : \phi$  is a  $\sigma_{1}$ -NN in  $\mathbb{R}^d$  with width  $\leq N$  and depth  $\leq L\}$ , and  $D_{i}$  is the weak derivative in the  $i$ -th variable.
By utilizing Theorem 1, we prove that our DNN approximation rate for approximating functions in Sobolev spaces  $W^{n,\infty}((0,1)^d)$  using Sobolev norms in  $W^{1,\infty}((0,1)^d)$  is nearly optimal. We present our construction of DNNs for this approximation in Theorem 3, and we demonstrate the optimality of such approximation in Theorem 4. Furthermore, we generalize our method to approximate DNNs in Sobolev spaces measured by Sobolev norms  $W^{m,\infty}((0,1)^d)$  for  $m \geq 2$ . The details of this generalization are presented in Corollaries 1 and 2. The Sobolev spaces, equipped with Sobolev (semi) norms, are defined as follows:
Definition 3 (Sobolev Spaces [13]). Denote  $\Omega$  as  $(0,1)^{d}$ ,  $D$  as the weak derivative of a single variable function and  $D^{\alpha} = D_{1}^{\alpha_{1}}D_{2}^{\alpha_{2}}\ldots D_{d}^{\alpha_{d}}$  as the partial derivative where  $\alpha = [\alpha_{1},\alpha_{2},\dots,\alpha_{d}]^{T}$  and  $D_{i}$  is the derivative in the  $i$ -th variable. Let  $n\in \mathbb{N}$  and  $1\leq p\leq \infty$ . Then we define Sobolev spaces
$$
W ^ {n, p} (\Omega) := \left\{f \in L ^ {p} (\Omega): D ^ {\alpha} f \in L ^ {p} (\Omega) \text {f o r a l l} \alpha \in \mathbb {N} ^ {d} \text {w i t h} | \alpha | \leq n \right\}
$$
with a norm  $\| f\|_{W^{n,p}(\Omega)}\coloneqq \left(\sum_{0\leq |\alpha |\leq n}\| D^{\alpha}f\|_{L^p (\Omega)}^p\right)^{1 / p}$ , if  $p < \infty$ , and  $\| f\|_{W^{n,\infty}(\Omega)}\coloneqq \max_{0\leq |\alpha |\leq n}\| D^{\alpha}f\|_{L^{\infty}(\Omega)}$ . Furthermore, for  $\mathbf{f} = (f_1,f_2,\ldots ,f_d)$ ,  $\mathbf{f}\in W^{1,\infty}(\Omega ,\mathbb{R}^d)$  if and only if  $f_{i}\in W^{1,\infty}(\Omega)$  for each  $i = 1,2,\dots,d$  and  $\| f\|_{W^{1,\infty}(\Omega ,\mathbb{R}^d)}\coloneqq \max_{i = 1,\dots,d}\{\| f_i\|_{W^{1,\infty}(\Omega)}\}$ .
Definition 4 (Sobolev semi-norm [13]). Let  $n \in \mathbb{N}_+$  and  $1 \leq p \leq \infty$ . Then we define Sobolev semi-norm  $|f|_{W^{n,p}(\Omega)} := \left( \sum_{|\alpha| = n} \| D^\alpha f \|_{L^p(\Omega)}^p \right)^{1/p}$ , if  $p < \infty$ , and  $|f|_{W^{n,\infty}(\Omega)} := \max_{|\alpha| = n} \| D^\alpha f \|_{L^\infty(\Omega)}$ . Furthermore, for  $\boldsymbol{f} \in W^{1,\infty}(\Omega, \mathbb{R}^d)$ , we define  $|f|_{W^{1,\infty}(\Omega, \mathbb{R}^d)} := \max_{i=1,\ldots,d} \{|f_i|_{W^{1,\infty}(\Omega)}\}$ .
In the second part of our paper, we utilize our previous work on estimating the VC-dimension of DNN derivatives to obtain an upper bound on the pseudo-dimension of DNN derivatives:
Theorem 2. For any  $N, L, d \in \mathbb{N}_{+}$ , there exists a constant  $\widehat{C}$  independent with  $N, L$  such that
$$
P d i m (D \Phi) \leq \widehat {C} N ^ {2} L ^ {2} \log_ {2} L \log_ {2} N, \tag {4}
$$
where  $D\Phi$  is defined in Theorem 1.
Based on Theorem 2, we can estimate the generalization error of loss functions defined by Sobolev norms, as demonstrated in Theorem 5. Specifically, the error is bounded by  $O(NL(\log_2N\log_2L)^{1 / 2})$  with respect to the width  $N$  and depth  $L$  of DNNs. This bound is significantly smaller than the previously reported bound of  $O(NL^{5 / 2}(\log_2N\log_2L)^{1 / 2})$  in [11]. We attribute this improvement to our more accurate estimation of the pseudo-dimension of DNN derivatives. Our findings indicate that learning target functions with loss functions defined by Sobolev norms does not require substantially more sample points than those defined by  $L^2$ -norms [14], as their generalization error orders are equivalent with respect to the width  $N$  and depth  $L$  of DNNs.
- We propose a method to achieve nearly optimal estimations of the VC-dimension and pseudodimension of DNN derivatives.
- By utilizing our estimation of the VC-dimension of DNN derivatives, we demonstrate the optimality of our DNN approximation, as measured by Sobolev norms.
- By applying our estimation of the pseudo-dimension of DNN derivatives, we obtain a bound for the generalization error measured by the Sobolev norm. Importantly, our results demonstrate that the degree of generalization error defined by Sobolev norms is equivalent to that defined by  $L^2$ -norms, corresponding to the width  $N$  and depth  $L$  of DNNs.
Let us summarize all basic notations used in the DNNs as follows:
1. Matrices are denoted by bold uppercase letters. For example,  $\mathbf{A} \in \mathbb{R}^{m \times n}$  is a real matrix of size  $m \times n$  and  $\mathbf{A}^{\intercal}$  denotes the transpose of  $\mathbf{A}$ .
2. Vectors are denoted by bold lowercase letters. For example,  $\pmb{v} \in \mathbb{R}^n$  is a column vector of size  $n$ . Furthermore, denote  $\pmb{v}(i)$  as the  $i$ -th elements of  $\pmb{v}$ .
3. For a  $d$ -dimensional multi-index  $\alpha = [\alpha_{1}, \alpha_{2}, \dots, \alpha_{d}] \in \mathbb{N}^{d}$ , we denote several related notations as follows: (a)  $|\alpha| = |\alpha_{1}| + |\alpha_{2}| + \dots + |\alpha_{d}|$ ; (b)  $\pmb{x}^{\alpha} = x_{1}^{\alpha_{1}}x_{2}^{\alpha_{2}}\dots x_{d}^{\alpha_{d}}$ ,  $\pmb{x} = [x_{1}, x_{2}, \dots, x_{d}]^{\intercal}$ ; (c)  $\pmb{\alpha}! = \alpha_{1}!\alpha_{2}!\dots \alpha_{d}!$ .
4. Let  $B_{r,|\cdot|}(\pmb{x}) \subset \mathbb{R}^d$  be the closed ball with a center  $\pmb{x} \in \mathbb{R}^d$  and a radius  $r$  measured by the Euclidean distance. Similarly,  $B_{r,\| \cdot \|_{\ell_\infty}}(\pmb{x}) \subset \mathbb{R}^d$  be the closed ball with a center  $\pmb{x} \in \mathbb{R}^d$  and a radius  $r$  measured by the  $\ell_\infty$ -norm.
5. Assume  $\pmb{n} \in \mathbb{N}_{+}^{n}$ , then  $f(\pmb{n}) = O(g(\pmb{n}))$  means that there exists positive  $C$  independent of  $\pmb{n}$ ,  $f, g$  such that  $f(\pmb{n}) \leq C g(\pmb{n})$  when all entries of  $\pmb{n}$  go to  $+\infty$ .
6. Define  $\sigma_1(x) \coloneqq \sigma(x) = \max\{0, x\}$  and  $\sigma_2 \coloneqq \sigma^2(x)$ . We call the neural networks with activation function  $\sigma_t$  with  $t \leq i$  as  $\sigma_i$  neural networks ( $\sigma_i$ -NNs). With the abuse of notations, we
$$
\text {d e f i n e} \sigma_ {i}: \mathbb {R} ^ {d} \to \mathbb {R} ^ {d} \text {a s} \sigma_ {i} (\boldsymbol {x}) = \left[ \begin{array}{c} \sigma_ {i} (x _ {1}) \\ \vdots \\ \sigma_ {i} (x _ {d}) \end{array} \right] \text {f o r a n y} \boldsymbol {x} = [ x _ {1}, \dots , x _ {d} ] ^ {T} \in \mathbb {R} ^ {d}.
$$
7. Define  $L, N \in \mathbb{N}_{+}, N_{0} = d$  and  $N_{L + 1} = 1$ ,  $N_{i} \in \mathbb{N}_{+}$  for  $i = 1,2,\ldots,L$ , then a  $\sigma_{i}$ -NN  $\phi$  with the width  $N$  and depth  $L$  can be described as follows:
$$
\boldsymbol {x} = \tilde {\boldsymbol {h}} _ {0} \stackrel {{W _ {1}, b _ {1}}} {{\longrightarrow}} \boldsymbol {h} _ {1} \stackrel {{\sigma_ {i}}} {{\longrightarrow}} \tilde {\boldsymbol {h}} _ {1} \dots \stackrel {{W _ {L}, b _ {L}}} {{\longrightarrow}} \boldsymbol {h} _ {L} \stackrel {{\sigma_ {i}}} {{\longrightarrow}} \tilde {\boldsymbol {h}} _ {L} \stackrel {{W _ {L + 1}, b _ {L + 1}}} {{\longrightarrow}} \phi (\boldsymbol {x}) = \boldsymbol {h} _ {L + 1},
$$
where  $\pmb{W}_i \in \mathbb{R}^{N_i \times N_{i-1}}$  and  $\pmb{b}_i \in \mathbb{R}^{N_i}$  are the weight matrix and the bias vector in the  $i$ -th linear transform in  $\phi$ , respectively, i.e.,  $\pmb{h}_i := \pmb{W}_i \tilde{\pmb{h}}_{i-1} + \pmb{b}_i$ , for  $i = 1, \dots, L+1$  and  $\tilde{\pmb{h}}_i = \sigma_i(\pmb{h}_i)$ , for  $i = 1, \dots, L$ . In this paper, an DNN with the width  $N$  and depth  $L$ , means (a) The maximum width of this DNN for all hidden layers less than or equal to  $N$ . (b) The number of hidden layers of this DNN less than or equal to  $L$ .
3 Nearly Optimal Approximation Results of DNNs in Sobolev Spaces Measured by Sobolev Norms
3.1 Approximation of functions in  $W^{n,\infty}$  with  $W^{1,\infty}$  norm by ReLU neural networks
In this subsection, we construct deep neural networks (DNNs) with a width of  $O(N \log N)$  and a depth of  $O(L \log L)$  to approximate functions in the Sobolev space  $W^{n,\infty}$ , as measured by Sobolev norms in  $W^{1,\infty}$ . The approximation rate achieved by these networks is  $O(N^{-2(n-1)/d} L^{-2(n-1)/d})$ .
Theorem 3. For any  $f \in W^{n,\infty}((0,1)^d)$  with  $n \geq 2$  and  $\| f \|_{W^{n,\infty}((0,1)^d)} \leq 1$ , any  $N, L \in \mathbb{N}_+$ , there is a  $\sigma_1$ -NN  $\phi$  with the width  $(34 + d)2^d n^{d + 1}(N + 1)\log_2(8N)$  and depth  $56d^{2}n^{2}(L + 1)\log_2(4L)$  such that
$$
\left\| f (\boldsymbol {x}) - \phi (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \leq C _ {9} (n, d) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d},
$$
where  $C_9$  is the constant independent with  $N, L$ .
The proof of Theorem 3 can be outlined in five parts, and the complete proof is provided in Appendix 7.2:
(i): First of all, define a sequence of subsets of  $\Omega$ :
Definition 5. Given  $K, d \in \mathbb{N}^{+}$ , and for any  $\pmb{m} = (m_{1}, m_{2}, \dots, m_{d}) \in \{1, 2\}^{d}$ , we define  $\Omega_{\pmb{m}} := \prod_{j=1}^{d} \Omega_{m_j}$ , where  $\Omega_1 := \bigcup_{i=0}^{K-1} \left[ \frac{i}{K}, \frac{i}{K} + \frac{3}{4K} \right]$ ,  $\Omega_2 := \bigcup_{i=0}^{K} \left[ \frac{i}{K} - \frac{1}{2K}, \frac{i}{K} + \frac{1}{4K} \right] \cap [0,1]$ .
Then we define a partition of unity  $\{g_{\pmb{m}}\}_{\pmb{m}\in \{1,2\}^d}$  on  $(0,1)^{d}$  with  $\operatorname {supp}g_{\pmb{m}}\cap (0,1)^{d}\subset \Omega_{\pmb{m}}$  for each  $\pmb {m}\in \{1,2\} ^d$  ..
Definition 6. Given  $K, d \in \mathbb{N}_{+}$ , we define
$$
g _ {1} (x) := \left\{ \begin{array}{l l} 1, & x \in \left[ \frac {i}{K} + \frac {1}{4 K}, \frac {i}{K} + \frac {1}{2 K} \right] \\ 0, & x \in \left[ \frac {i}{K} + \frac {3}{4 K}, \frac {i + 1}{K} \right] \\ 4 K \left(x - \frac {i}{K}\right), & x \in \left[ \frac {i}{K}, \frac {i}{K} + \frac {1}{4 K} \right] \\ - 4 K \left(x - \frac {i}{K} - \frac {3}{4 K}\right), & x \in \left[ \frac {i}{K} + \frac {1}{2 K}, \frac {i}{K} + \frac {3}{4 K} \right] \end{array} , g _ {2} (x) := g _ {1} \left(x + \frac {1}{2 K}\right), \right. \tag {5}
$$
for  $i\in \mathbb{Z}$ . For any  $\pmb {m} = (m_{1},m_{2},\dots ,m_{d})\in \{1,2\}^{d}$ , define  $g_{\pmb{m}}(\pmb {x}) = \prod_{j = 1}^{d}g_{m_j}(x_j)$ ,  $\pmb {x} = (x_{1},x_{2},\ldots ,x_{d})$
Figure 1: The schematic diagram of  $g_{i}$  for  $i = 1,2$ .
(ii): Then we use the following proposition to approximate  $\{g_{\pmb{m}}\}_{\pmb{m}\in \{1,2\}^{d}}$  by  $\sigma_{1}$ -NNs and construct a sequence of  $\sigma_{1}$ -NNs  $\{\phi_m\}_{m\in \{1,2\}^d}$
Proposition 1. Given any  $N, L, n \in \mathbb{N}_+$  for  $K = \lfloor N^{1/d} \rfloor^2 \lfloor L^{2/d} \rfloor$ , then for any  $\boldsymbol{m} = (m_1, m_2, \ldots, m_d) \in \{1, 2\}^d$ , there is a  $\sigma_1$ -NN with the width smaller than  $(9 + d)(N + 1) + d - 1$  and depth smaller than  $15d(d - 1)nL$  such as  $\| \phi_{\boldsymbol{m}}(\boldsymbol{x}) - g_{\boldsymbol{m}}(\boldsymbol{x}) \|_{W^{1,\infty}((0,1)^d)} \leq 50d^{\frac{5}{2}}(N + 1)^{-4dnL}$ .
The proof of Proposition 1 is presented in Appendix 7.2.1.
(iii): For each  $\Omega_{\pmb{m}} \subset [0,1]^d$ , where  $\pmb{m} \in \{1,2\}^d$ , we find a function  $f_{K,\pmb{m}}$  satisfying
$$
\left\| f - f _ {K, \boldsymbol {m}} \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {1} (n, d) K ^ {- (n - 1)},
$$
$$
\left\| f - f _ {K, \boldsymbol {m}} \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {1} (n, d) K ^ {- n}, \tag {6}
$$
where  $C_1$  is a constant independent of  $K$ . Moreover, each  $f_{K,m}$  can be expressed as  $f_{K,m} = \sum_{|\alpha| \leq n-1} g_{f,\alpha,m}(\boldsymbol{x}) \boldsymbol{x}^{\alpha}$ , where  $g_{f,\alpha,m}(\boldsymbol{x})$  is a piecewise constant function on  $\Omega_m$ . The proof of this result is based on the Bramble-Hilbert Lemma [7, Lemma 4.3.8], and the details are provided in Appendix 7.2.2.
(iv): The fourth step involves approximating  $f_{K,m}$  using neural networks  $\psi_{m}$ , following the approach outlined in [28]. This method is suitable for our work because  $g_{f,\alpha,m}(\pmb{x})$  is a piecewise constant function on  $\Omega_{m}$ , and the weak derivative of  $g_{f,\alpha,m}(\pmb{x})$  on  $\Omega_{m}$  is zero. This property allows for the use of the  $L^{\infty}$  norm approximation method presented in [28]. Thus, we obtain a neural network  $\psi_{m}$  with width  $O(N\log N)$  and depth  $O(L\log L)$  such that
$$
\left\| f _ {K, \boldsymbol {m}} - \psi_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {5} (n, d) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d}
$$
$$
\left\| f _ {K, \boldsymbol {m}} - \psi_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {5} (n, d) N ^ {- 2 n / d} L ^ {- 2 n / d}, \tag {7}
$$
where  $C_5$  is a constant independent of  $N$  and  $L$ .
By combining (iii) and (iv) and setting  $K = \lfloor N^{1 / d}\rfloor^2\lfloor L^{2 / d}\rfloor$  , we obtain that for each  $\pmb {m}\in \{1,2\} ^d$  there exists a neural network  $\psi_{\pmb{m}}$  with width  $\bar{O} (N\log N)$  and depth  $O(L\log L)$  such that
$$
\begin{array}{l} \left\| f (\boldsymbol {x}) - \psi_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {6} (n, d) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d} \\ \left\| f (\boldsymbol {x}) - \psi_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {6} (n, d) N ^ {- 2 n / d} L ^ {- 2 n / d}, \tag {8} \\ \end{array}
$$
where  $C_6$  is a constant independent of  $N$  and  $L$ . Further details are provided in Appendix 7.2.3.
(v): The final step is to combine the sequences  $\{\phi_m\}_{m\in \{1,2\}^d}$  and  $\{\psi_{m}\}_{m\in \{1,2\}^{d}}$  to construct a network that can approximate  $f$  over the entire space  $[0,1]^d$ . We define the sequence  $\{\phi_m\}_{m\in \{1,2\}^d}$  because  $\psi_{m}$  may not accurately approximate  $f$  on  $[0,1]^d\backslash \Omega_m$ . The purpose of  $\phi_{m}$  is to remove this portion of the domain and allow other networks to approximate  $f$  on  $[0,1]^d\backslash \Omega_m$ . Further details on this step are provided in Appendix 7.2.4.
While recent works [28, 23, 39, 18, 31, 9, 19] have studied the approximation of smooth functions or functions in Sobolev spaces by DNNs measured in the norm of  $L^p(\Omega)$  or  $W^{s,p}(\Omega)$ , they typically present results that are not optimal or are measured in  $L^p$ -norms. For example, in [28], they apply Taylor's expansion to approximate smooth functions but cannot be applied directly in Sobolev spaces. In [39], they improve on this by using the Bramble-Hilbert Lemma to approximate functions in Sobolev spaces, but their error is still measured in  $L^p$ -norms. In [18], the authors show that there exists a ReLU neural network that can approximate  $f \in W^{1,p}(\Omega)$ , but their approximation rate is not optimal and is the same as that in traditional methods such as the finite element theory. Our work provides a superior approximation rate. Later, a rigorous proof of optimality of Theorem 3 is discussed in Appendix 7.2.4 and Subsection 3.3.
3.2 Approximation of functions in  $W^{n,\infty}$  measured by  $W^{m,\infty}$  norm with  $m > 1$  by neural networks (sketches of the proofs of the Corollaries 1 and 2)