SlowGuess/NeurIPS_2023_OCR huggingface dataset:
is a  $\sigma_{2}$  neural network with  $12(\lfloor N^{1 / d}\rfloor)$  the width and two layers.
Similar with Lemma 1, we know that
$$
\hat {g} = \widetilde {g} \circ \psi_ {2} \circ \psi_ {3} \circ \psi_ {4} (x)
$$
is a  $\sigma_{2}$  neural network with  $12\bigl (\lfloor N^{1 / d}\rfloor \bigr)$  the width and  $5 + 2\lfloor L^{1 / d}\rfloor$  , and
$$
s _ {1} (x) = \hat {g} \left(x + \frac {1}{8 K}\right), s _ {2} (x) = s _ {1} \left(x + \frac {1}{2 K}\right), x \in [ 0, 1 ]. \tag {98}
$$
Based on (v) in Lemma 10, we have there is a  $\sigma_{2}$  neural network  $\lambda_{m}(\pmb{x})$  with the width  $16N + 2d$  and depth  $4L + 5$  such as
7.3.2 Proof of Corollaries 1 and 2
The proof is comprised of three parts, which include Theorem 8 and 9, followed by the combination of these results. Theorem 8 is to apply the Bramble-Hilbert Lemma 9 measured in the norm of  $W^{2,\infty}$ :
Theorem 8. Let  $K \in \mathbb{N}_{+}$  and  $n \geq 2$ . Then for any  $f \in W^{n,\infty}((0,1)^d)$  with  $\| f \|_{W^{n,\infty}((0,1)^d)} \leq 1$  and  $m \in \{1,2\}^d$ , there exist piece-wise polynomials function  $f_{K,m} = \sum_{|\alpha| \leq n-1} g_{f,\alpha,m}(x) x^{\alpha}$  on  $\Omega_m$  (Definition 5) with the following properties:
$$
\left\| f - f _ {K, \boldsymbol {m}} \right\| _ {W ^ {2, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {1} (n, d) K ^ {- (n - 2)},
$$
$$
\| f - f _ {K, \boldsymbol {m}} \| _ {W ^ {1, \infty} (\Omega_ {\boldsymbol {m}})} \leq C _ {1} (n, d) K ^ {- (n - 1)},
$$
$$
\left\| f - f _ {K, \boldsymbol {m}} \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {1} (n, d) K ^ {- n}. \tag {99}
$$
$$
\left| g _ {f, \alpha , \boldsymbol {m}} (\boldsymbol {x}) \right| \leq C _ {2} (n, d) \tag {100}
$$
The proof is the same as that of Theorem 6. Note that  $\{f_{K,\pmb{m}}\}_{\pmb{m}\in \{1,2\}^d}$  will be same in two theorems if  $f\in W^{n,\infty}((0,1)^d)$  in two theorem are same.
Theorem 9 is to establish  $\sigma_{2}$  neural networks  $\{\gamma_{m}\}_{\{1,2\}^{d}}$ , and each  $\gamma_{m}$  can approximate  $f$  well on  $\Omega_{m}$ .
Theorem 9. For any  $f \in W^{n,\infty}((0,1)^d)$  with  $\| f\|_{W^{n,\infty}((0,1)^d)} \leq 1$ , any  $N,L \in \mathbb{N}_{+}$  with  $NL + 2^{\lfloor \log_2N\rfloor} \geq n$  and  $L \geq \lceil \log_2N\rceil$ , and  $m = (m_{1},m_{2},\ldots ,m_{d}) \in \{1,2\}^{d}$ , there is a  $\sigma_{2}$  neural network  $\gamma_{m}$  with the width  $28n^{d + 1}(N + d)\log_2(8N)$  and depth  $11n^{2}(L + 2)\log_{2}(4L)$  such that
$$
\left\| f (\boldsymbol {x}) - \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {W ^ {2, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {1 0} (n, d) N ^ {- 2 (n - 2) / d} L ^ {- 2 (n - 2) / d}
$$
$$
\left\| f (\boldsymbol {x}) - \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {1 0} (n, d) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d}
$$
$$
\left\| f (\boldsymbol {x}) - \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {1 0} (n, d) N ^ {- 2 n / d} L ^ {- 2 n / d}, \tag {101}
$$
Proof. The proof is similar to that of Theorem 7; the difference is that  $xy$  and  $x^{\alpha}$  can be architected precisely by  $\sigma_{2}$  neural networks.
Without loss of the generalization, we consider the case for  $m_* = (1, 1, \ldots, 1)$ . Due to Theorem 8 and setting  $K = \lfloor N^{1/d} \rfloor^2 \lfloor L^{2/d} \rfloor$ , we have
$$
\| f - f _ {K, \boldsymbol {m} _ {*}} \| _ {W ^ {2, \infty} (\Omega_ {\boldsymbol {m} _ {*}})} \leq C _ {1} (n, d) K ^ {- (n - 2)} \leq C _ {1} (n, d) N ^ {- 2 (n - 2) / d} L ^ {- 2 (n - 2) / d}
$$
$$
\left\| f - f _ {K, \boldsymbol {m} _ {*}} \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \leq C _ {1} (n, d) K ^ {- n} \leq C _ {1} (n, d) N ^ {- 2 n / d} L ^ {- 2 n / d}, \tag {102}
$$
where  $f_{K,\pmb{m}_{*}} = \sum_{|\pmb{\alpha}| \leq n-1} g_{f,\pmb{\alpha},\pmb{m}_{*}}(\pmb{x}) \pmb{x}^{\pmb{\alpha}}$  for  $x \in \Omega_{\pmb{m}_{*}}$ . Note that  $g_{f,\pmb{\alpha},\pmb{m}_{*}}(\pmb{x})$  is a constant function for  $\pmb{x} \in \prod_{j=1}^{d} \left[ \frac{i_j}{K}, \frac{3+4i_j}{4K} \right]$  and  $\pmb{i} = (i_1, i_2, \dots, i_d) \in \{0, 1, \dots, K-1\}^d$ . The remaining part is to approximate  $f_{K,\pmb{m}_{*}}$  by neural networks.
The way to approximate  $g_{f,\pmb{\alpha},\pmb{m}_{*}}(\pmb{x})$  is same with Theorem 7, and we have that
$$
\begin{array}{l} \left\| \phi_ {\boldsymbol {\alpha}} \left(\phi_ {2} (\boldsymbol {x})\right) - g _ {f, \boldsymbol {\alpha}, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {W ^ {2, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} = \left\| \phi_ {\boldsymbol {\alpha}} \left(\phi_ {2} (\boldsymbol {x})\right) - g _ {f, \boldsymbol {\alpha}, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \\ = \left\| \phi_ {\boldsymbol {\alpha}} \left(\phi_ {2} (\boldsymbol {x})\right) - g _ {f, \boldsymbol {\alpha}, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \\ \leq 2 C _ {2} (n, d) N ^ {- 2 n} L ^ {- 2 n} \tag {103} \\ \end{array}
$$
Due to (v) in Lemma 10, there is a  $\sigma_{2}$  neural network  $\phi_{5,\alpha}(\pmb{x})$  with the width  $4N + 2d$  and depth  $L + \lceil \log_2N\rceil$  such that
$$
\phi_ {5, \alpha} (\boldsymbol {x}) = \boldsymbol {x} ^ {\alpha}, \boldsymbol {x} \in \mathbb {R} ^ {d}. \tag {104}
$$
Due to (iv) in Lemma 10, there is a  $\sigma_{2}$  neural network  $\phi_6(x)$  with the width 4 and depth 1 such that
$$
\phi_ {6} (x, y) = x y, x, y \in \mathbb {R}. \tag {105}
$$
Now we define the neural network  $\gamma_{\pmb{m}_{*}}(\pmb{x})$  to approximate  $f_{K,\pmb{m}_{*}}(\pmb{x})$  in  $\Omega_{\pmb{m}_{*}}$ :
$$
\gamma_ {\boldsymbol {m} _ {*}} (\boldsymbol {x}) = \sum_ {| \boldsymbol {\alpha} | \leq n - 1} \phi_ {6} \left[ \phi_ {\boldsymbol {\alpha}} \left(\phi_ {2} (\boldsymbol {x})\right), \phi_ {5, \boldsymbol {\alpha}} (\boldsymbol {x}) \right]. \tag {106}
$$
$$
\begin{array}{l} \widetilde {\mathcal {E}} := \left\| \sum_ {| \boldsymbol {\alpha} | \leq n - 1} \phi_ {6} \left[ \phi_ {\boldsymbol {\alpha}} \left(\phi_ {2} (\boldsymbol {x})\right), \phi_ {5, \boldsymbol {\alpha}} (\boldsymbol {x}) \right] - f _ {K, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {W ^ {2, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \\ \leq \sum_ {| \boldsymbol {\alpha} | \leq n - 1} \| \phi_ {6} [ \phi_ {\boldsymbol {\alpha}} (\phi_ {2} (\boldsymbol {x})), \phi_ {5, \boldsymbol {\alpha}} (\boldsymbol {x}) ] - g _ {f, \boldsymbol {\alpha}, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \boldsymbol {x} ^ {\boldsymbol {\alpha}} \| _ {W ^ {2, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \\ = \sum_ {| \boldsymbol {\alpha} | \leq n - 1} \| \phi_ {\boldsymbol {\alpha}} (\phi_ {2} (\boldsymbol {x})) \boldsymbol {x} ^ {\boldsymbol {\alpha}} - g _ {f, \boldsymbol {\alpha}, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \boldsymbol {x} ^ {\boldsymbol {\alpha}} \| _ {W ^ {2, \infty} \left(\Omega_ {m _ {*}}\right)} \\ \leq n ^ {2} \sum_ {| \boldsymbol {\alpha} | \leq n - 1} \| \phi_ {\boldsymbol {\alpha}} (\phi_ {2} (\boldsymbol {x})) - g _ {f, \boldsymbol {\alpha}, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \| _ {W ^ {2, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \\ \leq 2 n ^ {d + 2} C _ {2} (n, d) N ^ {- 2 n} L ^ {- 2 n}. \tag {107} \\ \end{array}
$$
At last, we finish the proof by estimating the network's the width and depth, implementing  $\gamma_{m_*(x)}$ . From Eq. (106), we know that  $\gamma_{m_*(x)}$  consists of the following subnetworks:
1.  $\phi_{5,\alpha}(\pmb {x})$  with the width  $4N + 2d$  and depth  $L + \lceil \log_2N\rceil$
2.  $\phi_2(x)$  with the width  $4N + 5$  and depth  $4L + 4$ .
3.  $\phi_{\alpha}$  with the width  $16n(N + 1)\log_2(8N)$  and depth  $(5L + 2)\log_2(4L)$ .
4.  $\phi_6(x,y)$  with the width 4 and depth 1.
Therefore  $\phi (\pmb {x})$  is a neural network with the width  $28n^{d + 1}(N + d)\log_2(8N)$  and depth  $11n^{2}(L+$  2)  $\log_2(4L)$
Combining Eqs. (102) and (107), we have that there is a neural network  $\gamma_{m_*}$  with the width  $28n^{d + 1}(N + d)\log_2(8N)$  and depth  $11n^2 (L + 2)\log_2(4L)$  such that
$$
\left\| f (\boldsymbol {x}) - \psi_ {\boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {W ^ {2, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \leq C _ {1 0} (n, d) N ^ {- 2 (n - 2) / d} L ^ {- 2 (n - 2) / d}
$$
$$
\left\| f (\boldsymbol {x}) - \psi_ {\boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \leq C _ {1 0} (n, d) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d}
$$
$$
\left\| f (\boldsymbol {x}) - \psi_ {\boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \leq C _ {1 0} (n, d) N ^ {- 2 n / d} L ^ {- 2 n / d}, \tag {108}
$$
where  $C_{10} = C_1 + 2n^{d + 2}C_2$  is the constant independent with  $N,L$ .
Similarly, we can construct a neural network  $\gamma_{m}$  with the width  $28n^{d + 1}(N + d)\log_2(8N)$  and depth  $11n^{2}(L + 2)\log_{2}(4L)$  which can approximate  $f$  on  $\Omega_{m}$  with same order of Eq. (108).
The last part is to combine  $\{\lambda_{m}\}_{m\in \{1,2\}^{d}}$  and  $\{\gamma_{m}\}_{m\in \{1,2\}^{d}}$  in  $[0,1]^{d}$  and obtain a  $\sigma_{2}$  neural network to approximate  $f$  measured in the norm of  $W^2$ .
Proof of Corollary 1. Based on Theorem 9, there is a sequence of the neural network  $\{\gamma_{\pmb{m}}(\pmb{x})\}_{\pmb{m} \in \{1,2\}^d}$  such that
$$
\left\| f (\boldsymbol {x}) - \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {1 0} (n, d) N ^ {- 2 n / d} L ^ {- 2 n / d}, \tag {109}
$$
where  $C_{10}$  is the constant independent with  $N, L$ , and each  $\gamma_{m}$  is a neural network with the width  $28n^{d + 1}(N + d)\log_2(8N)$  and depth  $11n^2 (L + 2)\log_2(4L)$ . According to Proposition 8, there is a sequence of the neural network  $\{s_m(\pmb {x})\}_{\pmb {m}\in \{1,2\} ^d}$  satisfies:
(i):  $\| s_{\pmb{m}}(\pmb{x})\|_{L^{\infty}((0,1)^d)}\leq 1,\| s_{\pmb{m}}(\pmb{x})\|_{W^{1,\infty}((0,1)^d)}\leq 8K$  and  $\| s_{\pmb{m}}(\pmb {x})\|_{W^{1,\infty}((0,1)^d)}\leq 64K^2$  for any  $\pmb {m}\in \{1,2\} ^d$
(ii):  $\{s_{\pmb{m}}(\pmb{x})\}_{\pmb{m} \in \{1,2\}^d}$  is a partition of the unity  $[0,1]^d$  with  $\operatorname{supp} s_{\pmb{m}}(\pmb{x}) \cap [0,1]^d = \Omega_{\pmb{m}}$  defined in Definition 5.
For each  $s_m$ , it is a  $\sigma_2$  neural network with the width  $16N + 2d$  and depth  $4L + 5$ .
$$
\phi_ {6} (x, y) = x y, x, y \in \mathbb {R}. \tag {110}
$$
$$
\gamma (\boldsymbol {x}) = \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \phi_ {6} \left(s _ {\boldsymbol {m}} (\boldsymbol {x}), \gamma_ {\boldsymbol {m}} (\boldsymbol {x})\right). \tag {111}
$$
$$
\begin{array}{l} \widetilde {\mathcal {R}} := \| f (\boldsymbol {x}) - \gamma (\boldsymbol {x}) \| _ {W ^ {2, \infty} ((0, 1) ^ {d})} \leq \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \| s _ {\boldsymbol {m}} (x) \cdot f (\boldsymbol {x}) - s _ {\boldsymbol {m}} (\boldsymbol {x}) \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \| _ {W ^ {2, \infty} ((0, 1) ^ {d})} \\ = \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \| s _ {\boldsymbol {m}} (x) \cdot f (\boldsymbol {x}) - s _ {\boldsymbol {m}} (\boldsymbol {x}) \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \| _ {W ^ {2, \infty} \left(\Omega_ {\boldsymbol {m}}\right)}. \tag {112} \\ \end{array}
$$
where the last equality is due to  $\operatorname{supp} s_{\boldsymbol{m}}(\boldsymbol{x}) \cap [0,1]^d = \Omega_{\boldsymbol{m}}$ .
Then due to chain rule, for each  $m \in \{1, 2\}^d$ , we have
$$
\begin{array}{l} \left\| s _ {\boldsymbol {m}} (x) \cdot f (\boldsymbol {x}) - s _ {\boldsymbol {m}} (\boldsymbol {x}) \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {W ^ {2, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \\ \leq \left\| s _ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {W ^ {2, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \left\| f (\boldsymbol {x}) - \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} + 2 \left\| s _ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \left\| f (\boldsymbol {x}) - \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \\ + \| s _ {\boldsymbol {m}} (x) \| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \| f (\boldsymbol {x}) - \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \| _ {W ^ {2, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} + \| s _ {\boldsymbol {m}} (x) \| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \| f (\boldsymbol {x}) - \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \\ + \left\| s _ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \left\| f (\boldsymbol {x}) - \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} + \left\| s _ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \left\| f (\boldsymbol {x}) - \gamma_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \\ \leq 9 1 C _ {1 0} (n, d) N ^ {- 2 (n - 2) / d} L ^ {- 2 (n - 2) / d}. \tag {113} \\ \end{array}
$$
$$
\widetilde {\mathcal {R}} \leq 2 ^ {d + 7} C _ {1 0} (n, d) N ^ {- 2 (n - 2) / d} L ^ {- 2 (n - 2) / d}.
$$
At last, we finish the proof by estimating the network's width and depth, implementing  $\gamma (\pmb {x})$ . From Eq. (111), we know that  $\gamma (\pmb {x})$  consists of the following subnetworks:
1.  $\gamma_{m}(\pmb{x})$  with the width  $28n^{d + 1}(N + d)\log_2(8N)$  and depth  $11n^{2}(L + 2)\log_{2}(4L)$ .
2.  $s_m(x)$  with the width  $16N + 2d$  and depth  $4L + 5$ .
3.  $\phi_6(x,y)$  with the width 4 and depth 1.
Therefore  $\gamma (\pmb {x})$  is a neural network with the width  $2^{d + 6}n^{d + 1}(N + d)\log_2(8N)$  and depth  $15n^{2}(L+$  2)  $\log_2(4L)$
Our method can easily extend to approximations measured by the norm of  $W^{m,\infty}$ . The primary difference in the proof lies in the need to establish a differential  $\{s_m(\pmb{x})\}_{\{1,2\}^d}$ , which can be achieved by constructing iterated  $s_m(\pmb{x})$  as piece-wise  $m$ -degree polynomial functions. By extending this approach, we can obtain Corollary 2 using our method.
7.4 Proof of Theorem 4
Proof. The Theorem 4 will be proved by contradiction. The idea of the proof is inspired by Ref. [28].
Claim 1. There exist  $\rho, C_1, C_2, C_3, J_0 > 0$  and  $s, d \in \mathbb{N}^+$  such that, for any  $f \in \mathcal{F}_{n,d}$ , we have
$$
\inf  _ {\phi \in \widehat {\Phi}} | \phi - f | _ {W ^ {1, \infty} ((0, 1) ^ {d})} \leq C _ {3} L ^ {- 2 (n - 1) / d - \rho} N ^ {- 2 (n - 1) / d - \rho}. \tag {114}
$$
for all  $NL \geq J_0$ , where
$$
\widehat {\Phi} := \left\{\phi : R e L U F N N s \phi w i t h \text {w i d t h} \leq C _ {1} N \log N \text {a n d} \text {d e p t h} \leq C _ {2} L \log L \right\}.
$$
The remaining question is to show Claim 1 is invalid.
$$
D \widehat {\Phi} := \left\{\psi : \psi = D _ {i} \phi , \phi \in \widehat {\Phi}, i = 1, \dots , d \right\},
$$
Due to Theorem 1, we obtain
$$
\operatorname {V C D i m} \left(D \widehat {\Phi}\right) \leq C _ {4} N ^ {2} L ^ {2} \log_ {2} L \log_ {2} N =: b _ {u}. \tag {115}
$$
Now we will use Claim 1 to estimate a lower bound
$$
b _ {l} := \left\lfloor (N L) ^ {\frac {2}{d} + \frac {\rho}{2 (n - 1)}} \right\rfloor^ {d}
$$
of  $\operatorname{VCDim}(D\widehat{\Phi})$ . In other words, we will construct  $\{\psi_{\beta}(\pmb{x}):\psi_{\beta}(\pmb{x})\in D\widehat{\Phi},\beta \in \mathcal{B}\}$  to scatter  $b_{l}$  points.  $\mathcal{B}$  will be defined later.
First, fix  $i = 1,\ldots ,d$ , and there exists  $\widetilde{g}\in C^{\infty}(0,1)^{d}$  such that  $\frac{\partial\widetilde{g}(\mathbf{0})}{\partial x_i} = 1$  and  $\widetilde{g} (\pmb {x}) = 0$  for  $\| \pmb {x}\| _2\geq 1 / 3$ . And we can find a constant  $C_5 > 0$  such that  $g\coloneqq \widetilde{g} /C_5\in \mathcal{F}_{n,d}$ .
Denote  $M = \left\lfloor (NL)^{\frac{2}{d} + \frac{\rho}{2(n - 1)}} \right\rfloor$ . Divide  $[0,1]^d$  into  $M^d$  non-overlapping sub-cubes  $\{Q_{\theta}\}_{\theta}$  as follows:
$$
Q _ {\boldsymbol {\theta}} := \left\{\boldsymbol {x} = \left[ x _ {1}, x _ {2}, \dots , x _ {d} \right] ^ {T} \in [ 0, 1 ] ^ {d}: x _ {i} \in \left[ \frac {\theta_ {i} - 1}{M}, \frac {\theta_ {i}}{M} \right], i = 1, 2, \dots , d \right\},
$$
for any index vector  $\pmb{\theta} = [\theta_1, \theta_2, \dots, \theta_d]^T \in \{1, 2, \dots, M\}^d$ . Denote the center of  $Q_{\pmb{\theta}}$  by  $\pmb{x}_{\pmb{\theta}}$  for all  $\pmb{\theta} \in \{1, 2, \dots, M\}^d$ . Define
$$
\mathcal {B} := \left\{\beta : \beta \text {i s a m a p f r o m} \{1, 2, \dots , M \} ^ {d} \text {t o} \{- 1, 1 \} \right\}.
$$
For each  $\beta \in \mathcal{B}$ , we define, for any  $\pmb{x} \in \mathbb{R}^d$ ,
$$
h _ {\beta} (\boldsymbol {x}) := \sum_ {\boldsymbol {\theta} \in \{1, 2, \dots , M \} ^ {d}} M ^ {- n} \beta (\boldsymbol {\theta}) g _ {\boldsymbol {\theta}} (\boldsymbol {x}), \quad \text {w h e r e} g _ {\boldsymbol {\theta}} (\boldsymbol {x}) = g \left(M \cdot \left(\boldsymbol {x} - \boldsymbol {x} _ {\boldsymbol {\theta}}\right)\right).
$$
Due to  $|\operatorname{supp}\widetilde{g}(\boldsymbol{x})| \leq \frac{2}{3}$  and  $|D^{\alpha}h_{\beta}(\boldsymbol{x})| \leq M^{-n + |\boldsymbol{\alpha}|}\| g\|_{W^{n,\infty}} \leq 1$ , we obtain that
$$
\left| D ^ {\boldsymbol {\alpha}} f _ {\beta} (\boldsymbol {x}) \right| \leq 1
$$
for any  $|\alpha| \leq n$  Therefore,  $f_{\beta} \in \mathcal{F}_{n,d}$ . And it is easy to check  $\{D_i h_{\beta} = h_{\beta} : \beta \in \mathcal{B}\}$  can scatters  $b_l$  points since  $\frac{\partial \widetilde{g}(\mathbf{0})}{\partial x_i} = 1$  and  $\widetilde{g}(\boldsymbol{x}) = 0$  for  $\| \boldsymbol{x} \|_2 \geq 1/3$ .
Note that for any  $h_{\beta} \in \mathcal{F}_{n,d}$ , there is a  $\phi_{\beta} \in \widehat{\Phi}$  such that  $C_3(NL)^{\frac{-2(n - 1)}{d} - \frac{\rho}{2}} \geq |D_i h_{\beta}(\pmb{x}_{\theta}) - D_i \phi_{\beta}(\pmb{x}_{\theta})|$  for any  $J_{\beta} \leq NL$  due to Claim 1. Denote  $J_{1} = \max_{\beta \in \mathcal{B}} \{J_{\beta}\}$ . There is a constant  $J_{2}$  such that  $\frac{M^{-n + 1}}{C_5} \geq C_3(NL)^{\frac{-2(n - 1)}{d} - \rho}$  for  $J_{2} \leq NL$ . Define  $J := \max \{J_1, J_2\}$ , then for any  $J \leq NL$ , we have
$$
\left| D _ {i} h _ {\beta} \left(\boldsymbol {x} _ {\boldsymbol {\theta}}\right) \right| = \left| M ^ {- n + 1} \frac {\partial g \left(\boldsymbol {x} _ {\boldsymbol {\theta}}\right)}{\partial x _ {i}} \right| = \frac {M ^ {- n + 1}}{C _ {5}} \geq C _ {3} (N L) ^ {\frac {- 2 (n - 1)}{d} - \rho} \geq \left| D _ {i} h _ {\beta} \left(\boldsymbol {x} _ {\boldsymbol {\theta}}\right) - D _ {i} \phi_ {\beta} \left(\boldsymbol {x} _ {\boldsymbol {\theta}}\right) \right|. \tag {116}
$$
In other words, for any  $\beta \in \mathcal{B}$  and  $\pmb{\theta} \in \{1, 2, \dots, M\}^d$ ,  $D_i f_{\beta}(\pmb{x}_{\theta})$  and  $D_i \phi_{\beta}(\pmb{x}_{\theta})$  have the same sign. Then  $\{D_i \phi_{\beta} : \beta \in \mathcal{B}\}$  shatters  $\{\pmb{x}_{\theta} : \pmb{\theta} \in \{1, 2, \dots, M\}^d\}$  since  $\{D_i h_{\beta} : \beta \in \mathcal{B}\}$  shatters  $\{\pmb{x}_{\theta} : \pmb{\theta} \in \{1, 2, \dots, M\}^d\}$  as discussed above. Hence,
$$
\operatorname {V C D i m} \left(\left\{\phi_ {\beta}: \beta \in \mathcal {B} \right\}\right) \geq M ^ {d} = b _ {l}, \tag {117}
$$
for  $N, L \in \mathbb{N}$  with  $NL \geq J$ .
By Eqs. (115,117), for any  $N,L\in \mathbb{N}$  with  $NL\geq J$  , we have  $b_{l}\leq \mathrm{VCDim}(\{\phi_{\beta}:\beta \in \mathcal{B}\})\leq$ $\operatorname {VCDim}(D\widehat{\Phi})\leq b_u$  , implying that
$$
\left\lfloor (N L) ^ {\frac {2}{d} + \frac {\rho}{2 (n - 1)}} \right\rfloor^ {d} \leq C _ {4} N ^ {2} L ^ {2} \log_ {2} L \log_ {2} N \tag {118}
$$
which is a contradiction for sufficiently large  $N, L \in \mathbb{N}$ . So we finish the proof of Theorem 4.
Based on the proof of Theorem 4, we can easily check that the estimation of VC-dimension of DNN derivatives (Theorem 1) is nearly optimal and prove Corollary 3. Assume  $\mathrm{VCDim}(D\widehat{\Phi})\leq b_u = O(N^{2 - \varepsilon}L^{2 - \varepsilon})$  in Eq. (118) for  $\varepsilon >0$ , and  $b_{l}$  must be larger than  $\lfloor (NL)^{\frac{2}{d}}\rfloor^d$  according the construction in the proof of Theorem 4 and Theorem 3. Hence we still obtain a contradiction in Eq. (118), and the estimation in Theorem 1 is nearly optimal.
7.5 Proof of Theorem 5
7.5.1 Bounding generalization error by Rademacher complexity
Definition 13 (Rademacher complexity [3]). Given a sample set  $S = \{z_{1}, z_{2}, \ldots, z_{M}\}$  on a domain  $\mathcal{Z}$ , and a class  $\mathcal{F}$  of real-valued functions defined on  $\mathcal{Z}$ , the empirical Rademacher complexity of  $\mathcal{F}$  in  $S$  is defined as
$$
\mathbf {R} _ {S} (\mathcal {F}) := \frac {1}{M} \mathbf {E} _ {\Sigma_ {M}} \left[ \sup _ {f \in \mathcal {F}} \sum_ {i = 1} ^ {M} \sigma_ {i} f (z _ {i}) \right],
$$
where  $\Sigma_M := \{\sigma_1, \sigma_2, \ldots, \sigma_M\}$  are independent random variables drawn from the Rademacher distribution, i.e.,  $\mathbf{P}(\sigma_i = +1) = \mathbf{P}(\sigma_i = -1) = \frac{1}{2}$  for  $i = 1, 2, \ldots, M$ . For simplicity, if  $S = \{z_1, z_2, \ldots, z_M\}$  is an independent random variable set with the uniform distribution, denote
$$
\mathbf {R} _ {M} (\mathcal {F}) := \mathbf {E} _ {S} \mathbf {R} _ {S} (\mathcal {F}).
$$
The following lemma will be used to bounded generalization error by Rademacher complexities:
Lemma 11 ([47], Proposition 4.11). Let  $\mathcal{F}$  be a set of functions. Then
$$
\mathbf {E} _ {X} \sup  _ {u \in \mathcal {F}} \left| \frac {1}{M} \sum_ {i = 1} ^ {M} u (x _ {j}) - \mathbf {E} _ {x \sim \mathcal {P} _ {\Omega}} u (x) \right| \leq 2 \mathbf {R} _ {M} (\mathcal {F}),
$$
where  $X\coloneqq \{x_{1},\ldots ,x_{M}\}$  is an independent random variable set with the uniform distribution.
Now we can show that generalization error can be bounded by Rademacher complexities of two function sets.
Lemma 12. Let  $d, N, L, M \in \mathbb{N}_+$ ,  $B, C_1, C_2 \in \mathbb{R}_+$ . For any  $f \in W^{1,\infty}((0,1)^d)$  with  $\| f \|_{W^{1,\infty}((0,1)^d)} \leq 1$ , set
$$
\widetilde {\Phi} := \left\{\phi : \phi \text {w i t h} \text {w i d t h} \leq C _ {1} N \log N \text {a n d} \text {d e p t h} \leq C _ {2} L \log L, \| \phi \| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \leq B \right\}
$$
$$
D \widetilde {\Phi} := \{\psi : \psi = D _ {i} \phi , i = 1, \dots , d \}. \tag {119}
$$
$$
2\sup_{\boldsymbol {\theta},\phi (\boldsymbol {x};\boldsymbol {\theta})\in \widetilde{\boldsymbol{\Phi}}}\left|\mathbf{E}(\mathcal{R}_{S}(\boldsymbol {\theta})) - \mathcal{R}_{D}(\boldsymbol {\theta})\right|\leq 4(B + 1)(d\mathbf{R}_{M}(D\widetilde{\boldsymbol{\Phi}}) + \mathbf{R}_{M}(\widetilde{\boldsymbol{\Phi}})),
$$
where  $\mathbf{E}$  is expected responding to  $X$ , and  $X \coloneqq \{\pmb{x}_1, \dots, \pmb{x}_M\}$  is an independent random variables set uniformly distributed on  $(0,1)^d$ .
Proof. For any  $\phi (\pmb {x};\pmb {\theta})\in \widetilde{\Phi}$  , we have