SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \phi_ {\boldsymbol {\alpha}} \left(\phi_ {2} (\boldsymbol {x})\right) - g _ {f, \boldsymbol {\alpha}, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {m _ {*}}\right)} = \left\| \phi_ {\boldsymbol {\alpha}} \left(\phi_ {2} (\boldsymbol {x})\right) - g _ {f, \boldsymbol {\alpha}, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {m _ {*}}\right)} \\ \leq 2 C _ {2} (n, d) N ^ {- 2 n} L ^ {- 2 n} \tag {71} \\ \end{array}
$$
which is due to  $\phi_{\alpha}(\phi_2(x)) - g_{f,\alpha ,m_*}(x)$  is a step function, and the first order weak derivative is 0 in  $\Omega_{m_*}$ .
Due to Proposition 6, there is a neural network  $\phi_{3,\alpha}$  with the width  $9(N + 1) + n - 1$  and depth  $14n^{2}L$  such that  $\| \phi_{3,\alpha}\|_{W^{1,\infty}((0,1)^d)}\leq 18$  and
$$
\left\| \phi_ {3, \boldsymbol {\alpha}} (\boldsymbol {x}) - \boldsymbol {x} ^ {\boldsymbol {\alpha}} \right\| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \leq 1 0 n (N + 1) ^ {- 7 n L}. \tag {72}
$$
Due to Proposition 4, there is a neural network  $\phi_4$  with the width  $15(N + 1)$  and depth  $4n(L + 1)$  such that  $\| \phi_4\|_{W^{1,\infty}(-C_3,C_3)^2}\leq 12(C_2(n,d))^2$  and
$$
\left\| \phi_ {4} (x, y) - x y \right\| _ {W ^ {1, \infty} \left((- C _ {3}, C _ {3}) ^ {2}\right)} \leq 6 \left(C _ {2} (n, d)\right) ^ {2} (N + 1) ^ {- 2 n (L + 1)}. \tag {73}
$$
where  $C_3(n,d) = \max \{3C_2(n,d),18\}$ .
Now we define the neural network  $\phi_{\pmb{m}_{*}}(\pmb{x})$  to approximate  $f_{K,\pmb{m}_{*}}(\pmb{x})$  in  $\Omega_{\pmb{m}_{*}}$ :
$$
\psi_ {\boldsymbol {m} _ {*}} (\boldsymbol {x}) = \sum_ {| \boldsymbol {\alpha} | \leq n - 1} \phi_ {4} \left[ \phi_ {\boldsymbol {\alpha}} \left(\phi_ {2} (\boldsymbol {x})\right), \phi_ {3, \boldsymbol {\alpha}} (\boldsymbol {x}) \right]. \tag {74}
$$
The remaining question is to find the error  $\mathcal{E}$ :
$$
\begin{array}{l} \mathcal {E} := \left\| \sum_ {| \boldsymbol {\alpha} | \leq n - 1} \phi_ {4} \left[ \phi_ {\boldsymbol {\alpha}} \left(\phi_ {2} (\boldsymbol {x})\right), \phi_ {3, \boldsymbol {\alpha}} (\boldsymbol {x}) \right] - f _ {K, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \\ \leq \sum_ {| \boldsymbol {\alpha} | \leq n - 1} \| \phi_ {4} [ \phi_ {\boldsymbol {\alpha}} (\phi_ {2} (\boldsymbol {x})), \phi_ {3, \boldsymbol {\alpha}} (\boldsymbol {x}) ] - g _ {f, \boldsymbol {\alpha}, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \boldsymbol {x} ^ {\boldsymbol {\alpha}} \| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \\ \leq \underbrace{\sum_{|\boldsymbol{\alpha}|\leq n - 1}\| \phi_{4}\left[\phi_{\boldsymbol{\alpha}}(\phi_{2}(\boldsymbol {x})),\phi_{3,\boldsymbol{\alpha}}(\boldsymbol {x})\right] - \phi_{\boldsymbol{\alpha}}(\phi_{2}(\boldsymbol {x}))\phi_{3,\boldsymbol{\alpha}}(\boldsymbol {x})\|_{W^{1,\infty}(\Omega_{m_{*}})}}_{=: \mathcal{E}_{1}} \\ + \underbrace {\sum_ {| \boldsymbol {\alpha} | \leq n - 1} \| \phi_ {\boldsymbol {\alpha}} (\phi_ {2} (\boldsymbol {x})) \phi_ {3 , \boldsymbol {\alpha}} (\boldsymbol {x}) - g _ {f , \boldsymbol {\alpha} , \boldsymbol {m} _ {*}} (\boldsymbol {x}) \phi_ {3 , \boldsymbol {\alpha}} (\boldsymbol {x}) \| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)}} _ {=: \mathcal {E} _ {2}} \\ + \underbrace {\sum_ {| \boldsymbol {\alpha} | \leq n - 1} \| g _ {f , \boldsymbol {\alpha} , \boldsymbol {m} _ {*}} (\boldsymbol {x}) \phi_ {3 , \boldsymbol {\alpha}} (\boldsymbol {x}) - g _ {f , \boldsymbol {\alpha} , \boldsymbol {m} _ {*}} (\boldsymbol {x}) \boldsymbol {x} ^ {\boldsymbol {\alpha}} \| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)}} _ {=: \mathcal {E} _ {3}}. \tag {75} \\ \end{array}
$$
As for  $\mathcal{E}_1$ , due to Lemma 3, we have
$$
\begin{array}{l} \mathcal {E} _ {1} \leq \sum_ {| \boldsymbol {\alpha} | \leq n - 1} 2 \sqrt {d} \max  \left\{\| \phi_ {4} (x, y) - x y \| _ {L ^ {\infty} \left((- C _ {3}, C _ {3}) ^ {2}\right)}, \| \phi_ {4} (x, y) - x y \| _ {W ^ {1, \infty} \left((- C _ {3}, C _ {3}) ^ {2}\right)} \right. \\ \left. \cdot \max  \left\{\| \phi_ {\boldsymbol {\alpha}} \left(\phi_ {2} (\boldsymbol {x})\right) \| _ {W ^ {1, \infty} \left(\Omega_ {m _ {*}}\right)}, \| \phi_ {3, \boldsymbol {\alpha}} (\boldsymbol {x}) \| _ {W ^ {1, \infty} \left(\Omega_ {m _ {*}}\right)} \right\} \right\} \\ \leq \sum_ {| \boldsymbol {\alpha} | \leq n - 1} 2 \sqrt {d} \max  \left\{\| \phi_ {4} (x, y) - x y \| _ {L ^ {\infty} \left((- C _ {3}, C _ {3}) ^ {2}\right)}, C _ {3} (n, d) \| \phi_ {4} (x, y) - x y \| _ {W ^ {1, \infty} \left((- C _ {3}, C _ {3}) ^ {2}\right)} \right\} \\ \leq \sum_ {| \alpha | \leq n - 1} 1 2 \sqrt {d} \left[ C _ {3} (n, d) + 1 \right] \left(C _ {2} (n, d)\right) ^ {2} (N + 1) ^ {- 2 n (L + 1)} \\ \leq C _ {4} (n, d) (N + 1) ^ {- 2 n (L + 1)} \tag {76} \\ \end{array}
$$
where  $C_4(n,d) = 12\sqrt{d} n^d [C_3(n,d) + 1](C_2(n,d))^2$
As for  $\mathcal{E}_2$ , due to Lemma 4, we have
$$
\begin{array}{l} \mathcal {E} _ {2} \leq \sum_ {| \boldsymbol {\alpha} | \leq n - 1} 2 \left\| \phi_ {\boldsymbol {\alpha}} \left(\phi_ {2} (\boldsymbol {x})\right) - g _ {f, \boldsymbol {\alpha}, \boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \cdot \left\| \phi_ {3, \boldsymbol {\alpha}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \\ \leq 7 2 n ^ {d} C _ {2} (n, d) N ^ {- 2 n} L ^ {- 2 n}. \tag {77} \\ \end{array}
$$
The estimation of  $\mathcal{E}_3$  is similar with that of  $\mathcal{E}_2$  which is
$$
\begin{array}{l} \mathcal {E} _ {3} \leq \sum_ {| \boldsymbol {\alpha} | \leq n - 1} \| g _ {f, \boldsymbol {\alpha}, \boldsymbol {m} _ {*}} \| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \cdot \| \phi_ {3, \boldsymbol {\alpha}} (\boldsymbol {x}) - \boldsymbol {x} ^ {\boldsymbol {\alpha}} \| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \\ \leq 1 0 n ^ {d} C _ {2} (n, d) n (N + 1) ^ {- 7 n L}. \tag {78} \\ \end{array}
$$
Therefore, using
$$
(N + 1) ^ {- 7 n L} \leq (N + 1) ^ {- 2 n (L + 1)} \leq N ^ {- 2 n} L ^ {- 2 n}
$$
the total error is
$$
\mathcal {E} \leq \mathcal {E} _ {1} + \mathcal {E} _ {2} + \mathcal {E} _ {3} \leq C _ {5} (n, d) K ^ {- 2 n} L ^ {- 2 n}, \tag {79}
$$
where  $C_5(n,d) = C_4(n,d) + 72n^d C_2(n,d) + 10n^d C_2(n,d)n$ .
At last, we finish the proof by estimating the network's width and depth, implementing  $\psi_{m_*(x)}$ . From Eq. (74), we know that  $\psi_{m_*}(\pmb{x})$  consists of the following subnetworks:
1.  $\phi_{3,\alpha}(\pmb {x})$  with the width  $9(N + 1) + n - 1$  and depth  $14n^{2}L$
2.  $\phi_2(x)$  with the width  $4N + 5$  and depth  $4L + 4$ .
3.  $\phi_{\alpha}$  with the width  $16n(N + 1)\log_2(8N)$  and depth  $(5L + 2)\log_2(4L)$ .
4.  $\phi_4(x,y)$  with the width  $15(N + 1)$  and depth  $4n(L + 1)$ .
Therefore  $\phi (\pmb {x})$  is a neural network with the width  $25n^{d + 1}(N + 1)\log_2(8N)$  and depth  $27n^{2}(L+$  2)  $\log_2(4L)$
Combining Eqs. (69) and (79), we have that there is a neural network  $\psi_{\pmb{m}_{*}}$  with the width  $25n^{d + 1}(N + 1)\log_2(8N)$  and depth  $27n^{2}(L + 2)\log_{2}(4L)$  such that
$$
\begin{array}{l} \| f (\boldsymbol {x}) - \psi_ {\boldsymbol {m} _ {*}} (\boldsymbol {x}) \| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \leq C _ {6} (n, d) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d} \\ \left\| f (\boldsymbol {x}) - \psi_ {\boldsymbol {m} _ {*}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m} _ {*}}\right)} \leq C _ {6} (n, d) N ^ {- 2 n / d} L ^ {- 2 n / d}, \tag {80} \\ \end{array}
$$
where  $C_6 = C_1 + C_5$  is the constant independent with  $N, L$ .
Similarly, we can construct a neural network  $\psi_{m}$  with the width  $25n^{d + 1}(N + 1)\log_2(8N)$  and depth  $27n^{2}(L + 2)\log_{2}(4L)$  which can approximate  $f$  on  $\Omega_{m}$  with same order of Eq. (80).
7.2.4 Proof of Theorem 3
Now we can prove Theorem 3 based on Theorem 7 and Proposition 1.
Proof of Theorem 3. Based on Theorem 7, there is a sequence of the neural network  $\{\psi_{\pmb{m}}(\pmb{x})\}_{\pmb{m} \in \{1,2\}^d}$  such that
$$
\begin{array}{l} \| f (\boldsymbol {x}) - \psi_ {\boldsymbol {m}} (\boldsymbol {x}) \| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {6} (n, d) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d} \\ \left\| f (\boldsymbol {x}) - \psi_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq C _ {6} (n, d) N ^ {- 2 n / d} L ^ {- 2 n / d}, \tag {81} \\ \end{array}
$$
where  $C_6 = C_1 + C_5$  is the constant independent with  $N, L$ , and each  $\psi_{\mathbf{m}}$  is a neural network with the width  $25n^{d+1}(N+1)\log_2(8N)$  and depth  $27n^2(L+2)\log_2(4L)$ . According to Proposition 1, there is a sequence of the neural network  $\{\phi_{\mathbf{m}}(\pmb{x})\}_{\pmb{m} \in \{1,2\}^d}$  such that
$$
\left\| \phi_ {\boldsymbol {m}} (\boldsymbol {x}) - g _ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left((0, 1) ^ {d}\right)} \leq 5 0 d ^ {\frac {5}{2}} (N + 1) ^ {- 4 d n L},
$$
where  $\{g_{\pmb{m}}\}_{\pmb{m}\in \{1,2\}^{d}}$  is defined in Definition 6 with  $\sum_{\pmb {m}\in \{1,2\}^d}g_{\pmb{m}}(\pmb {x}) = 1$  and  $\operatorname {supp}g_{\pmb{m}}\cap [0,1]^d = \Omega_m$ . For each  $\phi_{\pmb{m}}$ , it is a neural network with the width smaller than  $(9 + d)(N + 1) + d - 1$  and depth smaller than  $15d(d - 1)nL$ .
Due to Proposition 4, there is a neural network  $\widehat{\phi}$  with the width  $15(N + 1)$  and depth  $14n^{2}L$  such that  $\| \widehat{\phi}\|_{W^{1,\infty}(-C_7,C_7)^2}\leq 12(C_7(n,d))^2$  and
$$
\left\| \widehat {\phi} (x, y) - x y \right\| _ {W ^ {1, \infty} (- C _ {7}, C _ {7}) ^ {2}} \leq 6 \left(C _ {7}\right) ^ {2} (N + 1) ^ {- 7 n (L + 1)}, \tag {82}
$$
where  $C_7 = C_6 + 50d^{\frac{5}{2}} + 1$
Now we define
$$
\phi (\boldsymbol {x}) = \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \widehat {\phi} \left(\phi_ {\boldsymbol {m}} (\boldsymbol {x}), \psi_ {\boldsymbol {m}} (\boldsymbol {x})\right). \tag {83}
$$
$$
\begin{array}{l} \mathcal {R} := \| f (\boldsymbol {x}) - \phi (\boldsymbol {x}) \| _ {W ^ {1, \infty} ((0, 1) ^ {d})} = \left\| \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} g _ {\boldsymbol {m}} \cdot f (\boldsymbol {x}) - \phi (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \\ \leq \left\| \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \left[ g _ {\boldsymbol {m}} \cdot f (\boldsymbol {x}) - \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \cdot \psi_ {\boldsymbol {m}} (\boldsymbol {x}) \right] \right\| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \\ + \left\| \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \left[ \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \cdot \psi_ {\boldsymbol {m}} (\boldsymbol {x}) - \widehat {\phi} \left(\phi_ {\boldsymbol {m}} (\boldsymbol {x}), \psi_ {\boldsymbol {m}} (\boldsymbol {x})\right) \right] \right\| _ {W ^ {1, \infty} \left((0, 1) ^ {d}\right)}. \tag {84} \\ \end{array}
$$
As for the first part,
$$
\begin{array}{l} \left\| \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \left[ g _ {\boldsymbol {m}} \cdot f (\boldsymbol {x}) - \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \cdot \psi_ {\boldsymbol {m}} (\boldsymbol {x}) \right] \right\| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \\ \leq \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \| g _ {\boldsymbol {m}} \cdot f (\boldsymbol {x}) - \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \cdot \psi_ {\boldsymbol {m}} (\boldsymbol {x}) \| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \\ \leq \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \left[ \| (g _ {\boldsymbol {m}} - \phi_ {\boldsymbol {m}} (\boldsymbol {x})) \cdot f (\boldsymbol {x}) \| _ {W ^ {1, \infty} ((0, 1) ^ {d})} + \| (f _ {\boldsymbol {m}} - \psi_ {\boldsymbol {m}} (\boldsymbol {x})) \cdot \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \right] \\ = \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \left[ \| \left(g _ {\boldsymbol {m}} - \phi_ {\boldsymbol {m}} (\boldsymbol {x})\right) \cdot f (\boldsymbol {x}) \| _ {W ^ {1, \infty} \left((0, 1) ^ {d}\right)} + \| \left(f _ {\boldsymbol {m}} - \psi_ {\boldsymbol {m}} (\boldsymbol {x})\right) \cdot \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \right], \tag {85} \\ \end{array}
$$
where the last equality is due to Lemma 7. Based on Lemma 4 and  $\| f\|_{W^{1,\infty}((0,1)^d)}\leq 1$  , we have
$$
\left\| \left(g _ {\boldsymbol {m}} - \phi_ {\boldsymbol {m}} (\boldsymbol {x})\right) \cdot f (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left((0, 1) ^ {d}\right)} \leq \left\| \left(g _ {\boldsymbol {m}} - \phi_ {\boldsymbol {m}} (\boldsymbol {x})\right) \right\| _ {W ^ {1, \infty} \left((0, 1) ^ {d}\right)} \leq 5 0 d ^ {\frac {5}{2}} (N + 1) ^ {- 4 d n L}. \tag {86}
$$
$$
\begin{array}{l} \left\| \left(f _ {\boldsymbol {m}} - \psi_ {\boldsymbol {m}} (\boldsymbol {x})\right) \cdot \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \\ \leq \| \left(f _ {\boldsymbol {m}} - \psi_ {\boldsymbol {m}} (\boldsymbol {x})\right) \| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \cdot \| \phi_ {\boldsymbol {m}} \| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} + \| \left(f _ {\boldsymbol {m}} - \psi_ {\boldsymbol {m}} (\boldsymbol {x})\right) \| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \cdot \| \phi_ {\boldsymbol {m}} \| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)} \\ \leq C _ {6} (n, d) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d} \cdot \left(1 + 5 0 d ^ {\frac {5}{2}}\right) + C _ {6} (n, d) N ^ {- 2 n / d} L ^ {- 2 n / d} \cdot 5 4 d ^ {\frac {5}{2}} \lfloor N ^ {1 / d} \rfloor^ {2} \lfloor L ^ {2 / d} \rfloor \\ \leq C _ {7} (n, d) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d}, \tag {87} \\ \end{array}
$$
where the second inequality is due to
$$
\begin{array}{l} \left\| \phi_ {\boldsymbol {m}} \right\| _ {L ^ {\infty} \left(\Omega_ {\boldsymbol {m}}\right)} \leq \left\| \phi_ {\boldsymbol {m}} \right\| _ {L ^ {\infty} ([ 0, 1 ] ^ {d})} \leq \left\| g _ {\boldsymbol {m}} \right\| _ {L ^ {\infty} ([ 0, 1 ] ^ {d})} + \left\| \phi_ {\boldsymbol {m}} - g _ {\boldsymbol {m}} \right\| _ {L ^ {\infty} ([ 0, 1 ] ^ {d})} \leq 1 + 5 0 d ^ {\frac {5}{2}} \\ \| \phi_ {\boldsymbol {m}} \| _ {W ^ {1, \infty} (\Omega_ {\boldsymbol {m}})} \leq \| \phi_ {\boldsymbol {m}} \| _ {W ^ {1, \infty} ([ 0, 1 ] ^ {d})} \leq \| g _ {\boldsymbol {m}} \| _ {W ^ {1, \infty} ([ 0, 1 ] ^ {d})} + \| \phi_ {\boldsymbol {m}} - g _ {\boldsymbol {m}} \| _ {W ^ {1, \infty} ([ 0, 1 ] ^ {d})} \\ \leq 4 \left\lfloor N ^ {1 / d} \right\rfloor^ {2} \left\lfloor L ^ {2 / d} \right\rfloor + 5 0 d ^ {\frac {5}{2}}. \tag {88} \\ \end{array}
$$
$$
\left\| \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \left[ g _ {\boldsymbol {m}} \cdot f (\boldsymbol {x}) - \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \cdot \psi_ {\boldsymbol {m}} (\boldsymbol {x}) \right] \right\| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \leq 2 ^ {d} \left(C _ {7} (n, d) + 5 0 d ^ {\frac {5}{2}}\right) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d} \tag {89}
$$
due to  $(N + 1)^{-4dnL} \leq N^{-2n}L^{-2n}$ .
For the second part, due to Lemma 7, we have
$$
\begin{array}{l} \left\| \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \left[ \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \cdot \psi_ {\boldsymbol {m}} (\boldsymbol {x}) - \widehat {\phi} \left(\phi_ {\boldsymbol {m}} (\boldsymbol {x}), \psi_ {\boldsymbol {m}} (\boldsymbol {x})\right) \right] \right\| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \\ \leq \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \left\| \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \cdot \psi_ {\boldsymbol {m}} (\boldsymbol {x}) - \widehat {\phi} \left(\phi_ {\boldsymbol {m}} (\boldsymbol {x}), \psi_ {\boldsymbol {m}} (\boldsymbol {x})\right) \right\| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \\ = \sum_ {\boldsymbol {m} \in \{1, 2 \} ^ {d}} \left\| \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \cdot \psi_ {\boldsymbol {m}} (\boldsymbol {x}) - \widehat {\phi} \left(\phi_ {\boldsymbol {m}} (\boldsymbol {x}), \psi_ {\boldsymbol {m}} (\boldsymbol {x})\right) \right\| _ {W ^ {1, \infty} \left(\Omega_ {\boldsymbol {m}}\right)}. \tag {90} \\ \end{array}
$$
Similarly with the estimation of  $\mathcal{E}_1$  (76), we have that
$$
\begin{array}{l} \left\| \phi_ {\boldsymbol {m}} (\boldsymbol {x}) \cdot \psi_ {\boldsymbol {m}} (\boldsymbol {x}) - \widehat {\phi} (\phi_ {\boldsymbol {m}} (\boldsymbol {x}), \psi_ {\boldsymbol {m}} (\boldsymbol {x})) \right\| _ {W ^ {1, \infty} (\Omega_ {\boldsymbol {m}})} \\ \leq C _ {8} (n, d) (N + 1) ^ {- 7 n (L + 1)} \leq C _ {8} (n, d) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d}. \tag {91} \\ \end{array}
$$
Combining (89) and (91), we have that there is a  $\sigma_{1}$ -NN  $\phi$  with the width  $(34 + d)2^{d}n^{d + 1}(N + 1)\log_{2}(8N)$  and depth  $56d^{2}n^{2}(L + 1)\log_{2}(4L)$  such that
$$
\left\| f (\boldsymbol {x}) - \phi (\boldsymbol {x}) \right\| _ {W ^ {1, \infty} \left((0, 1) ^ {d}\right)} \leq C _ {9} (n, d) N ^ {- 2 (n - 1) / d} L ^ {- 2 (n - 1) / d},
$$
The method proposed in [28, 23, 39, 38, 37] may not be applied to prove Theorems 3. These works approximate the target function  $f$  using a deep neural network  $\phi$  in the unit cube except for an arbitrarily small region  $\Omega_{\delta}$ , as per [36, Lemma 2.2]. Since  $\| \phi \|_{L^{\infty}(\Omega)}$  can be bounded and is independent of the size of  $\Omega \delta$ ,  $\| f - \phi \|_{L^{p}(\Omega)}$  can be well estimated across the entire space for  $p \in [1, +\infty)$ . For approximations measured in the  $L^{\infty}(\Omega)$  norm, [28] translates the deep neural network  $\phi$ , while [39] constructs different neural networks in the unit cube away from various negligible regions. Both methods aim to find neural networks  $\{\phi_i(\boldsymbol{x})\} i = 1^N$  that approximate the target function  $f$  well in different regions. They then observe that the middle value of  $\{\phi_i(\boldsymbol{x})\}_{i=1}^N$  is close to  $f(\boldsymbol{x})$  for all  $\boldsymbol{x}^* \in \Omega$ , and construct the middle-value function using a ReLU neural network. However, these methods may not be generalized to prove the theorems presented in this paper.
Neither of the methods previously proposed can be applied to the approximation measured in Sobolev space. In the first method,  $\| \phi \|_{W^{1,\infty}(\Omega)}$  depends on the length of  $\Omega \delta$ , and the derivative is substantial in the negligible region, as shown in [36, Lemma 2.2]. Thus,  $\| f - \phi \|_{W^{1,p}(\Omega)}$  will be excessively large. In the second method, median value functions can only identify the median values, not the median values of functions and their derivatives simultaneously. In this paper, we overcome this difficulty using a partition of unity. We construct a partition of unity of  $\Omega$  and approximate them using ReLU DNNs denoted as  $\{\phi_m\}_{m\in \{1,2\}^d}$ . For each  $\phi m$ , its support set is the unit cube away from a small region, and we can construct a deep neural network  $\psi_{m}$  that approximates the target function  $f$  well on  $\operatorname{supp}\phi_{m}$ . We then combine  $\{\phi_m\}_{m\in \{1,2\}^d}$  and  $\{\psi_m\}_{m\in \{1,2\}^d}$  to obtain a deep neural network that can approximate the target function  $f$  well across the entire space. This approach resolves the issue of simultaneous approximation of both functions and their derivatives in Sobolev spaces.
7.3 Proofs of Corollaries 1 and 2
7.3.1 Preliminaries
First, we list a few basic lemmas of  $\sigma_{2}$  neural networks repeatedly applied in our main analysis. Lemma 10 ([23, Lemma 3.7]). The following basic lemmas of  $\sigma_{2}$  neural networks hold:
(i)  $\sigma_{1}$  neural networks are  $\sigma_{2}$  neural networks.
(ii) Any identity map in  $\mathbb{R}^d$  can be realized exactly by a  $\sigma_{2}$  neural network with one hidden layer and 2d neurons.
(iii)  $f(x) = x^{2}$  can be realized exactly by a  $\sigma_{2}$  neural network with one hidden layer and two neurons.
(iv)  $f(x, y) = xy = \frac{(x + y)^2 - (x - y)^2}{4}$  can be realized exactly by a  $\sigma_2$  neural network with one hidden layer and four neurons.
(v) Assume  $\pmb{x}^{\alpha} = x_{1}^{\alpha_{1}}x_{2}^{\alpha_{2}}\dots x_{d}^{\alpha_{d}}$  for  $\pmb {\alpha}\in \mathbb{N}^{d}$ . For any  $N,L\in \mathbb{N}^{+}$  such that  $NL + 2^{\lfloor \log_2N\rfloor}\geq |\pmb {\alpha}|$ , there exists a  $\sigma_{2}$  neural network  $\phi (\pmb {x})$  with the width  $4N + 2d$  and depth  $L + \lceil \log_2N\rceil$  such that
$$
\phi (\boldsymbol {x}) = \boldsymbol {x} ^ {\boldsymbol {\alpha}}
$$
for any  $\pmb {x}\in \mathbb{R}^d$
(vi) Assume  $P(\pmb{x}) = \sum_{j=1}^{J} c_j \pmb{x}^{\alpha_j}$  for  $\alpha_j \in \mathbb{N}^d$ . For any  $N, L, a, b \in \mathbb{N}^+$  such that  $ab \geq J$  and  $(L - 2b - b \log_2 N)N \geq b \max_j |\alpha_j|$ , there exists a  $\sigma_2$  neural network  $\phi(\pmb{x})$  with the width  $4Na + 2d + 2$  and depth  $L$  such that
$$
\phi (\boldsymbol {x}) = P (\boldsymbol {x}) f o r a n y \boldsymbol {x} \in \mathbb {R} ^ {d}.
$$
Next, we define a function which will be repeatedly used in the proof of Corollary 1 in this section.
Definition 11. Define  $s(x)$  from  $\mathbb{R}\to [0,1]$  as
$$
s (x) := \left\{ \begin{array}{l l} 2 x ^ {2}, & x \in [ 0, \frac {1}{2} ] \\ - 2 (x - 1) ^ {2} + 1, & x \in [ \frac {1}{2}, 1 ] \\ 1, & x \in [ 1, 2 ] \\ - 2 (x - 2) ^ {2} + 1, & x \in [ 2, \frac {5}{2} ] \\ 2 (x - 3) ^ {2}, & x \in [ \frac {5}{2}, 3 ] \\ 0, & \text {O t h e r w i s e .} \end{array} \right. \tag {92}
$$
Figure 4:  $s(x)$  in  $\mathbb{R}$ .
Definition 12. Given  $K\in \mathbb{N}_{+}$ , then we define two functions in  $\mathbb{R}$ :
$$
s _ {1} (x) = \sum_ {i = 0} ^ {K} s \left(4 K x + 1 - 4 i\right), s _ {2} (x) = s _ {1} \left(x + \frac {1}{2 K}\right). \tag {93}
$$
Then for any  $\pmb{m} = (m_1, m_2, \dots, m_d) \in \{1, 2\}^d$ , we define
$$
s _ {\boldsymbol {m}} (\boldsymbol {x}) := \prod_ {j = 1} ^ {d} s _ {m _ {j}} \left(x _ {j}\right) \tag {94}
$$
for any  $\pmb{x} = (x_{1}, x_{2}, \ldots, x_{d}) \in \mathbb{R}^{d}$ .
Proposition 8. Given  $N, L, d \in \mathbb{N}_+$  with  $NL + 2^{\lfloor \log_2 N \rfloor} \geq d$  and  $L \geq \lceil \log_2 N \rceil$ , and setting  $K = \lfloor N^{1/d} \rfloor \lfloor L^{2/d} \rfloor$ ,  $\{s_m(x)\}_{m \in \{1,2\}^d}$  defined in Definition 12 satisfies:
(i):  $\| s_{\pmb{m}}(\pmb{x})\|_{L^{\infty}((0,1)^d)}\leq 1,\| s_{\pmb{m}}(\pmb{x})\|_{W^{1,\infty}((0,1)^d)}\leq 8K$  and  $\| s_{\pmb{m}}(\pmb {x})\|_{W^{1,\infty}((0,1)^d)}\leq 64K^2$  for any  $\pmb {m}\in \{1,2\} ^d$
(ii):  $\{s_{\pmb{m}}(\pmb{x})\}_{\pmb{m} \in \{1,2\}^d}$  is a partition of the unity  $[0,1]^d$  with  $\operatorname{supp} s_{\pmb{m}}(\pmb{x}) \cap [0,1]^d = \Omega_{\pmb{m}}$  defined in Definition 5.
(iii): For any  $\pmb{m} \in \{1,2\}^d$ , there is a  $\sigma_2$  neural network  $\lambda_{\pmb{m}}(\pmb{x})$  with the width  $16N + 2d$  and depth  $4L + 5$  such as
$$
\lambda_ {\boldsymbol {m}} (\boldsymbol {x}) = \prod_ {j = 1} ^ {d} s _ {m _ {j}} (x _ {j}) = s _ {\boldsymbol {m}} (\boldsymbol {x}), \boldsymbol {x} \in [ 0, 1 ] ^ {d}.
$$
Proof. (i) and (ii) are proved by direct calculation. The proof of (iii) follows:
First, we architect  $s(x)$  by a  $\sigma_{2}$  neural network. The is a  $\sigma_{1}$  neural network  $g(x)$  with 3 the width and one layer such that:
$$
g (x) := \left\{ \begin{array}{l l} x, & x \in \left[ 0, \frac {1}{2} \right] \\ \frac {1}{2}, & x \in \left[ \frac {1}{2}, + \infty\right) \\ 0, & \text {O t h e r w i s e .} \end{array} \right. \tag {95}
$$
Based on (iii) in Lemma 10,  $g^{2}(x)$  is a  $\sigma_{2}$  neural network with 3 the width and two layers. Then by direct calculation, we notice that
$$
s (x) = 2 g ^ {2} (x) - 2 g ^ {2} (- x + 1) + 2 g ^ {2} (3 - x) - 2 g ^ {2} (2 + x) + \frac {1}{2}, \tag {96}
$$
which is a  $\sigma_{2}$  neural network with 12 the width and two layers. The  $\widetilde{g} (x)$  defined as
$$
\widetilde {g} (x) = \sum_ {i = 0} ^ {\lfloor N ^ {1 / d} \rfloor - 1} s \left(4 K x - 4 i - \frac {1}{2}\right) \tag {97}
$$