SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[32] D. Pollard. Empirical processes: theory and applications. Ims, 1990.
[33] M. Raissi, P. Perdikaris, and G. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[34] A. A Rusu, S. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295, 2015.
[35] B. Sau and V. Balasubramanian. Deep model compression: Distilling knowledge from noisy teachers. arXiv preprint arXiv:1610.09650, 2016.
[36] Z. Shen, H. Yang, and S. Zhang. Nonlinear approximation via compositions. Neural Networks, 119:74-84, 2019.
[37] Z. Shen, H. Yang, and S. Zhang. Deep network approximation characterized by number of neurons. Communications in Computational Physics, 28(5), 2020.
[38] Z. Shen, H. Yang, and S. Zhang. Optimal approximation rate of ReLU networks in terms of width and depth. Journal de Mathématiques Pures et Appliquées, 157:101-135, 2022.
[39] J. Siegel. Optimal approximation rates for deep ReLU neural networks on Sobolev spaces. arXiv preprint arXiv:2211.14400, 2022.
[40] H. Son, J. Jang, W. Han, and H. Hwang. Sobolev training for the neural network solutions of PDEs. arXiv preprint arXiv:2101.08932, 2021.
[41] E. Sontag et al. Vc dimension of neural networks. NATO ASI Series F Computer and Systems Sciences, 168:69-96, 1998.
[42] S. Srinivas and F. Fleuret. Knowledge transfer with jacobian matching. In International Conference on Machine Learning, pages 4723-4731. PMLR, 2018.
[43] E. Stein. Singular integrals and differentiability properties of functions, volume 2. Princeton university press, 1970.
[44] V. Vapnik and A. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Measures of complexity: festschrift for alexey chervonenkis, pages 11-30, 2015.
[45] N. Vlassis, R. Ma, and W. Sun. Geometric deep learning for computational mechanics part i: Anisotropic hyperelasticity. Computer Methods in Applied Mechanics and Engineering, 371:113299, 2020.
[46] N. Vlassis and W. Sun. Sobolev training of thermodynamic-informed neural networks for interpretable elasto-plasticity models with level set hardening. Computer Methods in Applied Mechanics and Engineering, 377:113695, 2021.
[47] M. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019.
[48] P. Werbos. Approximate dynamic programming for real-time control and neural modeling. Handbook of intelligent control, 1992.
7 Supplementary Material
7.1 Proofs of Theorems 1 and 2.
Proof of Theorem 1. An element in  $\Phi$  can be represented as  $\phi = W_{L + 1}\sigma_1(W_L\sigma_1(\ldots \sigma_1(W_1x + b_1)\ldots) + b_L) + b_{L + 1}$ . Therefore, an element in  $D\Phi$  can be represented as
$$
\begin{array}{l} \psi (\boldsymbol {x}) = D _ {i} \phi (\boldsymbol {x}) = \boldsymbol {W} _ {L + 1} \sigma_ {0} \left(\boldsymbol {W} _ {L} \sigma_ {1} \left(\dots \sigma_ {1} \left(\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}\right) \dots\right) + \boldsymbol {b} _ {L}\right) \\ \cdot \boldsymbol {W} _ {L} \sigma_ {0} (\dots \sigma_ {1} (\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}) \dots) \dots \boldsymbol {W} _ {2} \sigma_ {0} (\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}) (\boldsymbol {W} _ {1}) _ {i}, \tag {19} \\ \end{array}
$$
where  $\pmb{W}_i\in \mathbb{R}^{N_i\times N_{i - 1}}((\pmb {W})_i$  is  $i$ -th column of  $\pmb{W}$  and  $\pmb{b}_{i}\in \mathbb{R}^{N_{i}}$  are the weight matrix and the bias vector in the  $i$ -th linear transform in  $\phi$ , and  $\sigma_0(x) = \mathrm{sgn}(x) = 1[x > 0]$ , which is the derivative of the ReLU function and  $\sigma_0(\pmb {x}) = \mathrm{diag}(\sigma_0(x_i))$ . Denote  $W_{i}$  as the number of parameters in  $\pmb{W}_i$ ,  $\pmb{b}_i$ , i.e.,  $W_{i} = N_{i}N_{i - 1} + N_{i}$ .
$$
K := \left| \right.\left\{ \right.\left( \right.\operatorname {s g n} \left(f \left(\boldsymbol {x} _ {1}, \boldsymbol {\theta}\right)\right), \dots , \operatorname {s g n} \left(f \left(\boldsymbol {x} _ {m}, \boldsymbol {\theta}\right)\right): \boldsymbol {\theta} \in \mathbb {R} ^ {W} \left. \right\}\left. \right|. \tag {20}
$$
We define a sequence of sets of functions  $\{\mathbb{F}_j\}_{j = 0}^L$  with respect to parameters  $\pmb {\theta}\in \mathbb{R}^{W}$
$$
\mathbb {F} _ {L} := \left\{\left(\boldsymbol {W} _ {1}\right) _ {i}, \boldsymbol {W} _ {2} \sigma_ {0} \left(\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}\right), \dots , \boldsymbol {W} _ {L + 1} \sigma_ {0} \left(\boldsymbol {W} _ {L} \sigma_ {1} (\dots \sigma_ {1} \left(\boldsymbol {W} _ {1} \boldsymbol {x} + \boldsymbol {b} _ {1}\right) \dots) + \boldsymbol {b} _ {L}\right) \right\}. \tag {21}
$$
1. We have  $|\mathcal{S}_0| = 1$ , and for each  $n = 1, \ldots, L$ , we have  $\frac{|\mathcal{S}_n|}{|\mathcal{S}_{n-1}|} \leq 2 \left( \frac{2 \text{emn} N_k}{\sum_{i=1}^n W_i} \right)^{\sum_{i=1}^n W_i}$ .
2. For each  $n = 0, \dots, L - 1$ , each element  $S$  of  $\mathcal{S}_n$ , when  $\pmb{\theta}$  varies in  $S$ , the output of each term in  $\mathbb{F}_n$  is a fixed polynomial function in  $\sum_{i=1}^{n} W_i$  variables of  $\pmb{\theta}$ , with a total degree no more than  $n + 1$ .
3. For each element  $S$  of  $S_{L}$ , when  $\theta$  varies in  $S$ , the  $h$ -th term in  $\mathbb{F}_L$  for  $h \in \{1, 2, \ldots, L + 1\}$  is a fixed polynomial function in  $W_{h}$  variables of  $\theta$ , with a total degree no more than 1.
We define  $S_0 = \{\mathbb{R}^W\}$ , which satisfies properties 1,2 above, since  $\pmb{W}_1\pmb{x}_j + \pmb{b}_1$  and  $(\pmb{W}_1)_i$  are affine functions of  $\pmb{W}_1, \pmb{b}_1$ .
To define  $S_{n}$ , we use the last term of  $\mathbb{F}_{n-1}$  as inputs for the last two terms in  $\mathbb{F}_n$ . Assuming that  $S_0, S_1, \ldots, S_{n-1}$  have already been defined, we observe that the last two terms are new additions to  $\mathbb{F}_n$  when comparing it to  $\mathbb{F}_{n-1}$ . Therefore, all elements in  $\mathbb{F}_n$  except the last two are fixed polynomial functions in  $W_{n}$  variables of  $\theta$ , with a total degree no greater than  $n$  when  $\theta$  varies in  $S \in S_n$ . This is because  $S_{n}$  is a finer partition than  $S_{n-1}$ .
We denote  $p_{\boldsymbol{x}_j,n - 1,S,k}(\boldsymbol{\theta})$  as the output of the  $k$ -th node in the last term of  $\mathbb{F}_{n - 1}$  in response to  $\boldsymbol{x}_j$  when  $\boldsymbol{\theta} \in S$ . The collection of polynomials
$$
\left\{p _ {\boldsymbol {x} _ {j}, n - 1, S, k} (\boldsymbol {\theta}): j = 1, \dots , m, k = 1, \dots , N _ {n} \right\}
$$
can attain at most  $2\left(\frac{2emnN_n}{\sum_{i=1}^n W_i}\right)^{\sum_{i=1}^n W_i}$  distinct sign patterns when  $\theta \in S$  due to Lemma 1 for sufficiently large  $m$ . Therefore, we can divide  $S$  into  $2\left(\frac{2emnN_n}{\sum_{i=1}^n W_i}\right)^{\sum_{i=1}^n W_i}$  parts, each having the
property that  $p_{\pmb{x}_j,n-1,S,k}(\pmb{\theta})$  does not change sign within the subregion. By performing this for all  $S \in \mathcal{S}_{n-1}$ , we obtain the desired partition  $\mathcal{S}_n$ . This division ensures that the required property 1 is satisfied.
Additionally, since the input to the last two terms in  $\mathbb{F}_n$  is  $p_{\boldsymbol{x}_j,n - 1,S,k}(\boldsymbol{\theta})$ , and we have shown that the sign of this input will not change in each region of  $S_{n}$ , it follows that the output of the last two terms in  $\mathbb{F}_n$  is also a polynomial without breakpoints in each element of  $S_{n}$ . Therefore, the required property 2 is satisfied.
In the context of DNNs, the last layer is characterized by all terms containing the activation function  $\sigma_0$ . Consequently, for any element  $S$  of the partition  $S_L$ , when the vector of parameters  $\pmb{\theta}$  varies within  $S$ , the  $h$ -th term in  $\mathbb{F}_L$  for  $h \in \{1, 2, \dots, L + 1\}$  can be expressed as a polynomial function of at most degree 1, which depends on at most  $W_h$  variables of  $\pmb{\theta}$ . Hence, the required property 3 is satisfied.
Due to property 3, we multiply all the terms in  $\mathbb{F}_L$  and obtain a term in  $D\Phi$ . Hence, the output of each term in  $D\Phi$  is a polynomial function in  $\sum_{i=1}^{L+1} W_i$  variables of  $\theta \in S \in \mathcal{S}_L$ , of total degree no more than  $L+1$ . Therefore, for each  $S \in \mathcal{S}_L$  we have  $|\{\mathrm{sgn}(f(\pmb{x}_1, \pmb{\theta})), \dots, \mathrm{sgn}(f(\pmb{x}_m, \pmb{\theta})) : \pmb{\theta} \in S\}| \leq 2(2em(L+1) / \sum_{i=1}^{L+1} W_i)^{\sum_{i=1}^{L+1} W_i}$ . Then
$$
\begin{array}{l} K \leq 2 \left(2 e m (L + 1) / \sum_ {i = 1} ^ {L + 1} W _ {i}\right) ^ {\sum_ {i = 1} ^ {L + 1} W _ {i}} \cdot \prod_ {n = 1} ^ {L} 2 \left(\frac {2 e m n N _ {n}}{\sum_ {i = 1} ^ {n} W _ {i}}\right) ^ {\sum_ {i = 1} ^ {n} W _ {i}} \leq \prod_ {n = 1} ^ {L + 1} 2 \left(\frac {2 e m n N _ {n}}{\sum_ {i = 1} ^ {n} W _ {i}}\right) ^ {\sum_ {i = 1} ^ {n} W _ {i}} \\ \leq 2 ^ {L + 1} \left(\frac {2 e m (L + 2) (L + 1) N}{2 U}\right) ^ {U} \tag {22} \\ \end{array}
$$
where  $U \coloneqq \sum_{n=1}^{L+1} \sum_{i=1}^{n} W_i = O(N^2 L^2)$ ,  $N$  is the width of the network, and the last inequality is due to weighted AM-GM. For the definition of the VC-dimension, we have
$$
2 ^ {\operatorname {V C d i m} (D \Phi)} \leq 2 ^ {L + 1} \left(\frac {e \operatorname {V C d i m} (D \Phi) (L + 1) (L + 2) N}{U}\right) ^ {U}. \tag {23}
$$
Due to Lemma 2, we obtain that
$$
\operatorname {V C d i m} (D \Phi) \leq L + 1 + U \log_ {2} [ 2 (L + 1) (L + 2) \log_ {2} (L + 1) (L + 2) ] = O \left(N ^ {2} L ^ {2} \log_ {2} L \log_ {2} N\right) \tag {24}
$$
since  $U = \mathcal{O}(N^2 L^2)$
Proof of Theorem 2. Denote  $D\Phi_{\mathcal{N}} \coloneqq \{\eta(\boldsymbol{x}, y) : \eta(\boldsymbol{x}, y) = \psi(\boldsymbol{x}) - y, \psi \in D\Phi, (\boldsymbol{x}, y) \in \mathbb{R}^{d+1}\}$ . Based on the definition of VC-dimension and pseudo-dimension, we have that
$$
\operatorname {P d i m} (D \Phi) \leq \operatorname {V C d i m} (D \Phi_ {\mathcal {N}}). \tag {25}
$$
For the  $\mathrm{VCdim}(D\Phi_{\mathcal{N}})$ , it can be bounded by  $O(N^{2}L^{2}\log_{2}L\log_{2}N)$ . The proof is similar to that for the estimate of  $\mathrm{VCdim}(D\Phi)$  as given in Theorem 1.
We establish that  $\mathrm{Pdim}(D\Phi) \leq \mathrm{VCdim}(D\Phi_{\mathcal{N}})$ , where  $\Phi_{\mathcal{N}}$  represents DNNs with  $N + 1$  width and  $L + 1$  depth. This implies that  $\mathrm{Pdim}(D\Phi)$  is upper bounded by  $\bar{C}(N + 1)^2 (L + 1)^2 \log_2(L + 1) \log_2(N + 1) \leq 64 \bar{C} N^2 L^2 \log_2 L \log_2 N$ . Therefore, we conclude that  $64 \bar{C} \geq \hat{C}$ .
7.2 Proof of Theorem 3
7.2.1 Propositions of Sobolev spaces and ReLU neural networks
The following two lemmas estimate the Sobolev norms and Sobolev semi-norms for the composition and product, which will be used in later proof.
Lemma 3 ([18, Corollary B.5]). Let  $d, m \in \mathbb{N}_+$  and  $\Omega_1 \subset \mathbb{R}^d$  and  $\Omega_2 \subset \mathbb{R}^m$  both be open, bounded, and convex. Then for  $\pmb{f} \in W^{1,\infty}(\Omega_1, \mathbb{R}^m)$  and  $g \in W^{1,\infty}(\Omega_2)$  with  $\mathrm{ran}\pmb{f} \subset \Omega_2$ , we have
$$
\left\| g \circ \boldsymbol {f} \right\| _ {W ^ {1, \infty} \left(\Omega_ {2}\right)} \leq \sqrt {d} m \max  \left\{\left\| g \right\| _ {L ^ {\infty} \left(\Omega_ {2}\right)}, \left| g \right| _ {W ^ {1, \infty} \left(\Omega_ {2}\right)} \left| \boldsymbol {f} \right| _ {W ^ {1, \infty} \left(\Omega_ {1}, \mathbb {R} ^ {m}\right)} \right\}.
$$
Lemma 4 ([18, Corollary B.6]). Let  $d \in \mathbb{N}_+$  and  $\Omega \subset \mathbb{R}^d$ . Then for  $f, g \in W^{1,\infty}(\Omega)$ , we have
$$
\| g f \| _ {W ^ {1, \infty} (\Omega)} \leq \| g \| _ {L ^ {\infty} (\Omega)} | f | _ {W ^ {1, \infty} (\Omega)} + \| f \| _ {L ^ {\infty} (\Omega)} | g | _ {W ^ {1, \infty} (\Omega)}.
$$
Then we collect and establish some propositions for ReLU neural networks.
Proposition 2 ([28, Proposition 4.3]). Given any  $N, L \in \mathbb{N}_+$  and  $\delta \in \left(0, \frac{1}{3K}\right]$  for  $K = \lfloor N^{1/d} \rfloor^2 \lfloor L^{2/d} \rfloor$ , there exists a  $\sigma_1$ -NN  $\phi$  with the width  $4N + 5$  and depth  $4L + 4$  such that
$$
\phi (x) = k, x \in \left[ \frac {k}{K}, \frac {k + 1}{K} - \delta \cdot 1 _ {k <   K - 1} \right], k = 0, 1, \dots , K - 1.
$$
Proposition 3. [28, Proposition 4.4] Given any  $N, L, s \in \mathbb{N}_{+}$  and  $\xi_{i} \in [0,1]$  for  $i = 0,1,\dots N^{2}L^{2}-1$ , there exists a  $\sigma_{1}$ -NN  $\phi$  with the width  $16s(N + 1)\log_2(8N)$  and depth  $(5L + 2)\log_2(4L)$  such that
$$
1. | \phi (i) - \xi_ {i} | \leq N ^ {- 2 s} L ^ {- 2 s} f o r i = 0, 1, \dots N ^ {2} L ^ {2} - 1.
$$
$$
2. 0 \leq \phi (x) \leq 1, x \in \mathbb {R}.
$$
Proposition 4. For any  $N, L \in \mathbb{N}_{+}$  and  $a > 0$ , there is a  $\sigma_{1}$ -NN  $\phi$  with the width  $15N$  and depth  $2L$  such that  $\| \phi \|_{W^{1,\infty}((-a,a)^2)} \leq 12a^2$  and
$$
\left\| \phi (x, y) - x y \right\| _ {W ^ {1, \infty} \left((- a, a) ^ {2}\right)} \leq 6 a ^ {2} N ^ {- L}. \tag {26}
$$
$$
\phi (0, y) = \frac {\partial \phi (0 , y)}{\partial y} = 0, y \in (- a, a). \tag {27}
$$
Proof. We first need to construct a neural network to approximate  $x^{2}$  on  $(-1, 1)$ , and the idea is similar with [23, Lemma 3.2] and [28, Lemma 5.1]. The reason we do not use [23, Lemma 3.4] and [28, Lemma 4.2] directly is that constructing  $\phi(x, y)$  by translating a neural network in  $W^{1,\infty}[0,1]$  will lose the proposition of  $\phi(0.y) = 0$ . Here we need to define teeth functions  $T_{i}$  on  $\widetilde{x} \in [-1, 1]$ :
$$
T _ {1} (\widetilde {x}) = \left\{ \begin{array}{l l} 2 | \widetilde {x} |, & | \widetilde {x} | \leq \frac {1}{2}, \\ 2 (1 - | \widetilde {x} |), & | \widetilde {x} | > \frac {1}{2}, \end{array} \right.
$$
$$
T _ {i} = T _ {i - 1} \circ T _ {1}, \quad \text {f o r} i = 2, 3, \dots .
$$
$$
\widetilde {\psi} (\widetilde {x}) = \widetilde {x} - \sum_ {i = 1} ^ {s} \frac {T _ {i} (\widetilde {x})}{2 ^ {2 i}},
$$
According to [23, Lemma 3.2] and [28, Lemma 5.1], we know  $\psi$  is a neural network with the width  $5N$  and depth  $2L$  such that  $\| \widetilde{\psi} (\widetilde{x})\|_{W^{1,\infty}((-1,1))}\leq 2$ ,  $\| \widetilde{\psi} (\widetilde{x}) - \widetilde{x}^2\|_{W^{1,\infty}((-1,1))}\leq N^{-L}$  and  $\psi (0) = 0$ .
By setting  $x = a\widetilde{x}\in (-a,a)$  for  $\widetilde{x}\in (-1,1)$ , we define
$$
\psi (x) = a ^ {2} \widetilde {\psi} \left(\frac {x}{a}\right).
$$
Note that  $x^{2} = a^{2}\left(\frac{x}{a}\right)^{2}$ , we have
$$
\begin{array}{l} \| \psi (x) - x ^ {2} \| _ {\mathcal {W} ^ {1, \infty} (- a, a)} = a ^ {2} \left\| \widetilde {\psi} \left(\frac {x}{a}\right) - \left(\frac {x}{a}\right) ^ {2} \right\| _ {\mathcal {W} ^ {1, \infty} ((- a, a))} \\ \leq a ^ {2} N ^ {- L}, \\ \end{array}
$$
and  $\psi (0) = 0$  , which will be used to prove Eq. (27).
Then we can construct  $\phi (x,y)$  as
$$
\phi (x, y) = 2 \left[ \psi \left(\frac {| x + y |}{2}\right) - \psi \left(\frac {| x |}{2}\right) - \psi \left(\frac {| y |}{2}\right) \right] \tag {28}
$$
where  $\phi (x)$  is a neural network with the width  $15N$  and depth  $2L$  such that  $\| \phi \|_{W^{1,\infty}((-a,a)^2)}\leq 12a^2$  and
$$
\left\| \phi (x, y) - x y \right\| _ {W ^ {1, \infty} \left((- a, a) ^ {2}\right)} \leq 6 a ^ {2} N ^ {- L}. \tag {29}
$$
For the last equation Eq. (27) is due to  $\phi(x, y)$  in the proof can be read as Eq. (28) with  $\psi(0) = 0$ .
Proposition 5. For any  $N, L, s \in \mathbb{N}_{+}$  with  $s \geq 2$ , there exists a  $\sigma_{1}$ -NN  $\phi$  with the width  $9(N + 1) + s - 1$  and depth  $14s(s - 1)L$  such that  $\| \phi \|_{\mathcal{W}^{1,\infty}((0,1)^s)} \leq 18$  and
$$
\left\| \phi (\boldsymbol {x}) - x _ {1} x _ {2} \dots x _ {s} \right\| _ {\mathcal {W} ^ {1, \infty} ((0, 1) ^ {s})} \leq 1 0 (s - 1) (N + 1) ^ {- 7 s L}. \tag {30}
$$
Furthermore, for any  $i = 1,2,\ldots ,s$ , if  $x_{i} = 0$ , we will have
$$
\phi \left(x _ {1}, x _ {2}, \dots , x _ {i - 1}, 0, x _ {i + 1}, \dots , x _ {s}\right) = \frac {\partial \phi \left(x _ {1} , x _ {2} , \dots , x _ {i - 1} , 0 , x _ {i + 1} , \dots , x _ {s}\right)}{\partial x _ {j}} = 0, i \neq j. \tag {31}
$$
Proof. The proof of the first inequality Eq. (30) can be found in [23, Lemma 3.5]. The proof of Eq. (31) can be obtained via induction. For  $s = 2$ , based on Proposition 4, we know there is a neural network  $\phi_2$  satisfied Eq. (31).
Now assume that for any  $i \leq n - 1$ , there is a neural network  $\phi_i$  satisfied Eq. (31).  $\phi_n$  in [23] is constructed as
$$
\phi_ {n} \left(x _ {1}, x _ {2}, \dots , x _ {n}\right) = \phi_ {2} \left(\phi_ {n - 1} \left(x _ {1}, x _ {2}, \dots , x _ {n - 1}\right), \sigma \left(x _ {n}\right)\right), \tag {32}
$$
which satisfies Eq. (30). Then  $\phi_n(x_1, x_2, \ldots, x_{i-1}, 0, x_{i+1}, \ldots, x_n) = 0$  for any  $i = 1, 2, \ldots, n$ . For  $i = n$ , we have
$$
\frac {\phi \left(x _ {1} , x _ {2} , \dots , 0\right)}{\partial x _ {j}} = \underbrace {\frac {\partial \phi_ {2} \left(\phi_ {n - 1} \left(x _ {1} , x _ {2} , \dots , x _ {n - 1}\right) , 0\right)}{\partial \phi_ {n - 1} \left(x _ {1} , x _ {2} , \dots , x _ {n - 1}\right)}} _ {= 0, \text {b y t h e p r o p e r t y o f} \phi_ {2}.} \cdot \frac {\partial \phi_ {n - 1} \left(x _ {1} , x _ {2} , \dots , x _ {n - 1}\right)}{\partial x _ {j}} = 0. \tag {33}
$$
For  $i <   n$  and  $j <   n$  ,we have
$$
\begin{array}{l} \frac {\phi (x _ {1} , x _ {2} , \ldots , x _ {i - 1} , 0 , x _ {i + 1} , \ldots , x _ {n})}{\partial x _ {j}} \\ = \frac {\partial \phi_ {2} \left(\phi_ {n - 1} \left(x _ {1} , x _ {2} , \dots , x _ {i - 1} , 0 , x _ {i + 1} , \dots , x _ {n - 1}\right) , \sigma \left(x _ {n}\right)\right)}{\partial \phi_ {n - 1} \left(x _ {1} , \dots , 0 , x _ {i + 1} , \dots , x _ {n - 1}\right)} \cdot \underbrace {\frac {\partial \phi_ {n - 1} \left(x _ {1} , \dots , 0 , x _ {i + 1} , \dots , x _ {n - 1}\right)}{\partial x _ {j}}} _ {= 0, \text {v i a i n d u c t i o n .}} = 0. \tag {34} \\ \end{array}
$$
For  $i < n$  and  $j = n$ , we have
$$
\begin{array}{l} \frac {\phi (x _ {1} , x _ {2} , \ldots , x _ {i - 1} , 0 , x _ {i + 1} , \ldots , x _ {n})}{\partial x _ {n}} \\ = \underbrace {\frac {\partial \phi_ {2} \left(\phi_ {n - 1} \left(x _ {1} , x _ {2} , \dots , x _ {i - 1} , 0 , x _ {i + 1} , \dots , x _ {n - 1}\right) , \sigma \left(x _ {n}\right)\right)}{\partial \sigma \left(x _ {n}\right)}} _ {= 0, \text {b y t h e p r o p e r t y} \phi_ {2}.} \cdot \frac {\mathrm {d} \sigma \left(x _ {n}\right)}{\mathrm {d} x _ {n}} = 0. \tag {35} \\ \end{array}
$$
Therefore, Eq. (31) is valid.
Proposition 6 ([23, Proposition 3.6]). For any  $N, L, s \in \mathbb{N}_{+}$  and  $|\alpha| \leq s$ , there is a  $\sigma_{1}$ -NN  $\phi$  with the width  $9(N + 1) + s - 1$  and depth  $14s^{2}L$  such that  $\| \phi \|_{W^{1,\infty}((0,1)^d)} \leq 18$  and
$$
\left\| \phi (\boldsymbol {x}) - \boldsymbol {x} ^ {\boldsymbol {\alpha}} \right\| _ {W ^ {1, \infty} ((0, 1) ^ {d})} \leq 1 0 s (N + 1) ^ {- 7 s L}. \tag {36}
$$
Proposition 7 ([39, Proposition 1]). Given a sequence of the neural network  $\{p_i\}_{i=1}^M$ , and each  $p_i$  is a  $\sigma_1$ -NN from  $\mathbb{R} \to \mathbb{R}$  with the width  $N$  and depth  $L_i$ , then  $\sum_{i=1}^M p_i$  is a  $\sigma_1$ -NN with the width  $N + 4$  and depth  $\sum_{i=1}^M L_i$ .
We present the proof of Proposition 1 below.
Proof of Proposition 1. First, we construct  $g_{1}$  and  $g_{2}$  by neural networks in [0, 1]. Note that  $\lfloor L^{2 / d}\rfloor \leq L^{2 / d}\leq \left(\lfloor L^{1 / d}\rfloor +1\right)^{2}$ . We first construct a  $\sigma_{1}$ -NN in the small set  $\left[0,\lfloor N^{1 / d}\rfloor \lfloor L^{2 / d}\rfloor \right]$ . It is easy to check there is a neural network  $\hat{\psi}$  with the width 4 and one layer such as
$$
\hat {\psi} (x) := \left\{ \begin{array}{l l} 1, & x \in \left[ \frac {1}{8 K}, \frac {3}{8 K} \right] \\ 4 K \left(x - \frac {1}{8 K}\right), & x \in \left[ \frac {1}{8 K}, \frac {3}{8 K} \right] \\ - 4 K \left(x - \frac {7}{8 K}\right), & x \in \left[ \frac {5}{8 K}, \frac {7}{8 K} \right] \\ 0, & \text {O t h e r w i s e .} \end{array} \right. \tag {37}
$$
Figure 2:  $\psi_{1}$
Hence, we have a network  $\psi_{1}$  with the width  $4\lfloor N^{1 / d}\rfloor$  and one layer such as
$$
\psi_ {1} (x) := \sum_ {i = 0} ^ {\lfloor N ^ {1 / d} \rfloor - 1} \hat {\psi} \left(x - \frac {i}{K}\right).
$$
Next, we construct  $\psi_{i}$  for  $i = 2,3,4$  based on the symmetry and periodicity of  $g_{i}$ .  $\psi_{2}$  is the function with period  $\frac{2}{\lfloor N^{1 / d}\rfloor\lfloor L^{2 / d}\rfloor}$  in  $\left[0,\frac{1}{\lfloor L^{2 / d}\rfloor}\right]$ , and each period is a hat function with gradient 1.  $\psi_{3}$  is the function with period  $\frac{2}{\lfloor L^{2 / d}\rfloor}$  in  $\left[0,\frac{\lfloor L^{1 / d}\rfloor + 1}{\lfloor L^{2 / d}\rfloor}\right]$ , and each period is a hat function with gradient 1.  $\psi_{4}$  is the function with period  $\frac{2\left(\lfloor L^{1 / d}\rfloor + 1\right)}{\lfloor L^{2 / d}\rfloor}$  in  $\left[0,\frac{\left(\lfloor L^{1 / d}\rfloor + 1\right)^{2}}{\lfloor L^{2 / d}\rfloor}\right]$ , and each period is a hat function with gradient 1. The schematic diagram is in Fig. 3 (The diagram is shown the case for  $\lfloor N^{1 / d}\rfloor$  and  $\lfloor L^{1 / d}\rfloor +1$  is a even integer.).
Note that  $\psi_{2} \circ \psi_{3} \circ \psi_{4}(x)$  is the function with period  $\frac{2}{\lfloor N^{1/d}\rfloor\lfloor L^{2/d}\rfloor}$  in  $[0,1] \subset \left[0,\frac{\left(\lfloor L^{1/d}\rfloor + 1\right)^{2}}{\lfloor L^{2/d}\rfloor}\right]$ , and each period is a hat function with gradient 1. Then function  $\psi_{1} \circ \psi_{2} \circ \psi_{3} \circ \psi_{4}(x)$  is obtained by repeating reflection  $\psi_{1}$  in  $\left[0,\frac{\left(\lfloor L^{1/d}\rfloor + 1\right)^{2}}{\lfloor L^{2/d}\rfloor}\right]$ , which is the function we want.
Similar with  $\psi_{1},\psi_{2}$  is a network with  $4\lfloor N^{1 / d}\rfloor$  width and one layer. Due to Proposition 7, we know that  $\psi_{3}$  and  $\psi_{4}$  is a network with 7 width and  $\lfloor L^{1 / d}\rfloor +1$  depth. Hence
$$
\psi (x) := \psi_ {1} \circ \psi_ {2} \circ \psi_ {3} \circ \psi_ {4} (x) \tag {38}
$$
is a network with  $4\lfloor N^{1 / d}\rfloor$  width and  $2\lfloor L^{1 / d}\rfloor +4$  depth and  $g_{1} = \psi \left(x + \frac{1}{8K}\right)$  and  $g_{1} = \psi \left(x + \frac{5}{8K}\right)$ .
Now we can construct  $g_{m}$  for  $m \in \{1, 2\}^d$  based on Proposition 5: There is a neural network  $\phi_{\mathrm{prod}}$  with the width  $9(N + 1) + d - 1$  and depth  $14d(d - 1)nL$  such that  $\| \phi_{\mathrm{prod}} \|_{\mathcal{W}^{1,\infty}((0,1)^d)} \leq 18$  and
$$
\left\| \phi_ {\operatorname {p r o d}} (\boldsymbol {x}) - x _ {1} x _ {2} \dots x _ {d} \right\| _ {\mathcal {W} ^ {1, \infty} ((0, 1) ^ {d})} \leq 1 0 (d - 1) (N + 1) ^ {- 7 d n L}.
$$
Figure 3:  $\psi_{i}$  for  $i = 2,3,4$
Then denote  $\phi_{\pmb{m}}(\pmb{x}) \coloneqq \phi_{\mathrm{prod}}(g_{m_1}, g_{m_2}, \ldots, g_{m_d})$  which is a neural network with the width smaller than  $(9 + d)(N + 1) + d - 1$  and depth smaller than  $15d(d - 1)nL$ . Furthermore, due to Lemma 3, we have
$$
\begin{array}{l} \left\| \phi_ {\boldsymbol {m}} (\boldsymbol {x}) - g _ {\boldsymbol {m}} (\boldsymbol {x}) \right\| _ {\mathcal {W} ^ {1, \infty} ((0, 1) ^ {d})} \leq d ^ {\frac {3}{2}} \left\| \phi_ {\operatorname {p r o d}} (\boldsymbol {x}) - x _ {1} x _ {2} \dots x _ {d} \right\| _ {L ^ {\infty} ((0, 1) ^ {d})} \\ + d ^ {\frac {3}{2}} \left\| \phi_ {\operatorname {p r o d}} (\boldsymbol {x}) - x _ {1} x _ {2} \dots x _ {d} \right\| _ {\mathcal {W} ^ {1, \infty} ((0, 1) ^ {d})} | \psi | _ {W ^ {1, \infty} (0, 1)} \\ \leq d ^ {\frac {3}{2}} 1 0 (d - 1) (N + 1) ^ {- 7 n d L} \left(1 + 4 \lfloor N ^ {1 / d} \rfloor^ {2} \lfloor L ^ {2 / d} \rfloor\right) \\ \leq 5 0 d ^ {\frac {5}{2}} (N + 1) ^ {- 4 d n L}, \tag {39} \\ \end{array}
$$