SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In order to prove Theorem 2.1 for general networks we will extend it by replacing the strict equality conditions with softer ones. That is, we will show that
$$
\left(\mathbf {x} \approx \mathbf {x} ^ {\prime}\right) \text {a n d} \left(\| \mathbf {x} \| \approx 1\right) \text {a n d} \left(\| \mathbf {x} ^ {\prime} \| \approx 1\right) \Rightarrow \Psi_ {W ^ {i}} (\mathbf {x}) \approx \Psi_ {W ^ {i}} ^ {n} \left(\mathbf {x} ^ {\prime}\right) \tag {5}
$$
This will be enough to prove Theorem 2.1 for general networks. Indeed, the conditions  $\| \mathbf{x} \| \approx 1$  and  $\| \mathbf{x}' \| \approx 1$  are valid w.h.p. via a simple probabilistic argument. Thus, Eq. (5) implies that
$$
\mathbf {x} \approx \mathbf {x} ^ {\prime} \Rightarrow \Psi_ {W ^ {i}} (\mathbf {x}) \approx \Psi_ {W ^ {i}} ^ {n} \left(\mathbf {x} ^ {\prime}\right) \tag {6}
$$
Now, for  $\mathbf{x} \in \tilde{\mathbb{S}}^{d-1}$  Eq. (6) implies that  $\Psi_{W^1}(\mathbf{x}) \approx \Psi_{W^1}^n(\mathbf{x}')$ . Using Eq. (6) again we get that  $\Psi_{W^2} \circ \Psi_{W^1}(\mathbf{x}) \approx \Psi_{W^2}^n \circ \Psi_{W^1}^n(\mathbf{x}')$ . Using it  $i-3$  more times we get that  $\Psi_{W^{i-1}} \circ \dots \circ \Psi_{W^1}(\mathbf{x}) \approx \Psi_{W^{i-1}}^n \circ \Psi_{W^1}^n(\mathbf{x}')$ , or in other words that  $\Psi_{\vec{W}}^{i-1}(\mathbf{x}) \approx \Psi_{\vec{W}}^{i-1,n}(\mathbf{x})$ . As we will show “ $\approx$ ” stands for a sufficiently strong approximation, which guarantees that  $\mathbb{E}_{\vec{W}} \| \Psi_{\vec{W}}^{i-1}(\mathbf{x}) - \Psi_{\vec{W}}^{i-1,n}(\mathbf{x}) \| \leq \epsilon$ , and hence Lemma 2.6 implies Theorem 2.1.
To prove Eq. (5) we first prove Lemma 2.7 which softens the requirement that  $\mathbf{x} = \mathbf{x}'$ . That is, it shows that
$$
\left(\mathbf {x} \approx \mathbf {x} ^ {\prime}\right) a n d \left(\| \mathbf {x} \| = \| \mathbf {x} ^ {\prime} \| = 1\right) \Rightarrow \Psi_ {W ^ {i}} (\mathbf {x}) \approx \Psi_ {W ^ {i}} ^ {n} \left(\mathbf {x} ^ {\prime}\right)
$$
The second condition which requires that  $\| \mathbf{x} \| = \| \mathbf{x}' \| = 1$  is softened via Lemmas 2.8 and 2.9. Lemma 2.10 then wraps the two softenings together, and shows that Eq. (5) is valid. Finally, in section 2.5.1 we use Lemma 2.10 to prove Theorem 2.1.
Lemma 2.7. Fix  $\mathbf{x},\mathbf{x} + \mathbf{v}\in \tilde{\mathbb{S}}^{d_1 - 1}$  with  $\| \mathbf{v}\| \leq \epsilon$  . Let  $W\in \mathbb{R}^{d_2\times d_1}$  be a Xavier matrix. Then
$$
\underset {W} {\mathbb {E}} \left\| \sigma (W \mathbf {x}) - \sigma_ {n} (W (\mathbf {x} + \mathbf {v})) \right\| \leq \sqrt {2 \epsilon_ {\sigma} (n)} + \sqrt {\frac {2 L ^ {2}}{1 - \epsilon_ {\sigma} (n)} \epsilon}
$$
$$
\left\| \sigma (W \mathbf {x}) - \sigma_ {n} (W (\mathbf {x} + \mathbf {v})) \right\| \leq \left\| \sigma (W \mathbf {x}) - \sigma_ {n} (W \mathbf {x}) \right\| + \left\| \sigma_ {n} (W \mathbf {x}) - \sigma_ {n} (W (\mathbf {x} + \mathbf {v})) \right\|
$$
By Lemma 2.5 we have  $\mathbb{E}_W\| \sigma (W\mathbf{x}) - \sigma_n(W\mathbf{x})\| \leq \sqrt{2\epsilon_\sigma(n)}$  . It remains to bound  $\mathbb{E}_W\| \sigma_n(W\mathbf{x}) - \sigma_n(W(\mathbf{x} + \mathbf{v}))\|$  . By Lemma 2.4 We have
$$
\underset {W} {\mathbb {E}} \left\| \sigma_ {n} (W \mathbf {x}) - \sigma_ {n} (W (\mathbf {x} + \mathbf {v})) \right\| \leq \sqrt {\underset {X , Y} {\mathbb {E}} (\sigma_ {n} (X) - \sigma_ {n} (Y)) ^ {2}}
$$
where  $(X,Y)$  is a centered Gaussian vector with correlation matrix  $\begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix}$  for  $\rho = \langle \mathbf{x},\mathbf{x} + \mathbf{v}\rangle \geq 1 - \epsilon$ . Finally, we have
$$
\begin{array}{l} \underset {X, Y} {\mathbb {E}} \left(\sigma_ {n} (X) - \sigma_ {n} (Y)\right) ^ {2} = \frac {1}{1 - \epsilon_ {\sigma} (n)} \underset {X, Y} {\mathbb {E}} \left(\sum_ {i = 0} ^ {n} a _ {i} \left(h _ {i} (X) - h _ {i} (Y)\right)\right) ^ {2} \\ = \frac {1}{1 - \epsilon_ {\sigma} (n)} \sum_ {i = 0} ^ {n} \sum_ {j = 0} ^ {n} a _ {i} a _ {j} \underset {X, Y} {\mathbb {E}} (h _ {i} (X) - h _ {i} (Y)) (h _ {j} (X) - h _ {j} (Y)) \\ \stackrel {E q. (2)} {=} \frac {1}{1 - \epsilon_ {\sigma} (n)} \sum_ {i = 0} ^ {n} a _ {i} ^ {2} (2 - 2 \rho^ {i}) \\ \leq \frac {2}{1 - \epsilon_ {\sigma} (n)} (\hat {\sigma} (1) - \hat {\sigma} (\rho)) \\ \end{array}
$$
In Lemma A.1 we show that  $\hat{\sigma}$  is  $L^2$ -Lipschitz. Hence the above is at most  $\frac{2L^2}{1 - \epsilon_{\sigma}(n)}\epsilon$ .
We next give a lemma that allows us to "almost jointly project" a pair of points  $\mathbf{x}_1, \mathbf{x}_2 \in \mathbb{R}^d$  on a closed set  $A \subset \mathbb{R}^d$ , without expanding the distance too much. See Appendix A.6 for the proof.
Lemma 2.8. Let  $A\subset \mathbb{R}^d$  a closed set and fix  $\mathbf{x}_1,\mathbf{x}_2\in \mathbb{R}^d$  . There are  $\tilde{\mathbf{x}}_1,\tilde{\mathbf{x}}_2\in A$  such that
$$
\left\| \mathbf {x} _ {1} - \tilde {\mathbf {x}} _ {1} \right\| \leq 2 d (\mathbf {x} _ {1}, A), \quad \left\| \mathbf {x} _ {2} - \tilde {\mathbf {x}} _ {2} \right\| \leq 2 d (\mathbf {x} _ {2}, A) a n d \left\| \tilde {\mathbf {x}} _ {1} - \tilde {\mathbf {x}} _ {2} \right\| \leq 3 \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \|
$$
Lemma 2.9. Let  $\mathbf{x},\mathbf{x} + \mathbf{v}\in \mathbb{R}^{d_1}$  be vectors such that  $\| \mathbf{x}\| = 1$  and  $\| \mathbf{v}\| \leq \epsilon \leq 1$ . Let  $W\in \mathbb{R}^{d_2\times d_1}$  be a Xavier matrix. Then
$$
\underset {W} {\mathbb {E}} \left\| \sigma (W \mathbf {x}) - \sigma (W (\mathbf {x} + \mathbf {v})) \right\| \leq L \epsilon
$$
$$
\underset {W} {\mathbb {E}} \left\| \sigma_ {n} (W \mathbf {x}) - \sigma_ {n} (W (\mathbf {x} + \mathbf {v})) \right\| \leq 2 ^ {2 n + 1} \left(9 (4 n - 1)!!\right) ^ {1 / 4} \epsilon =: \lambda (n) \epsilon
$$
Proof. Fix a centered Gaussian vector  $(X,Y)$  with covariance matrix  $\begin{pmatrix} 1 & \langle \mathbf{x} + \mathbf{v},\mathbf{x}\rangle \\ \langle \mathbf{x} + \mathbf{v},\mathbf{x}\rangle & \| \mathbf{x} + \mathbf{v}\| ^2 \end{pmatrix}$ . Let  $Z = Y - X$ . Note that  $\operatorname{Var}(Z)\leq \epsilon^2$ . By Lemma 2.4 we have
$$
\underset {W} {\mathbb {E}} \| \sigma (W \mathbf {x}) - \sigma (W (\mathbf {x} + \mathbf {v})) \| \leq \sqrt {\mathbb {E} (\sigma (X) - \sigma (X + Z)) ^ {2}} \leq \sqrt {L ^ {2} \mathbb {E} Z ^ {2}} \leq L \epsilon
$$
We now prove the second part. In Lemma A.2, we show that  $|h_i(x) - h_i(x + y)| \leq 2^i \max(|x|, |x + y|, 1)^i |y|$ . Therefore,
$$
\begin{array}{l} | \sigma_ {n} (x) - \sigma_ {n} (x + y) | \leq \sum_ {i = 0} ^ {n} \frac {| a _ {i} |}{\sqrt {1 - \epsilon_ {\sigma} (n)}} | h _ {i} (x) - h _ {i} (x + y) | \\ \leq | y | \sum_ {i = 0} ^ {n} 2 ^ {i} \max  (| x | ^ {i}, | x + y | ^ {i}, 1) \\ \leq | y | 2 ^ {n + 1} \max  \left(| x | ^ {n}, | x + y | ^ {n}, 1\right) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left(\sigma_ {n} (X) - \sigma_ {n} (X + Z)\right) ^ {2} \leq 2 ^ {2 n + 2} \mathbb {E} Z ^ {2} \max  \left(| X | ^ {n}, | X + Z | ^ {n}, 1\right) ^ {2} \\ \leq \quad 2 ^ {2 n + 2} \sqrt {\mathbb {E} Z ^ {4}} \sqrt {\mathbb {E} \max (| X | ^ {4 n} , | X + Z | ^ {4 n} , 1)} \\ \leq \quad 2 ^ {2 n + 2} \sqrt {\mathbb {E} Z ^ {4}} \sqrt {\mathbb {E} [ | X | ^ {4 n} + | X + Z | ^ {4 n} + 1 ]} \\ \stackrel {E q. (4)} {=} 2 ^ {2 n + 2} \sqrt {3 \| \mathbf {v} \| ^ {4}} \sqrt {1 + (4 n - 1) ! ! (\| \mathbf {x} + \mathbf {v} \| ^ {4 n} + \| \mathbf {x} \| ^ {4 n})} \\ \leq \quad 2 ^ {2 n + 2} \sqrt {3 \epsilon^ {4}} \sqrt {3 (1 + \epsilon) ^ {4 n} (4 n - 1) ! !} \\ \leq \quad 2 ^ {2 n + 2} \sqrt {3 \epsilon^ {4}} \sqrt {3 \cdot 2 ^ {4 n} \cdot (4 n - 1) ! !} \\ \end{array}
$$
Lemma 2.4 now implies that
$$
\underset {W} {\mathbb {E}} \left\| \sigma_ {n} (W \mathbf {x}) - \sigma_ {n} (W (\mathbf {x} + \mathbf {v})) \right\| \leq 2 ^ {2 n + 1} (9 (4 n - 1)!!) ^ {1 / 4} \epsilon
$$
Lemma 2.10. Let  $\mathbf{x},\mathbf{x} + \mathbf{v}\in \mathbb{R}^{d_1}$  be vectors such that  $\| \mathbf{v}\| \leq \epsilon$ $||\mathbf{x}|| - 1|\leq \delta \leq 1 / 2$  and  $\| \mathbf{x} + \mathbf{v}\| -1|\leq \delta$  .Let  $W\in \mathbb{R}^{d_2\times d_1}$  be a Xavier matrix. Then
$$
\underset {W} {\mathbb {E}} \left\| \sigma (W \mathbf {x}) - \sigma_ {n} (W (\mathbf {x} + \mathbf {v})) \right\| \leq 2 L \delta + \sqrt {2 \epsilon_ {\sigma} (n)} + \sqrt {\frac {6 L ^ {2}}{1 - \epsilon_ {\sigma} (n)} \epsilon} + 2 \lambda (n) \delta
$$
Proof. By Lemma 2.8 there are vectors  $\mathbf{x}'$ ,  $\mathbf{v}'$  such that  $\| \mathbf{x}' \| = \| \mathbf{x}' + \mathbf{v}' \| = 1$  and
$$
\| \mathbf {x} - \mathbf {x} ^ {\prime} \| \leq 2 \delta , \| \mathbf {x} + \mathbf {v} - \mathbf {x} ^ {\prime} - \mathbf {v} ^ {\prime} \| \leq 2 \delta , a n d \| \mathbf {v} ^ {\prime} \| \leq 3 \| \mathbf {v} \|
$$
Now, we have, by Lemmas 2.7 and 2.9,
$$
\begin{array}{l} \underset {W} {\mathbb {E}} \| \sigma (W \mathbf {x}) - \sigma_ {n} (W (\mathbf {x} + \mathbf {v})) \| \leq \underset {W} {\mathbb {E}} \| \sigma (W \mathbf {x}) - \sigma (W \mathbf {x} ^ {\prime}) \| + \underset {W} {\mathbb {E}} \| \sigma (W \mathbf {x} ^ {\prime}) - \sigma_ {n} (W (\mathbf {x} ^ {\prime} + \mathbf {v} ^ {\prime})) \| \\ + \underset {W} {\mathbb {E}} \left\| \sigma_ {n} \left(W \left(\mathbf {x} ^ {\prime} + \mathbf {v} ^ {\prime}\right)\right) - \sigma_ {n} \left(W \left(\mathbf {x} + \mathbf {v}\right)\right) \right\| \\ \leq 2 L \delta + \sqrt {2 \epsilon_ {\sigma} (n)} + \sqrt {\frac {6 L ^ {2}}{1 - \epsilon_ {\sigma} (n)} \epsilon} + 2 \lambda (n) \delta \\ \end{array}
$$
2.5.1 Concluding the proof of Theorem 2.1
$$
\Psi_ {\vec {W}} ^ {i} (\mathbf {x}, \delta) = \left\{ \begin{array}{l l} 0 & | 1 - \| \Psi_ {\vec {W}} ^ {j} (\mathbf {x}) \| | > \delta \text {o r} | 1 - \| \Psi_ {\vec {W}} ^ {j, n} (\mathbf {x}) \| | > \delta \text {f o r s o m e} j <   i \\ \Psi_ {\vec {W}} ^ {i} (\mathbf {x}) & \text {o t h e r w i s e} \end{array} \right.
$$
$$
\Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}, \delta) = \left\{ \begin{array}{l l} 0 & | 1 - \| \Psi_ {\vec {W}} ^ {j} (\mathbf {x}) \| | > \delta \text {o r} | 1 - \| \Psi_ {\vec {W}} ^ {j, n} (\mathbf {x}) \| | > \delta \text {f o r s o m e} j <   i \\ \Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}) & \text {o t h e r w i s e} \end{array} \right.
$$
$$
\begin{array}{l} \frac {\mathbb {E}}{\vec {W}} \left\| \Psi_ {\vec {W}} ^ {i} (\mathbf {x}) - \Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}) \right\| \leq \frac {\mathbb {E}}{\vec {W}} \left\| \Psi_ {\vec {W}} ^ {i} (\mathbf {x}) - \Psi_ {\vec {W}} ^ {i} (\mathbf {x}, \delta) \right\| + \frac {\mathbb {E}}{\vec {W}} \left\| \Psi_ {\vec {W}} ^ {i} (\mathbf {x}, \delta) - \Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}, \delta) \right\| \\ + \underset {\vec {W}} {\mathbb {E}} \left\| \Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}, \delta) - \Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}) \right\| \\ \end{array}
$$
Theorem 2.1 now follows from Lemmas 2.11 and 2.12 below, together with Lemma 2.6.
Lemma 2.11. Let  $n$  be large enough so that  $\epsilon_{\sigma}(n) \leq \frac{1}{2}$  and let  $\delta < \frac{\sqrt{\epsilon_{\sigma}(n)}}{2L + 2\lambda(n)}$ . Then,
$$
\frac {\mathbb {E}}{\vec {W}} \left\| \Psi_ {\vec {W}} ^ {i} (\mathbf {x}, \delta) - \Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}, \delta) \right\| \leq 1 2 \cdot (L + 1) ^ {2} \cdot (\epsilon_ {\sigma} (n)) ^ {2 ^ {- i}}
$$
Proof. We will prove the result by induction on  $i$ . The case  $i = 0$  is clear as  $\Psi_{\vec{W}}^{0}(\mathbf{x},\delta) = \Psi_{\vec{W}}^{0,n}(\mathbf{x},\delta)$ . Fix  $i > 0$ . For every  $\delta < \frac{1}{2}$  and  $n$  we have by Lemma 2.10
$$
\mathop{\mathbb{E}}_{W^{i}}\left\| \Psi_{\tilde{W}}^{i}(\mathbf{x},\delta) - \Psi_{\tilde{W}}^{i,n}(\mathbf{x},\delta)\right\| \leq 2L\delta +\sqrt{2\epsilon_{\sigma}(n)} +\sqrt{\frac{6L^{2}}{1 - \epsilon_{\sigma}(n)}\left\| \Psi_{\tilde{W}}^{i - 1}(\mathbf{x},\delta) - \Psi_{\tilde{W}}^{i - 1,n}(\mathbf{x},\delta)\right\|} +2\lambda (n)\delta
$$
Taking expectation over  $W^1, \ldots, W^{i-1}$  we get
$$
\begin{array}{l} \begin{array}{r l} \underset {\vec {W}} {\mathbb {E}} & \left\| \Psi_ {\vec {W}} ^ {i} (\mathbf {x}, \delta) - \Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}, \delta) \right\| \end{array} \\ \leq \quad 2 L \delta + \sqrt {2 \epsilon_ {\sigma} (n)} + \underset {\vec {W}} {\mathbb {E}} \sqrt {\frac {6 L ^ {2}}{1 - \epsilon_ {\sigma} (n)} \left\| \Psi_ {\vec {W}} ^ {i - 1} (\mathbf {x} , \delta) - \Psi_ {\vec {W}} ^ {i - 1 , n} (\mathbf {x} , \delta) \right\|} + 2 \lambda (n) \delta \\ \begin{array}{c} \text {J e n s e n i n e q u a l i t y} \\ \leq \end{array} \qquad 2 L \delta + \sqrt {2 \epsilon_ {\sigma} (n)} + \sqrt {\frac {6 L ^ {2}}{1 - \epsilon_ {\sigma} (n)} \underset {W} {\mathbb {E}} \left\| \Psi_ {W} ^ {i - 1} (\mathbf {x} , \delta) - \Psi_ {W} ^ {i - 1 , n} (\mathbf {x} , \delta) \right\|} + 2 \lambda (n) \delta \\ \begin{array}{c} \delta <   \frac {\sqrt {\epsilon_ {\sigma} (n)}}{2 L + 2 \lambda (n)} \\ \leq \end{array} \qquad \qquad 4 \sqrt {\epsilon_ {\sigma} (n)} + \sqrt {\frac {6 L ^ {2}}{1 - \epsilon_ {\sigma} (n)} \frac {}{\mathbb {E}} _ {\vec {W}} \left\| \Psi_ {\vec {W}} ^ {i - 1} (\mathbf {x} , \delta) - \Psi_ {\vec {W}} ^ {i - 1 , n} (\mathbf {x} , \delta) \right\|} \\ \stackrel {\epsilon_ {\sigma} (n) \leq \frac {1}{2}} {\leq} 4 \sqrt {\epsilon_ {\sigma} (n)} + L \sqrt {1 2 \underset {\vec {W}} {\mathbb {E}} \left\| \Psi_ {\vec {W}} ^ {i - 1} (\mathbf {x} , \delta) - \Psi_ {\vec {W}} ^ {i - 1 , n} (\mathbf {x} , \delta) \right\|} \\ \begin{array}{l} \text {I n d u c t i o n h y p o t h e s i s} \\ \leq \end{array} \quad 4 \sqrt {\epsilon_ {\sigma} (n)} + L \sqrt {1 2 \cdot 1 2 \cdot (L + 1) ^ {2} \cdot (\epsilon_ {\sigma} (n)) ^ {2 ^ {- i + 1}}} \\ \leq \quad (L + 1) \sqrt {1 2 \cdot 1 2 \cdot (L + 1) ^ {2} \cdot (\epsilon_ {\sigma} (n)) ^ {2 ^ {- i + 1}}} \\ = \quad 1 2 \cdot (L + 1) ^ {2} \cdot \left(\epsilon_ {\sigma} (n)\right) ^ {2 ^ {- i}} \\ \end{array}
$$
Lemma 2.12. Fix  $i, n, \delta$  and  $\epsilon > 0$ . There is a constant  $D$  such that if  $d_1, \ldots, d_{i-1} \geq D$  then
$$
\underset {\vec {W}} {\mathbb {E}} \left\| \Psi_ {\vec {W}} ^ {i} (\mathbf {x}) - \Psi_ {\vec {W}} ^ {i} (\mathbf {x}, \delta) \right\| + \underset {\vec {W}} {\mathbb {E}} \left\| \Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}, \delta) - \Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}) \right\| \leq \epsilon
$$
Proof sketch (see Appendix A.7 for the formal proof). Let  $B_{i,\delta}$  be the event that for some  $j < i$ ,  $|1 - \| \Psi_{\vec{W}}^{j}(\mathbf{x})\| | > \delta$  or  $|1 - \| \Psi_{\vec{W}}^{j,n}(\mathbf{x})\| | > \delta$ . We have
$$
\underset {\vec {W}} {\mathbb {E}} \left\| \Psi_ {\vec {W}} ^ {i} (\mathbf {x}) - \Psi_ {\vec {W}} ^ {i} (\mathbf {x}, \delta) \right\| = \underset {\vec {W}} {\mathbb {E}} \left[ \left\| \Psi_ {\vec {W}} ^ {i} (\mathbf {x}) \right\| 1 _ {B _ {i, \delta}} \right] \leq \sqrt {\underset {\vec {W}} {\mathbb {E}} \left[ \left\| \Psi_ {\vec {W}} ^ {i} (\mathbf {x}) \right\| ^ {2} \right]} \sqrt {\operatorname * {P r} (B _ {i , \delta})}.
$$
$$
\underset {\vec {W}} {\mathbb {E}} \left\| \Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}) - \Psi_ {\vec {W}} ^ {i, n} (\mathbf {x}, \delta) \right\| \leq \sqrt {\mathbb {E} _ {\vec {W}} \left[ \left\| \Psi_ {\vec {W}} ^ {i , n} (\mathbf {x}) \right\| ^ {2} \right]} \sqrt {\operatorname * {P r} (B _ {i , \delta})}.
$$
Now, the lemma follows by proving that  $\mathbb{E}_{\vec{W}}\left\| \Psi_{\vec{W}}^{i}(\mathbf{x})\right\|^{2}$  and  $\mathbb{E}_{\vec{W}}\left\| \Psi_{\vec{W}}^{i,n}(\mathbf{x})\right\|^{2}$  are bounded by a constant (independent of  $d_0,\ldots ,d_i$ ), and that for every  $\delta ,\epsilon^{\prime},i$  and  $n$ , there is a constant  $D$  such that if  $d_{1},\dots ,d_{i - 1}\geq D$  then  $\operatorname *{Pr}(B_{i,\delta}) < \epsilon '$ .
3 Conclusion and Future work
One of the prominent approaches for explaining the success of neural networks is trying to show that they are capable of learning complex and "deep" models. So far this approach has relatively limited success. Despite that significant progress has been made to show that neural networks can learn shallow models, so far, neural networks were shown to learn only "toy" deep models (e.g. [21, 2, 10, 31]). Not only that, but there are almost no known rich families of deep models that are efficiently learnable by some algorithm (not necessarily gradient methods on neural networks). Our paper suggests that random neural networks might be candidate models. To take this approach further, a natural next step, and a central open question that arises from our work, is to show the existence of an algorithm that learns random networks in time that is polynomial both in  $\frac{1}{\epsilon}$  and the network size. This question is already open for depth-two ReLU networks with two hidden neurons. We note that as implied by [31], such a result, even for a single neuron, will have to go beyond polynomial approximation of the network, and even more generally, beyond kernel methods.
Our result requires a lower bound  $D$  for the network's width, where  $D$  is a constant. We conjecture that this requirement can be relaxed, and leave it to future work. Additional open directions are: (i) the analysis of random convolutional networks, (ii) achieving time and sample complexity of  $(\bar{d})^{O(\epsilon^{-2})}$  for random networks of any constant depth (and not only for depth two), and (iii) finding a PTAS for random networks of depth  $\omega(1)$ .
The research described in this paper was funded by the European Research Council (ERC) under the European Union's Horizon 2022 research and innovation program (grant agreement No. 101041711), and the Israel Science Foundation (grant number 2258/19). This research was done as part of the NSF-Simons Sponsored Collaboration on the Theoretical Foundations of Deep Learning.
[1] N. Agarwal, P. Awasthi, and S. Kale. A deep conditioning treatment of neural networks. In Algorithmic Learning Theory, pages 249–305. PMLR, 2021.
[2] Z. Allen-Zhu and Y. Li. What can resnet learn efficiently, going beyond kernels? arXiv preprint arXiv:1905.10337, 2019.
[3] B. Applebaum, B. Barak, and A. Wigderson. Public-key cryptography from different assumptions. In Proceedings of the forty-second ACM symposium on Theory of computing, pages 171-180, 2010.
[4] P. Awasthi, A. Tang, and A. Vijayaraghavan. Efficient algorithms for learning depth-2 neural networks with general relu activations. Advances in Neural Information Processing Systems, 34:13485–13496, 2021.
[5] A. Bakshi, R. Jayaram, and D. P. Woodruff. Learning two layer rectified neural networks in polynomial time. In Conference on Learning Theory, pages 195-268. PMLR, 2019.
[6] S. Chen, A. Gollakota, A. R. Klivans, and R. Meka. Hardness of noise-free learning for two-hidden-layer neural networks. arXiv preprint arXiv:2202.05258, 2022.
[7] S. Chen, A. R. Klivans, and R. Meka. Learning deep relu networks is fixed-parameter tractable. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 696-707. IEEE, 2022.
[8] S. Chen, Z. Dou, S. Goel, A. R. Klivans, and R. Meka. Learning narrow one-hidden-layer relu networks. arXiv preprint arXiv:2304.10524, 2023.
[9] A. Daniely. Sgd learns the conjugate kernel class of the network. In NIPS, 2017.
[10] A. Daniely and E. Malach. Learning parities with neural networks. In NIPS, 2020.
[31] G. Yehudai and O. Shamir. On the power and limitations of random features for understanding neural networks. arXiv preprint arXiv:1904.00687, 2019.
[32] X. Zhang, Y. Yu, L. Wang, and Q. Gu. Learning one-hidden-layer relu networks via gradient descent. In The 22nd international conference on artificial intelligence and statistics, pages 1524-1534. PMLR, 2019.
[33] Y. Zhang, J. D. Lee, and M. I. Jordan. 11-regularized neural networks are improperly learnable in polynomial time. In International Conference on Machine Learning, pages 993–1001. PMLR, 2016.
[34] K. Zhong, Z. Song, P. Jain, P. L. Bartlett, and I. S. Dhillon. Recovery guarantees for one-hidden-layer neural networks. In International conference on machine learning, pages 4140-4149. PMLR, 2017.
A.1 Some Technical Lemmas
Lemma A.1. If  $\sigma$  is  $L$ -Lipschitz then  $\hat{\sigma}$  is  $L^2$ -Lipschitz in  $[-1, 1]$
Proof. As shown in [14],  $(\hat{\sigma})' = \widehat{\sigma}'$ . Hence, for  $\rho \in [-1, 1]$ ,
$$
\begin{array}{l} | (\hat {\sigma}) ^ {\prime} (\rho) | = \left| \widehat {\sigma} ^ {\prime} (\rho) \right| \\ \leq \| \sigma^ {\prime} \| ^ {2} \\ \leq L ^ {2} \\ \end{array}
$$
Lemma A.2.  $|h_n(x) - h_n(x + y)| \leq 2^n \max(|x|, |x + y|, 1)^n |y|$
Proof. It is not hard to verify by induction on Eq. (1) that
$$
\left| h _ {n} (x) \right| \leq 2 ^ {n / 2} \max  (1, | x | ^ {n})
$$
This implies that for  $\xi \in [x,x + y]$
$$
\begin{array}{l} \left| h _ {n} (x) - h _ {n} (x + y) \right| = \left| h _ {n} ^ {\prime} (\xi) y \right| \\ \stackrel {E q. (3)} {=} \sqrt {n} | h _ {n - 1} (\xi) y | \\ \leq \sqrt {n} 2 ^ {n / 2} \max  (| x |, | x + y |, 1) ^ {n} | y | \\ \leq \quad 2 ^ {n} \max  (| x |, | x + y |, 1) ^ {n} | y | \\ \end{array}
$$
A.2 Bounds on  $\epsilon_{\sigma}(n)$
By Eq. (3) if  $\sigma$  is differentiable  $k$  times then we have  $\sigma^{(k)} = \sum_{i=k}^{\infty} \sqrt{\frac{i!}{(i-k)!}} a_i h_{i-k}$ . Hence, for  $k \leq n+1$ ,
$$
\epsilon_ {\sigma} (n) = \sum_ {i = n + 1} ^ {\infty} a _ {i} ^ {2} \leq \frac {(n + 1 - k) !}{(n + 1) !} \sum_ {i = n + 1} ^ {\infty} \frac {i !}{(i - k) !} a _ {i} ^ {2} \leq \frac {(n + 1 - k) !}{(n + 1) !} \left\| \sigma^ {(k)} \right\| ^ {2} \tag {7}
$$
Lemma A.3. For any  $L$ -Lipschitz  $\sigma$  we have  $\epsilon_{\sigma}(n) \leq \frac{L^2}{n}$ .
Proof. By Eq. (7) for  $k = 1$  we get
$$
\epsilon_ {\sigma} (n) \leq \frac {1}{n + 1} \| \sigma^ {\prime} \| ^ {2} \leq \frac {L ^ {2}}{n + 1}
$$
Lemma A.4. For the sigmoid activation  $\sigma(x) = \int_{0}^{x} e^{-\frac{t^2}{2}} dt$  we have  $\epsilon_{\sigma}(n) \leq 2^{-n}$ .
Proof. We have  $\sigma^{(k)}(x) = (-1)^{k - 1}\sqrt{(k - 1)!} h_{k - 1}(x)e^{-\frac{x^2}{2}}$ . Indeed, it is not hard to verify it for  $k = 1$  and  $k = 2$ . For  $k > 2$  we have via induction that
$$
\begin{array}{l} \sigma^ {(k + 1)} (x) \quad = \quad (- 1) ^ {k - 1} \sqrt {(k - 1) !} \left[ h _ {k - 1} ^ {\prime} (x) - x h _ {k - 1} (x) \right] e ^ {- \frac {x ^ {2}}{2}} \\ \stackrel {E q. (3)} {=} (- 1) ^ {k} \sqrt {k !} \frac {1}{\sqrt {k}} \left[ x h _ {k - 1} (x) - \sqrt {k - 1} h _ {k - 2} (x) \right] e ^ {- \frac {x ^ {2}}{2}} \\ \stackrel {E q. (1)} {=} (- 1) ^ {k} \sqrt {k !} h _ {k} (x) e ^ {- \frac {x ^ {2}}{2}} \\ \end{array}
$$
Hence,  $|\sigma^{(k)}(x)| \leq |\sqrt{(k - 1)!} h_{k - 1}(x)|$ , and now Eq. (7) implies that for any  $k \leq n + 1$
$$
\epsilon_ {\sigma} (n) \leq \frac {(n + 1 - k) !}{(n + 1) !} (k - 1)! = \frac {(n + 1 - k) ! k !}{(n + 1) ! k} = \frac {1}{k \binom {n + 1} {k}}
$$
Taking  $k = \left\lceil \frac{n + 1}{2} \right\rceil$  we conclude that  $\epsilon_{\sigma}(n) \leq 2^{-n}$ .
A.3 Proof of Lemma 2.2
Let  $A$  be the event that there is an entry in  $\vec{W}$  that is greater than  $\sum_{j=0}^{i} d_j$ . We have
$$
\underset {\vec {W}} {\mathbb {E}} \left\| \Phi_ {\vec {W}} ^ {i, n} (\mathbf {x}) - \tilde {\Phi} _ {\vec {W}} ^ {i, n} (\mathbf {x}) \right\| = \mathbb {E} \left[ \left\| \Phi_ {\vec {W}} ^ {i, n} (\mathbf {x}) \right\| \cdot 1 _ {A} \right] \leq \sqrt {\mathbb {E} \left\| \Phi_ {\vec {W}} ^ {i , n} (\mathbf {x}) \right\| ^ {2}} \sqrt {\operatorname * {P r} (A)}
$$
Now, it is not hard to verify that  $\mathbb{E}\left\| \Phi_{\vec{W}}^{i,n}(\mathbf{x})\right\|^2$  is polynomial in  $\sum_{j=0}^{i}d_{j}$  while  $\operatorname*{Pr}(A)$  converges to 0 exponentially fast in  $\sum_{j=0}^{i}d_{j}$ . Thus, if  $\min(d_1,\ldots,d_{i-1})$  is large enough then  $\mathbb{E}_{\vec{W}}\left\| \Phi_{\vec{W}}^{i,n}(\mathbf{x}) - \tilde{\Phi}_{\vec{W}}^{i,n}(\mathbf{x})\right\| < \epsilon$ .
A.4 Proof of Lemma 2.3
We assume that  $\tilde{\Phi}_{\vec{W}}^{i,n} = \Phi_{\vec{W}}^{i,n}$ , as otherwise  $\tilde{\Phi}_{\vec{W}}^{i,n} \equiv 0$ , in which case the lemma is clear. Write  $\sigma_n(x) = \sum_{k=0}^n b_k x^k$  and  $h_j(x) = \sum_{k=0}^j c_{j,k} x^k$ . Via induction on Eq. (1), we have  $|c_{j,k}| \leq 2^{\frac{j}{2}}$ . Hence,