SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We present a PTAS for learning random constant-depth networks. We show that for any fixed  $\epsilon > 0$  and depth  $i$ , there is a poly-time algorithm that for any distribution on  $\sqrt{d} \cdot \mathbb{S}^{d-1}$  learns random Xavier networks of depth  $i$ , up to an additive error of  $\epsilon$ . The algorithm runs in time and sample complexity of  $(\bar{d})^{\mathrm{poly}(\epsilon^{-1})}$ , where  $\bar{d}$  is the size of the network. For some cases of sigmoid and ReLU-like activations the bound can be improved to  $(\bar{d})^{\mathrm{polylog}(\epsilon^{-1})}$ , resulting in a quasi-poly-time algorithm for learning constant depth random networks.
One of the greatest mysteries surrounding deep learning is the discrepancy between its phenomenal capabilities in practice and the fact that despite a great deal of research, polynomial-time algorithms for learning deep models are known only for very restrictive cases. Indeed, state of the art results are only capable of dealing with two-layer networks under assumptions on the input distribution and the network's weights. Furthermore, theoretical study shows that even with very naive architectures, learning neural networks is worst-case computationally intractable.
In this paper, we contrast the aforementioned theoretical state of affairs, and show that, perhaps surprisingly, even though constant-depth networks are completely out of reach from a worst-case perspective, most of them are not as hard as one would imagine. That is, they are distribution-free learnable in polynomial time up to any desired constant accuracy. This is the first polynomial-time approximation scheme (PTAS) for learning neural networks of depth greater than 2 (see the related work section for more details). Moreover, we show that the standard SGD algorithm on a ReLU network can be used as a PTAS for learning random networks. The question of whether learning random networks can be done efficiently was posed by Daniely et al. [15], and our work provides a positive result in that respect.
In a bit more detail, we consider constant-depth random networks obtained using the standard Xavier initialization scheme [22, 26], and any input distribution supported on the sphere  $\sqrt{d} \cdot \mathbb{S}^{d-1}$ . For Lipschitz activation functions, our algorithm runs in time  $(\bar{d})^{\mathrm{poly}(\epsilon^{-1})}$ , where  $\bar{d}$  is the network's size including the  $d$  input components, and  $\epsilon$  is the desired accuracy. While this complexity is polynomial for constant  $\epsilon$ , we also consider the special cases of sigmoid and ReLU-like activations, where the bound can be improved to  $(\bar{d})^{\mathrm{polylog}(\epsilon^{-1})}$ .
The main technical idea in our work is that constant-depth random neural networks with Lipschitz activations can be approximated sufficiently well by low-degree polynomials. This result follows by analyzing the network obtained by replacing each activation function with its polynomial approximation using Hermite polynomials. It implies that efficient algorithms for learning polynomials can be
used for learning random neural networks, and specifically that we can use the SGD algorithm on ReLU networks for this task.
In this work, we show that random fully-connected feedforward neural networks can be well-approximated by low-degree polynomials, which implies a PTAS for learning random networks. We start by defining the network architecture. We will denote by  $\sigma: \mathbb{R} \to \mathbb{R}$  the activation function, and will assume that it is  $L$ -Lipschitz. To simplify the presentation, we will also assume that it is normalized in the sense that  $\mathbb{E}_{X \sim \mathcal{N}(0,1)} \sigma^2(X) = 1$ . Define  $\epsilon_{\sigma}(n) = \min_{\deg(p) = n} \mathbb{E}_{X \sim \mathcal{N}(0,1)} (\sigma(X) - p(X))^2$ , namely, the error when approximating  $\sigma$  with a degree- $n$  polynomial, and note that  $\lim_{n \to \infty} \epsilon_{\sigma}(n) = 0$ . We will consider fully connected networks of depth  $i$  and will use  $d_0 = d$  to denote the input dimension and  $d_1, \ldots, d_i$  to denote the number of neurons in each layer. Denote also  $\bar{d} = \sum_{j=0}^{i} d_j$ . Given weight matrices
$$
\vec {W} = \left(W ^ {1}, \dots , W ^ {i}\right) \in \mathbb {R} ^ {d _ {1} \times d _ {0}} \times \dots \times \mathbb {R} ^ {d _ {i} \times d _ {i - 1}}
$$
and  $\mathbf{x} \in \mathbb{R}^{d_0}$  we define  $\Psi_{\vec{W}}^0(\mathbf{x}) = \mathbf{x}$ . Then for  $1 \leq j \leq i$  we define recursively
$$
\Phi_ {\vec {W}} ^ {j} (\mathbf {x}) = W ^ {j} \Psi_ {\vec {W}} ^ {j - 1} (\mathbf {x}), \quad \Psi_ {\vec {W}} ^ {j} (\mathbf {x}) = \sigma \left(\Phi_ {\vec {W}} ^ {j} (\mathbf {x})\right)
$$
We will consider random networks in which the weight matrices are random Xavier matrices [22, 26]. That is, each entry in  $W^{j}$  is a centered Gaussian of variance  $\frac{1}{d_{j-1}}$ . This choice is motivated by the fact that it is a standard practice to initialize the network's weights with Xavier matrices, and furthermore, it ensures that the scale across the network is the same. That is, for any example  $\mathbf{x}$  and a neuron  $n$ , the second moment of the output of  $n$  (w.r.t. the choice of  $\vec{W}$ ) is 1.
Our main result shows that  $\Psi_{\vec{W}}^i$  can be approximated, up to any constant accuracy  $\epsilon$ , via constant degree polynomials (the constant will depend only on  $\epsilon$ , the depth  $i$ , and the activation  $\sigma$ ). We will consider the input space  $\tilde{\mathbb{S}}^{d-1} = \{\mathbf{x} \in \mathbb{R}^d : \| \mathbf{x} \| = 1\}$ . Here, and throughout the paper,  $\| \mathbf{x} \|$  stands for the normalized Euclidean norm  $\| \mathbf{x} \| = \sqrt{\frac{1}{d} \sum_{i=1}^{d} x_i^2}$ .
Theorem 1.1. For every  $i$  and  $n$  such that  $\epsilon_{\sigma}(n) \leq \frac{1}{2}$  there is a constant  $D = D(n, i, \sigma)$  such that if  $d_1, \ldots, d_{i-1} \geq D$  the following holds. For any weights  $\vec{W}$ , there is a degree  $n^{i-1}$  polynomial  $p_{\vec{W}}$  such that for any distribution  $\mathcal{D}$  on  $\tilde{\mathbb{S}}^{d-1}$
$$
\underset {\vec {W}} {\mathbb {E}} \underset {\mathbf {x} \sim \mathcal {D}} {\mathbb {E}} \left\| \Phi_ {\vec {W}} ^ {i} (\mathbf {x}) - p _ {\vec {W}} (\mathbf {x}) \right\| \leq 1 4 \cdot (L + 1) ^ {2} \cdot (\epsilon_ {\sigma} (n)) ^ {\frac {1}{2 ^ {i - 1}}} \leq \frac {1 4 \cdot (L + 1) ^ {3}}{n ^ {\frac {1}{2 ^ {i - 1}}}}.
$$
Furthermore, the coefficients of  $p_{\vec{W}}$  are bounded by  $(2\bar{d})^{4n^{i - 1}}$ .
Since constant degree polynomials are learnable in polynomial time, Theorem 1.1 implies a PTAS for learning random networks of constant depth. In fact, as shown in [9], constant degree polynomials with polynomial coefficients are efficiently learnable via SGD on ReLU networks starting from standard Xavier initialization. Thus, this PTAS can be standard SGD on neural networks. To be
more specific, for any constant  $\epsilon > 0$  there is an algorithm with  $(\bar{d})^{O\left(\left(\frac{14(L + 1)^3}{\epsilon}\right)^{(i - 1)2^{i - 1}}\right)}$  time and sample complexity that is guaranteed to return a hypothesis whose loss is at most  $\epsilon$  in expectation. For some specific activations, such as the sigmoid  $\sigma(x) = \operatorname{erf}(x) \coloneqq \frac{2}{\sqrt{\pi}} \int_{0}^{x} e^{-\frac{t^2}{2}} dt$ , or the ReLU-like activation  $\sigma(x) = \int_{0}^{x} \operatorname{erf}(t) + 1 dt$  we have that  $\epsilon_{\sigma}(n)$  approaches to 0 exponentially fast (see Lemma A.4 in the appendix). In this case, we get a quasi-polynomial time and sample complexity of  $(\bar{d})^{O\left(\left(\log\left(\frac{14(L + 1)^3}{\epsilon}\right)\right)^{(i - 1)}\right)}$ .
Corollary 1.2. For every constants  $\epsilon, i$  and  $\sigma$  there is a constant  $D$ , a univariate-polynomial  $p$  and a polynomial-time algorithm  $\mathcal{A}$  such that if  $d_1, \ldots, d_{i-1} \geq D$  the following holds. For any distribution
$\mathcal{D}$  on  $\tilde{\mathbb{S}}^{d - 1}$ , if  $h$  is the output of  $\mathcal{A}$  upon seeing  $p(d_0,\dots ,d_i)$  examples from  $\mathcal{D}$ , then
$$
\underset {h} {\mathbb {E}} \underset {\vec {W}} {\mathbb {E}} \underset {\mathbf {x} \sim \mathcal {D}} {\mathbb {E}} \left\| \Phi_ {\vec {W}} ^ {i} (\mathbf {x}) - h (\mathbf {x}) \right\| \leq \epsilon .
$$
Furthermore,  $\mathcal{A}$  can be taken to be SGD on a ReLU network starting from a Xavier initialization.
Learning neural networks efficiently. Efficiently learning classes of neural networks has attracted much interest in recent years. Several works established polynomial-time algorithms for learning one-hidden-layer neural networks with certain input distributions (such as the Gaussian distribution) under the assumption that the weight matrix of the hidden layer is non-degenerate [27, 34, 19, 20, 5, 32, 4]. For example, Awasthi et al. [4] showed such a result for non-degenerate one-hidden-layer ReLU networks with bias terms under Gaussian inputs, and also concluded that one-hidden-layer networks can be learned efficiently under the smoothed-analysis framework. Efficient algorithms for learning one-hidden-layer ReLU networks with Gaussian inputs were also shown in Diakonikolas et al. [18], Diakonikolas and Kane [17]. These results do not require non-degenerate weight matrices, but they require that the output layer weights are all positive, as well as a sub-linear upper bound on the number of hidden neurons. Chen et al. [8] recently showed an efficient algorithm for learning one-hidden-layer ReLU networks with Gaussian inputs, under the assumption that the number of hidden neurons is a constant. Note that all of the aforementioned works consider only one-hidden-layer networks. Chen et al. [7] gave an algorithm for learning deeper ReLU networks, whose complexity is polynomial in the input dimension but exponential in the other parameters (such as the number of hidden units, depth, spectral norm of the weight matrices, and Lipschitz constant of the overall network). Finally, several works established algorithms for learning neural networks, whose complexity is exponential unless we impose strong assumptions on the norms of both the inputs and the weights [23, 30, 33, 24].
Hardness of learning neural networks. As we discussed in the previous paragraph, efficient algorithms for learning ReLU networks are known only for depth-2 networks and under certain assumptions on both the network weights and the input distribution. The limited progress in learning ReLU networks can be partially understood by an abundance of hardness results.
Learning neural networks without any assumptions on the input distribution or the weights is known to be hard (under cryptographic and average-case hardness assumptions) already for depth-2 ReLU networks [28, 3, 11]. For depth-3 networks, hardness results were obtained already when the input distribution is Gaussian [13, 6]. All of the aforementioned hardness results are for improper learning, namely, they do not impose any restrictions on the learning algorithm or on the hypothesis that it returns. For statistical query  $(SQ)$  algorithms, unconditional superpolynomial lower bounds were obtained for learning depth-3 networks with Gaussian inputs [6], and superpolynomial lower bounds for Correlational  $SQ$  ( $CSQ$ ) algorithms were obtained already for learning depth-2 networks with Gaussian inputs [25, 18].
The above negative results suggest that assumptions on the input distribution may not suffice for obtaining efficient learning algorithms. Since in one-hidden-layer networks efficient algorithms exist when imposing assumptions on both the input distribution and the weights, a natural question is whether this approach might also work for deeper networks. Recently, Daniely et al. [15] gave a hardness result for improperly learning depth-3 ReLU networks under the Gaussian distribution even when the weight matrices are non-degenerate. This result suggests that learning networks of depth larger than 2 might require new approaches and new assumptions. Moreover, [15] showed hardness of learning depth-3 networks under the Gaussian distribution even when a small random perturbation is added to the network's parameters, namely, they proved hardness in the smoothed-analysis framework. While adding a small random perturbation to the parameters does not seem to make the problem computationally easier, they posed the question of whether learning random networks, which roughly correspond to adding a large random perturbation, can be done efficiently. The current work gives a positive result in that respect.
Daniely and Vardi [12] studied whether there exist some "natural" properties of the network's weights that may suffice to allow efficient distribution-free learning, where a "natural" property is any property
The leftmost expectation denoted  $\mathbb{E}_h$  is over the examples provided to  $\mathcal{A}$ , as well as the internal randomness of  $\mathcal{A}$ .
that holds w.h.p. in random networks. More precisely, they considered a setting where the target network is random, an adversary chooses some input distribution (that may depend on the target network), and the learning algorithm needs to learn the random target network under this input distribution. They gave a hardness result for improper learning (within constant accuracy) in this setting. Thus, they showed that learning random networks is hard when the input distribution may depend on the random network. Note that in the current work, we give a positive result in a setting where we first fix an input distribution and then draw a random network. Finally, learning deep random networks was studied in Das et al. [16], Agarwal et al. [1], where the authors showed hardness of learning networks of depth  $\omega (\log (d))$  in the SQ model.
2 Proof of Theorem 1.1
We recall that for vectors  $\mathbf{x} \in \mathbb{R}^d$  we use the normalized Euclidean norm  $\| \mathbf{x} \| = \sqrt{\frac{\sum_{i=1}^{d} x_i^2}{d}}$  and take the unit sphere  $\tilde{\mathbb{S}}^{d-1} = \{\mathbf{x} \in \mathbb{R}^d : \| \mathbf{x} \| = 1\}$  w.r.t. this norm as our instance space. Inner products will also be normalized: for  $\mathbf{x}, \mathbf{y} \in \mathbb{R}^d$  we denote  $\langle \mathbf{x}, \mathbf{y} \rangle = \frac{\sum_{i=1}^{d} x_i y_i}{d}$ . For  $\mathbf{x} \in \mathbb{R}^d$  and a closed set  $A \subset \mathbb{R}^d$  we denote  $d(\mathbf{x}, A) := \min_{\mathbf{x}' \in A} \| \mathbf{x} - \mathbf{x}' \|$ . Unless otherwise specified, a random scalar is assumed to be a standard normal, a random vector in  $\mathbb{R}^d$  is assumed to be a centered Gaussian vector with covariance matrix  $\frac{1}{d} I$ , and a random matrix is assumed to be a Xavier matrix. For  $f: \mathbb{R} \to \mathbb{R}$ , we denote  $\| f \|^2 = \mathbb{E}_X f^2(X)$ . We denote the Kronecker delta by  $\delta_{ij}$ , i.e.  $\delta_{ij} = 1$  if  $i = j$  and 0 otherwise.
2.2 Some Preliminaries
We will use the Hermite Polynomials [29] which are defined via the following recursion formula.
$$
h _ {n + 1} (x) = \frac {x}{\sqrt {n + 1}} h _ {n} (x) - \sqrt {\frac {n}{n + 1}} h _ {n - 1} (x), \quad h _ {0} (x) = 1, \quad h _ {1} (x) = x \tag {1}
$$
The Hermite polynomials are the sequence of normalized orthogonal polynomials w.r.t. the standard Gaussian measure. That is, it holds that
$$
\underset {X} {\mathbb {E}} h _ {i} (X) h _ {j} (X) = \delta_ {i j}
$$
More generally, if  $(X,Y)$  is a Gaussian vector with covariance matrix  $\begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix}$  then
$$
\underset {X, Y} {\mathbb {E}} h _ {i} (X) h _ {j} (Y) = \delta_ {i j} \rho^ {i} \tag {2}
$$
$$
h _ {n} ^ {\prime} = \sqrt {n} h _ {n - 1} \tag {3}
$$
and that for even  $n$
$$
\underset {X} {\mathbb {E}} X ^ {n} = (n - 1)!! \tag {4}
$$
Let  $\sigma = \sum_{i=0}^{\infty} a_i h_i$  be the representation of the activation function  $\sigma$  in the basis of the Hermite polynomials. We will also use the dual activation  $\hat{\sigma}(\rho) = \sum_{i=0}^{\infty} a_i^2 \rho^i$  as defined in [14]. We note that  $\hat{\sigma}$  is defined in  $[-1,1]$  and satisfies  $\hat{\sigma}(1) = \| \sigma \|^2 = 1$ .
2.3 Defining a Shadow Network
In order to approximate  $\Psi_W^i$  via a polynomial, we will use a "shadow network" that is obtained by replacing the activation  $\sigma$  with a polynomial approximation of it. We will show that for random networks we can approximate each activation sufficiently well with low-degree Hermite polynomials. Recall that  $\sigma = \sum_{i=0}^{\infty} a_i h_i$  is the representation of  $\sigma$  in the basis of the Hermite polynomials. Define  $\sigma_n = \frac{1}{\sqrt{\sum_{i=0}^n a_i^2}} \sum_{i=0}^n a_i h_i$ . We have  $\epsilon_\sigma(n) = \sum_{i=n+1}^{\infty} a_i^2$  and hence  $\sigma_n = \frac{1}{\sqrt{1 - \epsilon_\sigma(n)}} \sum_{i=0}^n a_i h_i$ .
We next define a shadow network. For  $\mathbf{x} \in \mathbb{R}^d$  we let  $\Psi_{\vec{W}}^{0,n}(\mathbf{x}) = \mathbf{x}$ . For  $1 \leq j \leq i$  we define recursively
$$
\Phi_ {\vec {W}} ^ {j, n} (\mathbf {x}) = W ^ {j} \Psi_ {\vec {W}} ^ {j - 1, n} (\mathbf {x}), \quad \Psi_ {\vec {W}} ^ {j, n} (\mathbf {x}) = \sigma_ {n} \left(\Phi_ {\vec {W}} ^ {j, n} (\mathbf {x})\right)
$$
for  $1 \leq j \leq i - 1$  and  $\Psi_{\vec{W}}^{i,n}(\mathbf{x}) = W^i\Psi_{\vec{W}}^{i - 1,n}(\mathbf{x})$ . We will prove the following theorem, which implies Theorem 1.1.
Theorem 2.1. Fix  $i$  and let  $n$  be large enough so that  $\epsilon_{\sigma}(n) \leq \frac{1}{2}$ . There is a constant  $D = D(n, i, \sigma)$  such that if  $d_1, \ldots, d_{i-2} \geq D$  then for any  $\mathbf{x} \in \tilde{\mathbb{S}}^{d-1}$ ,
$$
\underset {\vec {W}} {\mathbb {E}} \left\| \Phi_ {\vec {W}} ^ {i} (\mathbf {x}) - \Phi_ {\vec {W}} ^ {i, n} (\mathbf {x}) \right\| \leq 1 3 \cdot (L + 1) ^ {2} \cdot (\epsilon_ {\sigma} (n)) ^ {\frac {1}{2 ^ {i - 1}}}
$$
Since  $\epsilon_{\sigma}(n)$  is the error in the approximation of a single activation  $\sigma$  with a degree- $n$  polynomial, it is natural to expect that the above bound will depend on  $\epsilon_{\sigma}(n)$ . To see why Theorem 2.1 (together with Lemma A.3 which bounds  $\epsilon_{\sigma}(n)$ ) implies Theorem 1.1, note that  $\Phi_{\vec{W}}^{i,n}(\mathbf{x})$  is a polynomial of degree  $n^{i-1}$ . This implies Theorem 1.1, except the requirement that the coefficients of the polynomial are polynomially bounded. To deal with this, define
$$
\tilde {\Phi} _ {\vec {W}} ^ {i, n} (\mathbf {x}) = \left\{ \begin{array}{l l} \Phi_ {\vec {W}} ^ {i, n} (\mathbf {x}) & \text {i f a l l e n t r i e s i n} \vec {W} \text {a r e a t m o s t} \sum_ {j = 0} ^ {i} d _ {j} \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
As we show next  $\lim_{\min(d_1, \ldots, d_{i-1}) \to \infty} \mathbb{E}_{\vec{W}} \left\| \Phi_{\vec{W}}^{i,n}(\mathbf{x}) - \tilde{\Phi}_{\vec{W}}^{i,n}(\mathbf{x}) \right\| = 0$ . Hence, in the theorem we can replace  $\Phi^{i,n}$  by  $\tilde{\Phi}^{i,n}$  which has polynomially bounded coefficients. See Appendix A.3 and A.4 for the proofs.
Lemma 2.2. For every  $\epsilon$  and  $n$  there is a constant  $D$  such that if  $d_{1},\ldots ,d_{i - 1}\geq D$  then for any  $\mathbf{x}\in \tilde{\mathbb{S}}^{d - 1}$ $\mathbb{E}_{\vec{W}}\left\| \Phi_{\vec{W}}^{i,n}(\mathbf{x}) - \tilde{\Phi}_{\vec{W}}^{i,n}(\mathbf{x})\right\| <  \epsilon .$
Lemma 2.3.  $\tilde{\Phi}_{\vec{W}}^{i,n}$  computes a polynomial whose sum of coefficients is at most  $(2d)^{4n^{i-1}}$ .
2.4 Proof of Theorem 2.1 for depth-two networks
We will first prove Theorem 2.1 for depth-2 networks (i.e. for  $i = 2$ ). We will prove Lemma 2.5 below which implies that for every  $\epsilon$  there is  $n$  such that for any  $\mathbf{x} \in \tilde{\mathbb{S}}^{d - 1}$ ,  $\mathbb{E}_{\vec{W}}\left\| \Psi_{\vec{W}}^{1,n}(\mathbf{x}) - \Psi_{\vec{W}}^{1}(\mathbf{x})\right\| \leq \epsilon$ . We will then prove Lemma 2.6, that together with Lemma 2.5 will show that  $\mathbb{E}_{\vec{W}}\left\| \Phi_{\vec{W}}^{2,n}(\mathbf{x}) - \Phi_{\vec{W}}^{2}(\mathbf{x})\right\| \leq \epsilon$ , thus proving Theorem 2.1 for  $i = 2$ . We will start however with the following lemma that will be useful throughout (see Appendix A.5 for the proof).
Lemma 2.4. Fix  $f, g: \mathbb{R} \to \mathbb{R}$ ,  $\mathbf{x}, \mathbf{y} \in \mathbb{R}^{d_1}$  and a Xavier matrix  $W \in \mathbb{R}^{d_2 \times d_1}$ . Let  $(X,Y)$  be a centered Gaussian vector with covariance matrix  $\begin{pmatrix} \| \mathbf{x} \|^2 & \langle \mathbf{x}, \mathbf{y} \rangle \\ \langle \mathbf{x}, \mathbf{y} \rangle & \| \mathbf{y} \|^2 \end{pmatrix}$ . Then
$$
\underset {W} {\mathbb {E}} \left\| f (W \mathbf {x}) - g (W \mathbf {y}) \right\| \leq \sqrt {\underset {W} {\mathbb {E}} \left\| f (W \mathbf {x}) - g (W \mathbf {y}) \right\| ^ {2}} = \sqrt {\underset {X , Y} {\mathbb {E}} (f (X) - g (Y)) ^ {2}}
$$
Lemma 2.5. Fix  $\mathbf{x} \in \tilde{\mathbb{S}}^{d_1 - 1}$ . Let  $W \in \mathbb{R}^{d_2 \times d_1}$  be a Xavier matrix. Then
$$
\mathop{\mathbb{E}}\limits_{W}\left\| \sigma (W\mathbf{x}) - \sigma_{n}(W\mathbf{x})\right\| \leq \sqrt{2\epsilon_{\sigma}(n)}
$$
Proof. By Lemma 2.4 we have
$$
\underset {W} {\mathbb {E}} \left\| \sigma (W \mathbf {x}) - \sigma_ {n} (W \mathbf {x}) \right\| \leq \sqrt {\underset {W} {\mathbb {E}} \left\| \sigma (W \mathbf {x}) - \sigma_ {n} (W \mathbf {x}) \right\| ^ {2}} = \sqrt {\underset {X} {\mathbb {E}} (\sigma (X) - \sigma_ {n} (X)) ^ {2}}.
$$
Now, the above equals to
$$
\begin{array}{l} \sqrt {\sum_ {i = 0} ^ {n} \left(1 - \frac {1}{\sqrt {1 - \epsilon_ {\sigma} (n)}}\right) ^ {2} a _ {i} ^ {2} + \sum_ {i = n + 1} ^ {\infty} a _ {i} ^ {2}} = \sqrt {(1 - \epsilon_ {\sigma} (n)) \left(1 - \frac {1}{\sqrt {1 - \epsilon_ {\sigma} (n)}}\right) ^ {2} + \epsilon_ {\sigma} (n)} \\ = \sqrt {\left(1 - \epsilon_ {\sigma} (n)\right) \left(\frac {\sqrt {1 - \epsilon_ {\sigma} (n)} - 1}{\sqrt {1 - \epsilon_ {\sigma} (n)}}\right) ^ {2} + \epsilon_ {\sigma} (n)} \\ = \sqrt {2 - \epsilon_ {\sigma} (n) - 2 \sqrt {1 - \epsilon_ {\sigma} (n)} + \epsilon_ {\sigma} (n)} \\ = \sqrt {2 (1 - \sqrt {1 - \epsilon_ {\sigma} (n)})} \\ \leq \sqrt {2 (1 - \sqrt {1 - \epsilon_ {\sigma} (n)}) (1 + \sqrt {1 - \epsilon_ {\sigma} (n)})} \\ = \sqrt {2 \epsilon_ {\sigma} (n)} \\ \end{array}
$$
Lemma 2.5 implies that  $\mathbb{E}_{\vec{W}}\left\| \Psi_{\vec{W}}^{1,n}(\mathbf{x}) - \Psi_{\vec{W}}^{1}(\mathbf{x})\right\| \leq \sqrt{2\epsilon_{\sigma}(n)}$ . Thus, given  $\epsilon >0$ , for sufficiently large  $n$ ,  $\mathbb{E}_{\vec{W}}\left\| \Psi_{\vec{W}}^{1,n}(\mathbf{x}) - \Psi_{\vec{W}}^{1}(\mathbf{x})\right\| \leq \epsilon$ . The following lemma therefore implies that  $\mathbb{E}_{\vec{W}}\left\| \Phi_{\vec{W}}^{2,n}(\mathbf{x}) - \Phi_{\vec{W}}^{2}(\mathbf{x})\right\| \leq \sqrt{2\epsilon_{\sigma}(n)}$  and thus implies Theorem 2.1 for depth two networks.
Lemma 2.6. For any  $\mathbf{x} \in \tilde{\mathbb{S}}^{d-1}$
$$
\underset {W ^ {i}} {\mathbb {E}} \left\| \Phi_ {\vec {W}} ^ {i, n} (\mathbf {x}) - \Phi_ {\vec {W}} ^ {i} (\mathbf {x}) \right\| \leq \left\| \Psi_ {\vec {W}} ^ {i - 1, n} (\mathbf {x}) - \Psi_ {\vec {W}} ^ {i - 1} (\mathbf {x}) \right\|
$$
$$
\begin{array}{l} \mathop{\mathbb{E}}_{W^{i}}\left\| \Phi_{\vec{W}}^{i,n}(\mathbf{x}) - \Phi_{\vec{W}}^{i}(\mathbf{x})\right\| = \mathop{\mathbb{E}}_{W^{i}}\left\| W^{i}\left(\Psi_{\vec{W}}^{i - 1,n}(\mathbf{x}) - \Psi_{\vec{W}}^{i - 1}(\mathbf{x})\right)\right\| \\ \leq \sqrt {\frac {\mathbb {E}}{X \sim \mathcal {N} \left(0 , \left\| \Psi_ {\tilde {W}} ^ {i - 1 , n} (\mathbf {x}) - \Psi_ {\tilde {W}} ^ {i - 1} (\mathbf {x}) \right\| ^ {2}\right)} X ^ {2}} \\ = \left\| \Psi_ {\vec {W}} ^ {i - 1, n} (\mathbf {x}) - \Psi_ {\vec {W}} ^ {i - 1} (\mathbf {x}) \right\| \\ \end{array}
$$
2.5 Proof of Theorem 2.1 for General Networks
For  $\mathbf{x} \in \tilde{\mathbb{R}}^{d_{i-1}}$  we denote  $\Psi_{W^i}(\mathbf{x}) = \sigma(W^i\mathbf{x})$  and  $\Psi_{W^i}^n(\mathbf{x}) = \sigma_n(W^i\mathbf{x})$ . Lemma 2.5 can be roughly phrased as
$$
\left(\mathbf {x} = \mathbf {x} ^ {\prime}\right) \text {a n d} \left(\| \mathbf {x} \| = 1\right) \Rightarrow \Psi_ {W ^ {i}} (\mathbf {x}) \approx \Psi_ {W ^ {i}} ^ {n} \left(\mathbf {x} ^ {\prime}\right)
$$