SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 4.1. Choose  $\sigma_w > 0$  small such that  $\gamma \coloneqq 2\sqrt{2}\sigma_w < 1$ . Then for every  $x, x' \in \mathbb{S}^{n_{in} - 1}$ ,  $\lim_{\ell \to \infty} \lim_{n \to \infty} \frac{1}{n} \left\langle h^\ell(x), h^\ell(x') \right\rangle$  and  $\lim_{n \to \infty} \lim_{\ell \to \infty} \frac{1}{n} \left\langle h^\ell(x), h^\ell(x') \right\rangle$  exist and equal to  $\Sigma^*(x, x')$  a.s., i.e.,
$$
\Sigma^ {*} \left(x, x ^ {\prime}\right) := \lim  _ {\ell \rightarrow \infty} \lim  _ {n \rightarrow \infty} A _ {n, \ell} = \lim  _ {n \rightarrow \infty} \lim  _ {\ell \rightarrow \infty} A _ {n, \ell} = \lim  _ {\ell , n \rightarrow \infty} A _ {n, \ell}, \tag {17}
$$
where  $A_{n,\ell} \coloneqq \frac{1}{n}\left\langle h^{\ell}(x),h^{\ell}(x^{\prime})\right\rangle$
Proof. Proof is provided in Appendix F
Lemma 4.1 confirms the two iterated limits of the empirical covariance  $\frac{1}{n}\left\langle h_n^\ell (x),h_n^\ell (x')\right\rangle$  of the pre-activation  $g_{k}^{\ell}$  exist and equal to the double limit  $\Sigma^{*}(x,x^{\prime})$  for any  $x,x^{\prime}\in \mathbb{S}^{n_{in} - 1}$ . Consequently, it establishes the commutation of the two limits, implying that the limit-depth-then-width and limitwidth-then-depth have the same limit. Building upon this result, we can state Theorem 4.4, which asserts that as the width  $n$  of the infinite-depth neural network  $f_{\theta}$  tends to infinity, the output functions  $f_{\theta ,k}$  converge to independent and identically distributed (i.i.d.) centered Gaussian processes with the covariance function  $\Sigma^{*}(x,x^{\prime})$ . The detailed proofs can be found in Appendix G.
Theorem 4.4. Choose  $\sigma_w > 0$  small such that  $\gamma \coloneqq 2\sqrt{2}\sigma_w < 1$ . For infinite-depth neural network  $f_{\theta}$  defined on (1), as width  $n \to \infty$ , the output functions  $f_{\theta,k}$  tend to i.i.d. centered Gaussian processes with covariance function  $\Sigma^*$  defined by
$$
\Sigma^ {*} (x, x ^ {\prime}) = \lim  _ {\ell \rightarrow \infty} \Sigma^ {\ell} (x, x ^ {\prime}), \tag {18}
$$
where  $\Sigma^{\ell}$  are defined in Theorem 4.1.
4.4 The Strict Positive Definiteness of  $\Sigma^{*}$
We conclude this section by establishing the strictly positive definiteness of the limiting covariance function  $\Sigma^{*}$ . Notably, the proof techniques used in Theorem 4.3 are not applicable here, as the strict positive definiteness of  $\Sigma^L$  may diminish as  $L$  approaches infinity.
Instead, we leverage the inherent properties of  $\Sigma^{*}$  itself and Hermitian expansion of the dual activation  $\hat{\phi}$  of  $\phi$  [10]. To explore the essential properties of  $\Sigma^{*}$ , we perform a fine analysis on the pointwise convergence of the covariance function  $\Sigma^L$  for each pair of inputs  $(x,x^{\prime})$ .
Lemma 4.2. Choose  $\sigma_w > 0$  small for which  $\beta \coloneqq \frac{\sigma_w^2}{2}\mathbb{E}|z|^2 |z^2 -1| < 1$ , where  $z$  is standard Gaussian random variable. Then for all  $x,x'\in \mathbb{S}^{n_{in} - 1}$ , the function  $\Sigma^{\ell}$  satisfies
1.  $\Sigma^{\ell}(x,x) = \Sigma^{\ell}(x^{\prime},x^{\prime})$
2.  $\Sigma^{\ell}(x,x)\leq (1 + 1 / \beta)\Sigma^{2}(x,x).$
Consequently,  $\Sigma^{*}(x,x^{\prime}) = \lim_{\ell \to \infty}\Sigma^{\ell}(x,x^{\prime})$  is well-defined and satisfies for all  $x,x^{\prime}\in \mathbb{S}^{n_{in} - 1}$
$$
0 <   \Sigma^ {*} (x, x) = \Sigma^ {*} \left(x ^ {\prime}, x ^ {\prime}\right) <   \infty .
$$
Lemma 4.2, proven in Appendix E, ensures the well-definedness of the limiting covariance function  $\Sigma^{*}$  for all  $x, x' \in \mathbb{S}^{n_{in} - 1}$  by choosing a small  $\sigma_w > 0$ . The lemma also guarantees that  $\Sigma^{*}(x, x)$  and  $\Sigma^{*}(x', x')$  are strictly positive, equal, and finite for all  $x, x' \in \mathbb{S}^{n_{in} - 1}$ . These findings are crucial for demonstrating the strict positive definiteness of  $\Sigma^{*}$ . Specifically, by leveraging these properties of  $\Sigma^{*}$ , we can derive its Hermitian expansion of the limiting kernel  $\Sigma^{*}$ .
By utilizing [23, Theorem 3], we establish in Theorem 4.5, as proven in Appendix H, that  $\Sigma^{*}$  is strictly positive definite if  $\phi$  is non-polynomial. It is important to note that our analysis can be extended to analyze the covariance or kernel functions induced by neural networks, particularly those that are defined as limits or induced by infinite-depth neural networks. This is because the analysis does not rely on the existence of the positive definiteness of priors. Instead, we examine the intrinsic properties of  $\Sigma^{*}$ , which remain independent of the properties of the activation function  $\phi$ .
Theorem 4.5. For a non-polynomial Lipschitz nonlinear  $\phi$ , for any input dimension  $n_0$ , the restriction of the limiting covariance  $\Sigma^*$  to the unit sphere  $\mathbb{S}^{n_0 - 1} = \{x: \|x\| = 1\}$ , is strictly positive definite.
In this section, we present a series of numerical experiments to validate the theoretical results we have established. Our experiments aim to verify the well-posedness of the fixed point of the transition equation (2). We also investigate whether the DEQ behaves as a Gaussian process when the width is sufficiently large, as stated in our main result, Theorem 4.4. Additionally, we examine the strictly positive definiteness of the limiting covariance function  $\Sigma^{*}$ , as established in Theorem 4.5, by computing the smallest eigenvalue of the associated covariance matrix  $K^{*}$ . These experiments serve to empirically support our theoretical findings.
5.1 Convergence to the fixed point
Proposition 3.1 guarantees the existence of a unique fixed point for the DEQ. To verify this, we conducted simulations using neural networks with the transition equation (2). We plotted the relative error between  $h^{\ell}$  and  $h^{\ell + 1}$  in Figure 1. As shown in the figure, we observed that the iterations required for convergence to the fixed point were approximately 25 for various widths. This observation aligns with our theoretical findings in Lemma F.1, where the random initialization (5) scales the weight matrix  $W$  such that  $\| W \|_{\mathrm{op}} = \mathcal{O}(\sigma_w)$ .
5.2 The Gaussian behavior
Theorem 4.4 predicts that the outputs of DEQ tends to follow a Gaussian process as the width approaches infinity. To demonstrate this, we consider a specific DEQ with  $n_{in} = 10$  and  $n_{out} = 10$ , activated by tanh. We analyze the output distributions of 10,000 neural networks. An important
Figure 1: Convergence to the fixed point (left); distribution of the first neuron of the output for 10,000 neural networks, KS statistics, p-value (middle); joint distributions for the first neuron for three outputs for 10,000 neuron networks, with orange curve denotes the Gaussian distribution (right)
Figure 2: Joint distributions for the first neuron for two different inputs over 10,000 neural networks (left); Covariance matrix obtained by a neural network (middle); Covariance matrix obtained by Gaussian process (right)
implication of Theorem 4.4 is that the output forms an independent identical Gaussian distribution. To visualize this, we plot a pairplot in Figure 1 illustrating the randomly selected three outputs, confirming the validity of this implication.
Next, we generate histograms of the 10,000 neural networks to approximate the distribution of the first neuron in the output layer. In the third plot of Figure 1, we present the histogram for a width of 1000. Remarkably, the output distribution exhibits a strong adherence to the Gaussian model, as evidenced by a Kolmogorov-Smirnov (KS) statistic of 0.0056 and a corresponding p-value of 0.9136. Furthermore, in Figure 5 in the supplementary material, we provide histograms for widths of 10, 50, 100, 500, and 1000. As the width increases, the output distribution progressively converges towards a Gaussian distribution. This is evident from the decreasing KS statistics and the increasing p-values as the width extends from 10 to 1000.
Based on Theorem 4.4, the outputs of the neural network exhibit a behavior reminiscent of a joint Gaussian distribution for different inputs  $x$  and  $x'$ . To illustrate this, we plot the first output of the 10,000 neural networks for two distinct inputs as the first plot in Figure 2. Notably, the predicted limiting Gaussian level curves, derived from the limiting kernel function stated in Lemma 4.2, perfectly match the results of the simulations when the width is set to 1000.
5.3 Convergence of the kernel
According to Theorem 4.4, the DEQs tend to a Gaussian process with a covariance function  $\Sigma^{*} = \lim_{\ell \to \infty}\Sigma^{\ell}$ . Given  $N$  distinct inputs  $\{x_i\}_{i=1}^N$ , as stated in Theorem 4.1, the limiting covariance matrix  $K^{*}$  can be computed recursively, i.e.,  $K_{ij}^{\ell} = \Sigma^{\ell}(x_i,x_j)$ . By Lemma 4.1, each element  $K_{ij}^{\ell}$  can be approximated by  $\frac{1}{n}\langle h^{\ell}(x_i),h^{\ell}(x_j)\rangle$ . We conduct a series of numerical experiments to visually assess this convergence
First of all, we examine the convergence in width. We fix a large depth  $\ell$  and vary the widths by  $2^{2-13}$ . We draw the errors between limiting covariance matrix  $K^{*}$  and finite-width empirical estimate  $K_{n}^{\ell}$  in the first two plots of Figure 3. The relative errors  $\| K_n^\ell - K^* \|_F / \| K^* \|_F$  consistently decreases as the growth of the width, and a convergence rate of order  $n^{-1}$  is observed.
Figure 3: Covariance behaviors with varying width and depth
Figure 4: From left to right:  $\lambda_{\mathrm{min}}(K^{\ell})$  across varying depths  $\ell$ ;  $\lambda_{\mathrm{min}}(K^{*})$  for different  $\sigma_w$  (blue curve: theory; orange curve: simulation); Test accuracy of the MNIST dataset using NNGP and DEQs with various widths; MSE of the MNIST dataset using NNGP and DEQs with various widths.
Next, we examine the convergence in depth by fixing a large width. The results are shown in the third and fourth plots of Figure 3. From these plots, we can observe that the error converges rapidly as the depth of the network increases, illustrating an exponential convergence rate.
5.4 The positive definiteness of the kernel
Theorem 4.5 establishes that the NNGP kernel is strictly positive. As discussed earlier, the kernel matrix  $K^L$  can be computed recursively, as stated in Theorem 4.1 or Corollary 4.2. We refer to this computation as the theoretical approach. Alternatively, it can be calculated as the covariance through simulation, which we denote as simulation approach. We employ both methods to compute the smallest eigenvalue of the kernel matrix  $K^L$ . The results are summarized in Figure 4. It is evident from the figure that the smallest eigenvalues increase with increasing depths and become stable once the kernel is well approximated. Furthermore, the smallest eigenvalue increases with higher values of  $\sigma_w$ .
5.5 Test Performance
To complement the theoretical analysis, we conducted numerical experiments demonstrating the NNGP correspondence for DEQs on real datasets with varying widths. A visual representation of these findings is available in Figure 4. Intriguingly, our observations consistently reveal that the NNGP continually outperforms trained finite-width DEQs. Moreover, a compelling trend emerges: as network width increases, the performance of DEQs converges more closely to NNGP performance. Notably, this phenomenon mirrors observations made in the context of standard feedforward neural networks [25, 28]. These experiments stand as practical evidence, effectively shedding light on the behavior of DEQs across different network sizes. The insights gleaned from these experiments have been thoughtfully integrated into our paper to enhance its comprehensiveness and practical relevance.
This paper establishes that DEQs (Deep Equilibrium Models) can be characterized as Gaussian processes with a strict positive definite covariance function  $\Sigma^{*}$  in the limit of the width of the network approaching infinity. This finding contributes to the understanding of the convergence properties of infinite-depth neural networks, demonstrating that when and how the depth and width limits commute. An important direction for future research is to leverage the results presented in this paper to investigate the training and generalization performance of DEQs. While the results obtained in this paper hold for commonly used activation functions, it would be interesting to explore more complex transition functions in future work.
We would like to acknowledge the generous support of the National Science Foundation (NSF) under grant DMS-1812666 and III-2104797.
[39] Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
[40] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes overparameterized deep relu networks. Machine Learning, 109(3):467-492, 2020.
A Useful Mathematical Results
Theorem A.1. Let  $A$  be  $m \times m$  random matrix whose entries  $A_{ij}$  are independent identically distributed standard Gaussian random variables. Then, there exists absolute constant  $c, C > 0$  such that
$$
\left\| A \right\| _ {o p} \leq C \sqrt {m}, \quad \text {w i t h p r o b a b i l i t y a t l e a s t} 1 - 2 e ^ {- c m}. \tag {19}
$$
Theorem A.2 (Strong Bai-Yin theorem). Let  $A$  be  $m \times m$  random matrix whose entries  $A_{ij}$  are independent identically distributed standard Gaussian random variables. Then
$$
\lim  _ {m \rightarrow \infty} \| A \| _ {o p} / \sqrt {m} = \sqrt {2}, \quad a l m o s t s u r e l y. \tag {20}
$$
Theorem A.3 (Kolmogorov's SLLN for i.i.d.). Let  $\{X_{n}\}$  be sequence of i.i.d. random variables and  $S_{n} = \sum_{i = 1}^{n}X_{i}$ . Then  $\frac{S_n}{n}\stackrel {a.s.}{\to}\mathbb{E}X_1$  if and only if  $\mathbb{E}|X_1| < \infty$ .
Lemma A.1 (Almost surely convergence). Some important properties of almost surely convergence.
1. If  $X_{n}\stackrel {a.s.}{\to}X$  , then  $g(X_{n})\stackrel {a.s.}{\to}g(X)$  for all continuous function  $g$
2. If  $X_{n} \stackrel{a.s.}{\to} X$  and  $Y_{n} \stackrel{a.s.}{\to} Y$ , then  $X_{n}Y_{n} \stackrel{a.s.}{\to} XY$ .
3. If  $X_{n} \xrightarrow{a.s.} X$  and  $Y_{n} \xrightarrow{a.s.} Y$ , then  $aX_{n} + bY_{n} \xrightarrow{a.s.} aX + bY$ .
Lemma A.2 (Gaussian smoothing). Let  $f, g$  be a real-valued function. Define function  $F(\sigma) \coloneqq \mathbb{E}_{z \sim \mathcal{N}(\mu, \sigma^2)} f(z)$  and  $G(\mu) = \mathbb{E}_{z \sim \mathcal{N}(\mu, \sigma^2)} g(z)$  for  $\sigma > 0$ . Suppose  $f(x), g(x) \in o(e^{-x^2})$ , then
$$
F ^ {\prime} (\sigma) = \frac {1}{\sigma} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} [ f (\mu + \sigma z) (z ^ {2} - 1) ]
$$
$$
G ^ {\prime} (\mu) = \frac {1}{\sigma} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} [ g (\mu + \sigma z) z ]
$$
Proof. Note that  $F(\sigma) = \mathbb{E}_{z\sim \mathcal{N}(0,1)}f(\mu +\sigma z)$ , then
$$
\begin{array}{l} F ^ {\prime} (\sigma) = \frac {d}{d \sigma} \int_ {- \infty} ^ {\infty} f (\mu + \sigma z) \frac {1}{\sqrt {2 \pi}} e ^ {- z ^ {2} / 2} d z \\ = \int_ {- \infty} ^ {\infty} f ^ {\prime} (\mu + \sigma z) z \frac {1}{\sqrt {2 \pi}} e ^ {- z ^ {2} / 2} d z \\ = \int_ {- \infty} ^ {\infty} f ^ {\prime} (u) \left(\frac {u - \mu}{\sigma}\right) \frac {1}{\sigma \sqrt {2 \pi}} e ^ {- \frac {(u - \mu) ^ {2}}{2 \sigma^ {2}}} d u, \quad u = \mu + \sigma z \\ = \frac {1}{\sigma} \int_ {- \infty} ^ {\infty} f (u) \left[ \frac {(u - \mu) ^ {2}}{\sigma^ {2}} - 1 \right] \frac {1}{\sigma \sqrt {2 \pi}} e ^ {- \frac {(u - \mu) ^ {2}}{2 \sigma^ {2}}} d u \\ = \frac {1}{\sigma} \int_ {- \infty} ^ {\infty} f (\mu + \sigma z) [ z ^ {2} - 1 ] \frac {1}{\sqrt {2 \pi}} e ^ {- \frac {z ^ {2}}{2}} d z \\ = \frac {1}{\sigma} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} [ f (\mu + \sigma z) (z ^ {2} - 1) ] \\ \end{array}
$$
$$
\begin{array}{l} G ^ {\prime} (\mu) = \frac {d}{d \mu} \int_ {- \infty} ^ {\infty} g (\mu + \sigma z) \frac {1}{\sqrt {2 \pi}} e ^ {- z ^ {2} / 2} d z \\ = \int_ {- \infty} ^ {\infty} g ^ {\prime} (\mu + \sigma z) \frac {1}{\sqrt {2 \pi}} e ^ {- z ^ {2} / 2} d z \\ = \int_ {- \infty} ^ {\infty} g ^ {\prime} (u) \frac {1}{\sigma \sqrt {2 \pi}} e ^ {- \frac {(u - \mu) ^ {2}}{2 \sigma^ {2}}} d u, \quad u = \mu + \sigma z \\ = \frac {1}{\sigma} \int_ {- \infty} ^ {\infty} g (u) \left(\frac {u - \mu}{\sigma}\right) \frac {1}{\sigma \sqrt {2 \pi}} e ^ {- \frac {(u - \mu) ^ {2}}{2 \sigma^ {2}}} d u \\ = \frac {1}{\sigma} \int_ {- \infty} ^ {\infty} g (\mu + \sigma z) z \frac {1}{\sqrt {2 \pi}} e ^ {- \frac {z ^ {2}}{2}} d z \\ = \frac {1}{\sigma} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} [ g (\mu + \sigma z) z ] \\ \end{array}
$$
Lemma A.3 (Gaussian conditioning). Given  $G \in \mathbb{R}^{n \times m}$  and  $H \in \mathbb{R}^{n \times m}$ , let  $W \in \mathbb{R}^{n \times n}$  to follow matrix Gaussian distribution, i.e.,  $W \sim \mathcal{MN}(0, \sigma I_n, \sigma I_n)$  for some  $\sigma > 0$ , suppose  $G = WH$  has feasible solutions. Then the conditional distribution of  $W$  given on  $G = WH$  is
$$
W | _ {G = W H} \sim \mathcal {M N} (G H ^ {\dagger}, I _ {n}, \sigma^ {2} \Pi \Pi^ {T}).
$$
where  $\Pi = I_n - HH^\dagger$  is the orthogonal projection onto the null  $(H^T)$ .
Proof. First, we consider the optimization problem
$$
\min  _ {W} \frac {1}{2} \| W \| _ {F} ^ {2}, \quad s. t. \quad G = W H.
$$
The Lagrange function is given by
$$
L (W, V) = \frac {1}{2} \| W \| _ {F} ^ {2} + \langle V, G - W H \rangle .
$$
The KKT condition implies  $\nabla_W L(W, V) = W - V H^T = 0$  and further  $W = V H^T$ . Since  $G = W H$ , we have  $V = G(H^T H)^\dagger$  and so  $W^* = G(H^T H)^\dagger H^T = G H^\dagger$ .
Then let  $\Pi = I_n - HH^\dagger$  be the orthogonal projection onto the  $\mathrm{null}(H^T)$ . Thus, the conditional distribution of  $W$  given  $G = WH$  is
$$
W | _ {G = W H} = G H ^ {\dagger} + \tilde {W} \Pi^ {T} = \mathcal {M N} (G H ^ {\dagger}, I _ {n}, \sigma^ {2} \Pi \Pi^ {T}).
$$
Lemma A.4 (Conditional distribution). Let  $X \sim \mathcal{MN}(M, U, V)$ . Partition  $X$ ,  $M$ , and  $V$  such that
$$
X = \left[ \begin{array}{c} X _ {1} \\ X _ {2} \end{array} \right], \quad M = \left[ \begin{array}{c} M _ {1} \\ M _ {2} \end{array} \right] \quad U = \left[ \begin{array}{c c} U _ {1 1} & U _ {1 2} \\ U _ {2 1} & U _ {2 2} \end{array} \right]
$$
where  $X_{1}\in \mathbb{R}^{m\times p}$  . Then
$$
\begin{array}{l} X _ {1} \sim \mathcal {M N} (M _ {1}, U _ {1 1}, V) \\ X _ {2} | X _ {1} \sim \mathcal {M N} \left(M _ {2} + U _ {2 1} U _ {1 1} ^ {- 1} \left(X _ {1} - M _ {2}\right), U _ {2 2} - U _ {2 1} U _ {1 1} ^ {- 1} U _ {1 2}, V\right). \\ \end{array}
$$
Note, if  $U_{21} = 0$ , then  $X_{2}|X_{1}\sim \mathcal{MN}(M_{2},U_{22},V)$  indicates  $X_{2}$  and  $X_{1}$  are independent.
Lemma A.5. Let  $\sigma$  be a  $L$ -Lipschitz continuous function. Then  $\sigma$  is also a controllable function. In addition,  $\phi(x,y) \coloneqq \sigma(x)\sigma(y)$  is also a controllable function.
Proof. WOLG, we can assume  $L = 1$ . As  $\sigma$  is Lipschitz continuous on its region, there must exist some  $x_0$  such that  $\sigma(x_0) = c$ . Then we have
$$
| \sigma (x) | \leq | \sigma (x) - \sigma (x _ {0}) | + | c | \leq | x - x _ {0} | + | c | \leq e ^ {| c | ^ {- 1} | x - x _ {0} |} \leq e ^ {| c | ^ {- 1} | x _ {0} |} e ^ {| c | ^ {- 1} | x |} = C _ {1} e ^ {C _ {2} | x |}.
$$
$$
| \phi (x, y) | = | \sigma (x) | | \sigma (y) | \leq C _ {1} e ^ {C _ {2} (| x | + | y |)}.
$$
Lemma A.6. Let  $f$  be a controllable function. Then for all  $\mu \in \mathbb{R}$  and  $\sigma \geq 0$ , we have
$$
\mathbb {E} _ {z \sim \mathcal {N} (\mu , \sigma^ {2})} | f (z) | \leq 2 C _ {1} e ^ {C _ {2} | \mu | + C _ {2} ^ {2} \sigma^ {2} / 2}.
$$
$$
\begin{array}{l} \mathbb {E} _ {z \sim \mathcal {N} (\mu , \sigma^ {2})} | f (z) | = \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} | f (\sigma z + \mu) | \\ \leq \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} C _ {1} e ^ {C _ {2} (\sigma | z | + | \mu |)} \\ = C _ {1} e ^ {C _ {2} | \mu |} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} e ^ {C _ {2} \sigma | z |} \\ = C _ {1} e ^ {| \mu |} \int_ {- \infty} ^ {\infty} e ^ {C _ {2} \sigma | z |} \frac {1}{\sqrt {2 \pi}} e ^ {- z ^ {2} / 2} d z \\ = C _ {1} e ^ {| \mu |} \left[ \int_ {- \infty} ^ {0} e ^ {- C _ {2} \sigma z} \frac {1}{\sqrt {2 \pi}} e ^ {- z ^ {2} / 2} d z + \int_ {0} ^ {\infty} e ^ {C _ {2} \sigma z} \frac {1}{\sqrt {2 \pi}} e ^ {- z ^ {2} / 2} d z \right] \\ = C _ {1} e ^ {| \mu |} \left[ \int_ {- \infty} ^ {0} \frac {1}{\sqrt {2 \pi}} e ^ {- \frac {1}{2} (z + C _ {2} \sigma) ^ {2} + \frac {C _ {2} ^ {2} \sigma^ {2}}{2}} d z + \int_ {0} ^ {\infty} \frac {1}{\sqrt {2 \pi}} e ^ {- \frac {1}{2} (z - C _ {2} \sigma) ^ {2} + \frac {C _ {2} ^ {2} \sigma^ {2}}{2}} d z \right] \\ \leq 2 C _ {1} e ^ {C _ {2} | \mu | + C _ {2} ^ {2} \sigma^ {2} / 2} \\ \end{array}
$$
In this Appendix, we show the preactivation  $g_{k}^{\ell}$  acts like Gaussian random variable. As a consequence, the finite-depth neural network  $f_{\theta}^{L}$  tends to a Gaussian process as width  $n \to \infty$ .
Lemma B.1. Suppose the activation function  $\phi$  is nonlinear Lipschitz continuous function. For input  $x$ , let  $g^{1},\dots ,g^{\ell}$  be the resulting pre-activations for  $\ell \in [L]$ . Then for any  $\ell \in [L]$  and for any controllable function  $\Phi :\mathbb{R}^{\ell}\to \mathbb{R}$ , we have as  $m\to \infty$
$$
\frac {1}{n} \sum_ {k = 1} ^ {n} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}\right) \xrightarrow {a . s _ {i}} \mathbb {E} \left[ \Phi \left(z ^ {1}, \dots , z ^ {\ell}\right) \right], \tag {21}
$$
where  $(z^i,z^j)\sim \mathcal{N}(0,\Sigma)$  and the covariance matrix  $\Sigma \in \mathbb{R}^{2\times 2}$  are computed recursively as follows
$$
\Sigma \left(z ^ {1}, z ^ {i}\right) = \delta_ {1, i} \sigma_ {u} ^ {2} \| x \| ^ {2} / n _ {i n}, \quad \forall i \geq 1, \tag {22}
$$
$$
\Sigma \left(z ^ {i}, z ^ {j}\right) = \sigma_ {w} ^ {2} \mathbb {E} \phi \left(u ^ {i - 1}\right) \phi \left(u ^ {j - 1}\right), \quad \forall i \geq 2. \tag {23}
$$
where  $u^1 = z^1$  and  $u^\ell = z^\ell + z^1$  with covariance
$$
\Sigma \left(u ^ {1}, u ^ {i}\right) = \sigma_ {u} ^ {2} \| x \| ^ {2} / n _ {i n}, \quad \forall i \geq 1, \tag {24}
$$
$$
\Sigma \left(u ^ {i}, u ^ {j}\right) = \Sigma \left(z ^ {i}, z ^ {j}\right) + \Sigma \left(z ^ {1}, z ^ {1}\right), \quad \forall i \geq 2. \tag {25}
$$
If, in addition,  $W^i$  and  $W^j$  are independent, then
$$
\Sigma \left(z ^ {i}, z ^ {j}\right) = 0, \quad \forall i \neq j. \tag {26}
$$
Lemma B.2. [39, Theorem 5.4] For any NETSOR program whose weight matrices are random initiated as (5) and all activation functions are controllable. If  $g^1, \dots, g^\ell$  are any  $G$ -vars (i.e., pre-activation in our case), then for any controllable function  $\Phi: \mathbb{R}^\ell \to \mathbb{R}$ , we have
$$
\frac {1}{n} \sum_ {k = 1} ^ {n} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}\right) \xrightarrow {a . s .} \mathbb {E} _ {z \sim \mathcal {N} (\mu , \Sigma)} \Phi (z), \tag {27}
$$
where  $z\coloneqq (z^1,\dots ,z^\ell)$  and  $\mu$  and  $\Sigma$  can computed by [39, Definition 5.2].
Intuitively, these two Lemmas indicate that  $(g_k^1,\dots ,g_k^\ell)$  acts like a multidimensional Gaussian vector whose covariance can be computed recursively. Lemma B.1 is a special case of Lemma B.2 as Lemma B.1 requires each pre-activation  $g^{\ell}$  encoded same input  $x$ , while Lemma B.2 does not make such assumption. In fact, the proof techniques are identical, i.e., uses Gaussian conditions and smoothing inductively on previous results. To make the paper self-contained, here we provide a proof for Lemma B.1 where we simplify the proof of [39, Theorem 5.4] in the following subsections by removing so-called core set.
B.1 Proof of Theorem 4.1 by Using Master Theorem B.1 or B.2
Based on Lemma B.1 or B.2, we can immediately obtain the desired result.
For simplicity, we assume  $\sigma_{\ell} = 1$ . We prove the desired result by induction. For  $L = 1$ , we have  $f_{\theta}^{L}(x) = g^{1}(x) = W^{1}x$  and