SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f _ {\theta , k} ^ {1} (x) = g _ {k} ^ {1} (x) = \langle w _ {k}, x \rangle \stackrel {i. i. d.} {\sim} \mathcal {N} (0, \| x \| ^ {2} / n _ {i n}).
$$
$$
\hat {\Sigma} ^ {1} (x, x ^ {\prime}) = \operatorname {c o v} \left(f _ {\theta , k} ^ {L} (x), f _ {\theta , k} ^ {L} \left(x ^ {\prime}\right)\right) = \langle x, x ^ {\prime} \rangle := \Sigma^ {1} (x, x ^ {\prime}).
$$
For  $L = 2$ , we have  $f_{\theta}^{L}(x) = g^{2}(x) = W^{2}h^{1}(x)$ . By condition on  $g^{1}$ , we have
$$
f _ {\theta , k} ^ {2} (x) = g _ {k} ^ {2} (x) = \left\langle w _ {k} ^ {2}, h ^ {1} (x) \right\rangle \stackrel {i. i. d.} {\sim} \mathcal {N} (0, \| h ^ {1} (x) \| ^ {2} / n).
$$
$$
\begin{array}{l} \hat {\Sigma} ^ {2} (x, x ^ {\prime}) = \left\langle h ^ {1} (x), h ^ {1} \left(x ^ {\prime}\right) \right\rangle / n \\ = \left\langle \phi \left(g ^ {1} (x)\right), \phi \left(g ^ {1} \left(x ^ {\prime}\right)\right) \right\rangle / n \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \phi \left(g _ {i} ^ {1} (x)\right) \phi \left(g _ {i} ^ {1} \left(x ^ {\prime}\right)\right) \\ \xrightarrow {a . s} \mathbb {E} \phi (z ^ {1} (x)) \phi (z ^ {2} (x ^ {\prime})) \\ =: \Sigma^ {2} (x, x ^ {\prime}), \\ \end{array}
$$
$$
(z ^ {1} (x), z ^ {1} (x ^ {\prime})) \sim \mathcal {N} \left(0, \left[ \begin{array}{c c} \Sigma^ {1} (x, x) & \Sigma^ {1} (x, x ^ {\prime}) \\ \Sigma^ {1} (x ^ {\prime}, x) & \Sigma^ {1} (x ^ {\prime}, x ^ {\prime}) \end{array} \right]\right).
$$
Now, we assume the results hold for  $L$ . Then we show the result for  $f_{\theta}^{L + 1}(x)$ . In this case, we have  $f_{\theta}^{L + 1}(x) = g^{L + 1}(x)$ . By condition on the values  $g^{L}$ , we have the output  $f_{\theta ,k}^{L + 1}$  are i.i.d. centered Gaussian random variables, i.e.,
$$
f _ {\theta , k} ^ {L + 1} (x) = g _ {k} ^ {L + 1} (x) = \left\langle w _ {k} ^ {L + 1}, h ^ {L} (x) \right\rangle \stackrel {i. i. d.} {\sim} \mathcal {N} (0, \| h ^ {L} (x) \| ^ {2} / n).
$$
$$
\begin{array}{l} \hat {\Sigma} ^ {L + 1} (x, x ^ {\prime}) = \operatorname {C o v} \left(f _ {\theta , k} ^ {L + 1} (x), f _ {\theta , k} ^ {L + 1} (x ^ {\prime})\right) \\ = \left\langle h ^ {L} (x), h ^ {L} \left(x ^ {\prime}\right) \right\rangle / n \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \phi \left(g _ {i} ^ {L} (x) + g _ {i} ^ {1} (x)\right) \phi \left(g _ {i} ^ {L} \left(x ^ {\prime}\right) + g _ {i} ^ {1} \left(x ^ {\prime}\right)\right) \\ \xrightarrow {a . s .} \mathbb {E} \phi \left(z ^ {L} (x) + z ^ {1} (x)\right) \phi \left(z ^ {L} \left(x ^ {\prime}\right) + z ^ {1} \left(x ^ {\prime}\right)\right) \\ =: \Sigma^ {L + 1} (x, x ^ {\prime}). \\ \end{array}
$$
$$
\left[ \begin{array}{c} z ^ {1} (x) \\ z ^ {L} (x) \\ z ^ {1} (x ^ {\prime}) \\ z ^ {L} (x ^ {\prime}) \end{array} \right] \sim \mathcal {N} \left(0, \left[ \begin{array}{c c c c} \Sigma^ {1} (x, x) & 0 & \Sigma^ {1} (x, x ^ {\prime}) & 0 \\ 0 & \Sigma^ {L} (x, x) & 0 & \Sigma^ {L} (x, x ^ {\prime}) \\ \hline \Sigma^ {1} (x ^ {\prime}, x) & 0 & \Sigma^ {1} (x ^ {\prime}, x ^ {\prime}) & 0 \\ 0 & \Sigma^ {L} (x ^ {\prime}, x) & 0 & \Sigma^ {L} (x ^ {\prime}, x ^ {\prime}) \end{array} \right]\right).
$$
Here the covariance is deterministic and independent of  $g^{L}$ . Consequently, the conditioned and unconditioned distributions of  $f_{\theta ,k}^{L + 1}$  are equal in the limit: they are i.i.dcentered Gaussian random variables with covariance  $\Sigma^{L + 1}$ .
B.2 Proof of Lemma B.1: the basic case  $\ell = 1$
WLOG, we can assume  $\sigma_{\ell} = 1$ . We prove by induction. When  $\ell = 1$ , we have
$$
g ^ {1} = W _ {1} x
$$
$$
g _ {k} ^ {1} \stackrel {i. i. d.} {\sim} \mathcal {N} (0, \| x \| ^ {2} / n _ {i n}).
$$
Given a controllable function  $\Phi$ , the random variables  $X_{k} = \Phi(g_{k}^{1})$  are still i.i.d.. It follows from Lemma A.6 that
$$
\mathbb {E} \left| X _ {1} \right| = \mathbb {E} _ {z \sim \mathcal {N} (0, \| x \| ^ {2})} | \Phi (z) | \leq C _ {1} e ^ {C _ {2} ^ {2} \| x \| ^ {2}} <   \infty .
$$
Then the desired result is obtained by following Theorem A.3 the classical Kolmogorov's SLLN for i.i.d. random variables.
B.3 Proof of Lemma B.1: general case for independent matrices  $W^{k} \neq W^{\ell}$
Suppose the desired result hold for  $\ell$ , then we show the result also hold for  $\ell + 1$ . In addition, we assume the weight matrices  $W^{\ell}$  are independent to each other. Thus, the weight matrix  $W^{\ell + 1}$  are not used in previous layers. For brevity, we denote  $W := W^{\ell + 1}$  and so we have expression
$$
g ^ {\ell + 1} = W h ^ {\ell}.
$$
Here the randomness of  $g^{\ell + 1}$  comes from both  $W$  and  $h^\ell$ . As  $W$  is not used before,  $W$  and  $h^\ell$  are independent. Let  $\mathcal{B}$  be the  $\sigma$ -algebra spanned by all previous  $g^1, g^2, \dots, g^\ell$ . Then the conditional distribution of  $g^{\ell + 1}$  on  $\mathcal{B}$  is given by
$$
g ^ {\ell + 1} | \mathcal {B} \sim \mathcal {N} (0, \| h ^ {\ell} \| ^ {2} / n I _ {n}),
$$
$$
g _ {k} ^ {\ell + 1} \left| \mathcal {B} \right. \stackrel {i. i. d.} {\sim} \mathcal {N} \left(0, \| h ^ {\ell} \| ^ {2} / n\right). \tag {28}
$$
By using the inductive hypothesis, we have
$$
\sigma_ {n} ^ {2} := \| h ^ {\ell} \| ^ {2} / n = \frac {1}{n} \sum_ {k = 1} ^ {n} \phi \left(g _ {k} ^ {\ell} + g _ {k} ^ {1}\right) ^ {2} \stackrel {a. s.} {\rightarrow} \mathbb {E} \left[ \phi \left(z ^ {\ell} + z ^ {1}\right)\right] ^ {2} = \Sigma \left(z ^ {\ell + 1}, z ^ {\ell + 1}\right) := \sigma^ {2}, \tag {29}
$$
where we use the fact  $\Phi (x,y)\coloneqq \phi (x + y)$  is controllable, i.e.,
$$
| \Phi (x, y) | = | \phi (x + y) | \leq | x + y | \leq e ^ {| x | + | y |}.
$$
By using triangle inequality, we have
$$
\left| \frac {1}{n} \sum_ {k = 1} ^ {n} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell + 1}\right) - \mathbb {E} \left[ \Phi \left(z ^ {1}, \dots , z ^ {\ell + 1}\right) \right] \right| \leq | A _ {n} | + | B _ {n} | + | C _ {n} |,
$$
$$
A _ {n} = \frac {1}{n} \sum_ {k = 1} ^ {n} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell + 1}\right) - \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} \left(0, \sigma_ {n} ^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) \tag {30}
$$
$$
B _ {n} = \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} \left(0, \sigma_ {n} ^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) - \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} \left(0, \sigma^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) \tag {31}
$$
$$
C _ {n} = \frac {1}{n} \sum_ {k = 1} ^ {n} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) - \mathbb {E} \left[ \Phi \left(z ^ {1}, \dots , z ^ {\ell + 1}\right) \right] \tag {32}
$$
$A_{n}$  converges to 0 almost surely
Define random variables  $Z_{k}\coloneqq \Phi (g_{k}^{1},\dots ,g_{k}^{\ell},g_{k}^{\ell +1}) - \mathbb{E}_{z\sim \mathcal{N}(0,\sigma_{n}^{2})}\Phi (g_{k}^{1},\dots ,g_{k}^{\ell},z)$ . By equation (28), we have  $g_{k}^{\ell +1}|\mathcal{B}^{\textit{i.i.d.}}\sim \mathcal{N}(0,\sigma_{n}^{2})$ , we can easily show  $X_{k}$  are centered and uncorrelated. Observe that
$$
\begin{array}{l} \mathbb {E} Z _ {k} = \mathbb {E} _ {\mathcal {B}} \mathbb {E} _ {g ^ {\ell + 1} | \mathcal {B}} Z _ {k} \\ = \mathbb {E} _ {\mathcal {B}} \mathbb {E} _ {g ^ {\ell + 1} | \mathcal {B}} \left[ \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, g _ {k} ^ {\ell + 1}\right) - \mathbb {E} _ {z \sim \mathcal {N} \left(0, \sigma_ {n} ^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) \right] \\ = \mathbb {E} _ {\mathcal {B}} \left[ \mathbb {E} _ {g ^ {\ell + 1} | \mathcal {B}} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, g _ {k} ^ {\ell + 1}\right) - \mathbb {E} _ {z \sim \mathcal {N} \left(0, \sigma_ {n} ^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) \right] \\ = \mathbb {E} _ {\mathcal {B}} \left[ \mathbb {E} _ {z \sim \mathcal {N} \left(0, \sigma_ {n} ^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) - \mathbb {E} _ {z \sim \mathcal {N} \left(0, \sigma_ {n} ^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) \right] \\ = \mathbb {E} _ {\mathcal {B}} [ 0 ] = 0. \\ \end{array}
$$
Similarly, we obtain  $\mathbb{E}Z_kZ_{k'} = \delta_{kk'}\mathbb{E}|Z_k|^2$ . Moreover, we can upper bound  $\mathbb{E}[Z_k|\mathcal{B}]^2$  as follows
$$
\begin{array}{l} \mathbb {E} \left[ Z _ {k} | \mathcal {B} \right] ^ {2} = \mathbb {E} _ {g ^ {\ell + 1} | \mathcal {B}} \left| \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, g _ {k} ^ {\ell + 1}\right) - \mathbb {E} _ {z \sim \mathcal {N} \left(0, \sigma_ {n} ^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) \right| ^ {2} \\ \leq 8 \mathbb {E} _ {z \sim \mathcal {N} \left(0, \sigma_ {n} ^ {2}\right)} \left| \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) \right| ^ {2}, \quad (a) \\ = 8 \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \left| \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, \sigma_ {n} z\right) \right| ^ {2} \\ \leq 8 \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} C _ {1} e ^ {2 C _ {2} \left(\sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right| + \sigma_ {n} | z |\right)}, \quad \Phi \text {i s c o n t r o l l a b l e} \\ = 8 C _ {1} e ^ {2 C _ {2} \sum_ {i = 1} ^ {\ell} | g _ {k} ^ {i} |} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} e ^ {2 C _ {2} \sigma_ {n} | z |} \\ \leq 8 C _ {1} e ^ {2 C _ {2} \sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right|} e ^ {2 C _ {2} ^ {2} \sigma_ {n} ^ {2}}. \\ \end{array}
$$
where  $(a)$  is due to maximal and Jensen's inequality.
Since  $e^{2C_2\sum_{i = 1}^{\ell}|g_k^i|}$  is controllable and  $\sigma_n\xrightarrow{a.s.}\sigma$  , it follows from the inductive hypothesis that
$$
\frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} \left[ Z _ {k} | \mathcal {B} \right] ^ {2} \leq 8 C _ {1} \cdot \left(\frac {1}{n} \sum_ {k = 1} ^ {n} e ^ {2 C _ {2} \sum_ {i = 1} ^ {\ell} | g _ {k} ^ {i} |}\right) \cdot e ^ {2 C _ {2} ^ {2} \sigma_ {n} ^ {2}} \xrightarrow {a . s .} 8 C _ {1} \mathbb {E} e ^ {2 C _ {2} \sum_ {i = 1} ^ {\ell} | z _ {i} |} \cdot e ^ {2 C _ {2} ^ {2} \sigma_ {2} ^ {2}}.
$$
As the RHS is a deterministic constant, we have
$$
\frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} \left[ Z _ {k} | \mathcal {B} \right] ^ {2} \in o (n ^ {\rho}), \quad \forall \rho > 0.
$$
or equivalently,  $\frac{1}{n}\sum_{k = 1}^{n}\mathbb{E}[Z_k|\mathcal{B}]^2\leq n^\rho$  for large enough  $n$ .
Now, we will first show  $A_{n^2} \stackrel{a.8}{\to} 0$ . For any  $\epsilon > 0$ , we have for large enough  $n$
$$
\begin{array}{l} \mathbb {P} \left(\left| A _ {n ^ {2}} \right| \geq \epsilon\right) \leq \epsilon^ {- 2} n ^ {- 4} \mathbb {E} \left| A _ {n ^ {2}} \right| ^ {2} \\ = \epsilon^ {- 2} n ^ {- 4} \sum_ {k, k ^ {\prime} = 1} ^ {n ^ {2}} \mathbb {E} \left[ Z _ {k} Z _ {k ^ {\prime}} \right] \\ = \epsilon^ {- 2} n ^ {- 4} \sum_ {k = 1} ^ {n ^ {2}} \mathbb {E} \left| Z _ {k} \right| ^ {2} \\ = \epsilon^ {- 2} n ^ {- 2} \mathbb {E} _ {\mathcal {B}} \left[ \frac {1}{n ^ {2}} \sum_ {k = 1} ^ {n ^ {2}} \mathbb {E} | Z _ {k} | \mathcal {B} | ^ {2} \right] \\ = \epsilon^ {- 2} n ^ {- 2} \mathbb {E} _ {B} \left[ n ^ {2 \rho} \right] \\ <   \epsilon^ {- 2} n ^ {- 2 + 2 \rho}. \\ \end{array}
$$
Furthermore, we obtain
$$
\sum_ {n = 1} ^ {\infty} \mathbb {P} \left(\left| A _ {n ^ {2}} \right| \geq \epsilon\right) \leq \sum_ {n = 1} ^ {\infty} \epsilon^ {- 2} n ^ {- 2 + 2 \rho} <   \infty ,
$$
provided we choose  $0 < \rho < 1/2$ . Thus, it follows from Borel-Cantelli lemma that  $A_{n^2} \stackrel{a.s.}{\to} 0$ .
Now for each  $n$ , we define  $k_{n} \coloneqq \sup \{k \in \mathbb{N} : k^{2} \leq n\}$ , then we have  $k_{n}^{2} \leq n \leq (k_{n} + 1)^{2}$ . Note that
$$
A _ {n} = \frac {1}{n} \sum_ {i = 1} ^ {n} Z _ {i} = \frac {1}{n} \sum_ {i = 1} ^ {k _ {n} ^ {2}} Z _ {i} + \frac {1}{n} \sum_ {i = k _ {n} ^ {2} + 1} ^ {n} Z _ {i}.
$$
We will show the two terms goes 0 a.s.. As we just proved, the first term goes to 0 a.s., since
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {k _ {n} ^ {2}} Z _ {i} \right| \leq \left| \frac {1}{k _ {n} ^ {2}} \sum_ {i = 1} ^ {k _ {n} ^ {2}} Z _ {i} \right| \stackrel {{a. s.}} {\Rightarrow} 0.
$$
For the second term, let  $T_{n} \coloneqq \frac{1}{n}\sum_{i = k_{n}^{2} + 1}^{n}Z_{i}$ , then for  $n$  large enough
$$
\begin{array}{l} \mathbb {P} (| T _ {n} | \geq \epsilon) \leq \epsilon^ {- 2} n ^ {- 2} \sum_ {i = k _ {n} ^ {2} + 1} ^ {n} \mathbb {E} Z _ {i} ^ {2} \\ \leq \epsilon^ {- 2} k _ {n} ^ {- 4} \sum_ {i = k _ {n} ^ {2} + 1} ^ {n} \mathbb {E} Z _ {i} ^ {2} \\ \leq \epsilon^ {- 2} k _ {n} ^ {- 4} \left(n - k _ {n} ^ {2}\right) \left(\frac {1}{n - k _ {n} ^ {2}} \sum_ {i = k _ {n} ^ {2} + 1} ^ {n} \mathbb {E} Z _ {i} ^ {2}\right) \\ \le \epsilon^ {- 2} k _ {n} ^ {- 4} \left(n - k _ {n} ^ {2}\right) ^ {1 + \rho} \\ \leq C \epsilon^ {- 2} k _ {n} ^ {- 4} (2 k _ {n} + 1) ^ {1 + \rho} \\ \leq C \epsilon^ {- 2} k _ {n} ^ {- 3 + \rho} \\ \end{array}
$$
where  $C$  is some fixed constant. Then we have
$$
\begin{array}{l} \sum_ {n = 1} ^ {\infty} \mathbb {P} (| T _ {n} | \geq \epsilon) \leq \sum_ {n = 1} ^ {\infty} C \epsilon^ {- 2} k _ {n} ^ {- 3 + \rho} \\ \leq \sum_ {n = 1} ^ {\infty} C \epsilon^ {- 2} (\sqrt {n} - 1) ^ {- 3 + \rho} \\ \leq \sum_ {n = 1} ^ {4} C \epsilon^ {- 2} (\sqrt {n} - 1) ^ {- 3 + \rho} + 2 C \epsilon^ {- 2} \sum_ {n = 4} ^ {\infty} n ^ {- (3 - \rho) / 2} \\ <   \infty , \\ \end{array}
$$
provided we choose  $0 < \rho < 1$ . Therefore, by choosing  $0 < \rho < 1/2$ , it follows from Borel-Cantelli lemma that  $T_{n} \stackrel{a.s.}{\to} 0$  and further  $A_{n} \stackrel{a.s.}{\to} 0$ .
$B_{n}$  converges to 0 almost surely
First of all, we will show  $\sigma > 0$  by which we can use Gaussian smoothing to show  $B_{n} \xrightarrow{a.s.} 0$ .
Lemma B.3. For  $\ell \geq 1$ , if  $\Sigma(z^{\ell}, z^{\ell}) > 0$ , then  $\Sigma(z^{\ell + 1}, z^{\ell + 1}) > 0$ .
Proof. We prove by contradiction. Assume  $\Sigma(z^{\ell + 1}, z^{\ell + 1}) = 0$ . Then we have
$$
0 = \Sigma (z ^ {\ell + 1}, z ^ {\ell + 1}) = \mathbb {E} \phi (z ^ {\ell} + z ^ {1}) ^ {2} = \mathbb {E} \phi (u ^ {\ell}) ^ {2},
$$
where  $u^{\ell} \sim \mathcal{N}(0, \Sigma(z^{\ell}, z^{\ell}) + \|x\|^{2}/n_{in})$ . It implies  $\phi(z) = 0$  almost surely, but it contradicts  $\phi$  is non-constant function since  $\Sigma(z^{\ell}, z^{\ell}) + \|x\|^{2}/n_{in} > 0$ .
It follows from Lemma B.3 that  $\sigma > 0$ . Then  $\sigma_n \stackrel{a.s.}{\to} \sigma$ , we have  $\sigma_n \geq \sigma / 2 > 0$  eventually, almost surely. To use Gaussian smoothing, we define the following functions
$$
f _ {k} (x) := \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, x\right), \quad F _ {k} (\sigma) := \mathbb {E} _ {z \sim \mathcal {N} (0, \sigma^ {2})} f _ {k} (z).
$$
By using Gaussian smoothing, we have for large enough  $n$
$$
\begin{array}{l} | B _ {n} | \leq \frac {1}{n} \sum_ {k = 1} ^ {n} | F _ {k} (\sigma_ {n}) - F _ {k} (\sigma) | \\ \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \int_ {\sigma} ^ {\sigma_ {n}} | F _ {k} ^ {\prime} (t) | d t, \quad \text {a s s u m e} \sigma \leq \sigma_ {n} \\ \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \int_ {\sigma} ^ {\sigma_ {n}} t ^ {- 1} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} | f _ {k} (t z) (t ^ {2} - 1) | d t, \quad (a) \\ \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \int_ {\sigma} ^ {\sigma_ {n}} t ^ {- 1} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} C _ {1} e ^ {C _ {2} \sum_ {i = 1} ^ {\ell} | g _ {k} ^ {i} | + C _ {2} t | z | + t} d t, \quad (b) \\ \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \int_ {\sigma} ^ {\sigma_ {n}} t ^ {- 1} C _ {1} e ^ {C _ {2} \sum_ {i = 1} ^ {\ell} | g _ {k} ^ {i} | + C _ {2} t ^ {2} / 2 + t} d t, \quad (c) \\ = C _ {1} \left(\frac {1}{n} \sum_ {k = 1} ^ {n} e ^ {C _ {2} \sum_ {i = 1} ^ {\ell} | g _ {k} ^ {i} |}\right) (\alpha (\sigma_ {n}) - \alpha (\sigma)) \\ \end{array}
$$
where  $(a)$  is by Lemma A.2 and Jensen's inequality,  $(b)$  is because  $f_{k}$  is controllable since  $\Phi$  is,  $(c)$  is by Lemma A.6, and  $\alpha (t)$  is the anti-derivative of the function  $\dot{\alpha} (t) = t^{-1}C_1e^{C_2t^2 /2 + t}$ . Here,  $\dot{\alpha} (t)$  is continuous, so that  $\alpha (t)$  is well-defined and continuous. Since  $e^{C_2\sum_{i = 1}^{\ell}|g_k^i|}$  is controllable, it follows from result for the basic case that
$$
\frac {1}{n} \sum_ {k = 1} ^ {n} e ^ {C _ {2} \sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right|} \xrightarrow {a . s _ {i}} \mathbb {E} _ {z \sim \mathcal {N} (0, \Sigma | g ^ {1})} e ^ {C _ {2} \sum_ {i = 1} ^ {\ell} \left| z ^ {i} \right|}.
$$
Since  $\sigma_n \stackrel{a.s.}{\to} \sigma$  and  $\alpha$  is continuous, it follows from Lemma A.1 that  $\alpha(\sigma_n) \stackrel{a.s.}{\to} \alpha(\sigma)$  and further
$$
\left| B _ {n} \right| \leq C _ {1} \left(\frac {1}{n} \sum_ {k = 1} ^ {n} e ^ {C _ {2} \sum_ {i = 1} ^ {\ell} | g _ {k} ^ {i} |}\right) (\alpha (\sigma_ {n}) - \alpha (\sigma)) \xrightarrow {a . s ;} 0.
$$
$C_n$  converges to 0 almost surely
Define function  $\hat{\Phi}(z^1, \dots, z^\ell) \coloneqq \mathbb{E}_{z \sim \mathcal{N}(0,1)} \Phi(z^1, \dots, z^\ell, \sigma z)$ . Since  $\Phi$  is controllable,  $\hat{\Phi}$  is also a controllable function. Then it follows from the inductive hypothesis that
$$
\begin{array}{l} \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} (0, \sigma^ {2})} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) = \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, \sigma z\right) \\ = \frac {1}{n} \sum_ {k = 1} ^ {n} \hat {\Phi} \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}\right) \\ \stackrel {a. s.} {\rightarrow} \mathbb {E} \left[ \hat {\Phi} \left(z ^ {1}, \dots , z ^ {\ell}\right)\right] \\ = \mathbb {E} \left[ \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \Phi \left(z ^ {1}, \dots , z ^ {\ell}, \sigma z\right) \right] \\ = \mathbb {E} \left[ \Phi \left(z ^ {1}, \dots , z ^ {\ell}, z ^ {\ell + 1}\right) \right] \\ \end{array}
$$
Thus,  $C_n \xrightarrow{a \cdot \xi} 0$ .
B.4 Proof of Lemma B.1: general case for shared matrices
Now in this section, we prove the desired result when the weight matrices are shared, i.e.,  $W^{\ell} = W$ . Assume the result holds for  $\ell$ , then we will show the desired result still holds for  $\ell + 1$ . Note that
As  $W$  is used before, we have
$$
g ^ {i} = W h ^ {i - 1}, \quad \forall i \in [ \ell ].
$$
Then define
$$
G := \left[ \begin{array}{l l l l} g ^ {1} & g ^ {2} & \dots & g ^ {\ell} \end{array} \right] \in \mathbb {R} ^ {n \times \ell}, \quad H := \left[ \begin{array}{l l l l} h ^ {0} & h ^ {1} & \dots & h ^ {\ell - 1} \end{array} \right] \in \mathbb {R} ^ {n \times \ell}. \tag {33}
$$
Then we have  $G = WH$ . Let  $\mathcal{B}$  be the  $\sigma$ -algebra spanned by all previous  $g^1, g^2, \dots, g^\ell$ . To obtain the conditional distribution of  $g^{\ell + 1}$  on  $\mathcal{B}$ , we first compute the conditional distribution of  $W$  on  $\mathcal{B}$ . It follows from Lemma A.3 that
$$
\begin{array}{l} W | \mathcal {B} = G (H ^ {T} H) ^ {\dagger} H ^ {T} + \tilde {W} \Pi_ {H} ^ {T} \\ \sim \mathcal {M N} \left(G \left(H ^ {T} H\right) ^ {\dagger} H ^ {T}, I _ {n}, \Pi_ {H} \Pi_ {H} ^ {T} / n\right) \\ \end{array}
$$
where  $\Pi = I_n - HH^\dagger$  is the orthogonal projection onto  $\mathrm{null}(H^T)$ , respectively. Therefore, we obtain
$$
g ^ {\ell + 1} | \mathcal {B} \sim \mathcal {N} \left(G \left(H ^ {T} H\right) ^ {\dagger} H ^ {T} h ^ {\ell}, \| \Pi^ {T} h ^ {\ell} \| ^ {2} / n I _ {n}\right)
$$
$$
g _ {k} ^ {\ell + 1} \left| \mathcal {B} \right. \stackrel {\text {i n d e p e n d e n t}} {\sim} \mathcal {N} \left(G _ {k} \left(H ^ {T} H\right) ^ {\dagger} H ^ {T} h ^ {\ell}, \| \Pi^ {T} h ^ {\ell} \| ^ {2} / n\right),
$$
where  $G_{k}\in \mathbb{R}^{1\times \ell}$  is the  $k$  -th row of  $G$
Since the activation function  $\phi$  is controllable by Lemma A.5, it follows from the inductive hypothesis that
$$
\left(h ^ {i}\right) ^ {T} \left(h ^ {j}\right) / n = \frac {1}{n} \sum_ {k = 1} ^ {n} \phi \left(g _ {k} ^ {i} + g _ {k} ^ {1}\right) \phi \left(g _ {k} ^ {j} + g _ {k} ^ {1}\right) \xrightarrow {a . s _ {\mathrm {i}}} \mathbb {E} \phi \left(z ^ {i} + z ^ {1}\right) \phi \left(z ^ {j} + z ^ {1}\right) = \Sigma \left(z ^ {i + 1}, z ^ {j + 1}\right) \quad \forall i, j.
$$
Then we have as  $n\to \infty$
$$
H ^ {T} H / n \xrightarrow {a . s _ {\cdot}} \Sigma (Z ^ {\ell}, Z ^ {\ell})
$$
$$
H ^ {T} h ^ {\ell} / n \xrightarrow {a . s .} \Sigma (Z ^ {\ell}, z ^ {\ell + 1})
$$
where  $Z^{\ell} = [z^{1}\dots z^{\ell}]^{T}\in \mathbb{R}^{\ell \times 1}$ . Since (pseudo-)inverse is continuous function, we further obtain
$$
v _ {n} := \left(H ^ {T} H\right) ^ {\dagger} H ^ {T} h ^ {\ell} = \left(H ^ {T} H / n\right) ^ {\dagger} H ^ {T} h ^ {\ell} / n \xrightarrow {a . s .} \Sigma \left(Z ^ {\ell}, Z ^ {\ell}\right) ^ {\dagger} \Sigma \left(Z ^ {\ell}, z ^ {\ell + 1}\right) := v. \tag {34}
$$
By using the equality  $HH^{\dagger} = H(H^{T}H)^{\dagger}H^{T}$ , we have
$$
\begin{array}{l} \left\| \Pi^ {T} h ^ {\ell} \right\| ^ {2} / n = \frac {1}{n} \left(h ^ {\ell}\right) ^ {T} \left(I _ {n} - H H ^ {\dagger}\right) ^ {2} h ^ {\ell} \\ = \frac {1}{n} \left(h ^ {\ell}\right) ^ {T} \left(I _ {n} - H H ^ {\dagger}\right) h ^ {\ell} \\ = \frac {1}{n} \left(h ^ {\ell}\right) ^ {T} h ^ {\ell} - \left(\left(h ^ {\ell}\right) ^ {T} H / n\right) \left(H ^ {T} H / n\right) ^ {\dagger} \left(H ^ {T} h ^ {\ell} / n\right) \\ \xrightarrow {a . s .} \Sigma (z ^ {\ell + 1}, z ^ {\ell + 1}) - \Sigma (z ^ {\ell + 1}, Z ^ {\ell}) \Sigma (Z ^ {\ell}, Z ^ {\ell}) ^ {\dagger} \Sigma (Z ^ {\ell}, z ^ {\ell + 1}). \\ \end{array}
$$
By using triangular inequality, we have
$$
\left| \frac {1}{n} \sum_ {k = 1} ^ {n} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, g _ {k} ^ {\ell + 1}\right) - \mathbb {E} \left[ \Phi \left(z ^ {1}, \dots ,, z ^ {\ell + 1}\right) \right] \right| \leq | A _ {n} | + | B _ {n} | + | C _ {n} | + | D _ {n} |,
$$
$$
A _ {n} = \frac {1}{n} \sum_ {k = 1} ^ {n} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, g _ {k} ^ {\ell + 1}\right) - \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} \left(\mu_ {k, n}, \sigma_ {n} ^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) \tag {35}
$$
$$
B _ {n} = \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} \left(\mu_ {k, n}, \sigma_ {n} ^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) - \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} \left(\mu_ {k, n}, \sigma^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) \tag {36}
$$
$$
C _ {n} = \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} \left(\mu_ {k, n}, \sigma^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) - \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} \left(\mu_ {k}, \sigma^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) \tag {37}
$$
$$
D _ {n} = \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} \left(\mu_ {k}, \sigma^ {2}\right)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) - \mathbb {E} \left[ \Phi \left(z ^ {1}, \dots , z ^ {\ell + 1}\right) \right] \tag {38}
$$
$$
\mu_ {k, n} = G _ {k} ^ {\ell} \left(H ^ {T} H\right) ^ {\dagger} H ^ {T} h _ {\ell} = G _ {k} ^ {\ell} v _ {n}, \tag {39}
$$
$$
\mu_ {k} = G _ {k} ^ {\ell} \Sigma \left(Z ^ {\ell}, Z ^ {\ell}\right) ^ {\dagger} \Sigma \left(Z ^ {\ell}, z ^ {\ell + 1}\right) = G _ {k} ^ {\ell} v, \tag {40}
$$
$$
\sigma_ {n} ^ {2} = \left\| \Pi^ {T} h ^ {\ell} \right\| ^ {2} \tag {41}
$$
$$
\sigma^ {2} = \Sigma \left(z ^ {\ell + 1}, z ^ {\ell + 1}\right) - \Sigma \left(z ^ {\ell + 1}, Z ^ {\ell}\right) \Sigma \left(Z ^ {\ell}, Z ^ {\ell}\right) ^ {\dagger} \Sigma \left(Z ^ {\ell}, z ^ {\ell + 1}\right). \tag {42}
$$
B.4.1  $A_{n}$  converges to 0 almost surely