SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Define random variables  $Z_{k} = \Phi (g_{k}^{1},\dots ,g_{k}^{\ell},g_{k}^{\ell +1}) - \mathbb{E}_{z\sim \mathcal{N}(\mu_{k,n},\sigma_{n}^{2})}\Phi (g_{k}^{1},\dots ,g_{k}^{\ell},z)$ . As  $X_{k}|\mathcal{B}$  are independent, we can easily show  $X_{k}$  are centered and uncorrelated. By using Jensen's inequality,  $Z_{k}^{2}|\mathcal{B}$  can be upper bounded as follows
$$
\mathbb {E} \left[ Z _ {k} ^ {2} | \mathcal {B} \right] \leq 8 \mathbb {E} _ {z \sim \mathcal {N} \left(\mu_ {k, n}, \sigma_ {n} ^ {2}\right)} \left| \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z\right) \right| ^ {2} \leq 8 C _ {1} e ^ {2 C _ {2} \sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right|} e ^ {2 C _ {2} \left| \mu_ {k, n} \right|} e ^ {2 C _ {2} ^ {2} \sigma_ {n} ^ {2}} \tag {43}
$$
As  $v_n \stackrel{a.s.}{\to} v$  by equation (34), we have  $\| v_n \| \leq 1 + \| v \|$ , eventually, almost surely. Thus, for large enough  $n$ , we have
$$
\left| \mu_ {k, n} \right| = \left| G _ {k} ^ {\ell} \left(H ^ {T} H\right) ^ {\dagger} \left(H ^ {T} h ^ {\ell}\right) \right| = \left| \sum_ {i = 1} ^ {\ell} v _ {n, i} g _ {k} ^ {i} \right| \leq \left(\| v \| + 1\right) \sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right|, \tag {44}
$$
where we also use the Cauchy-Schwartz inequality and square root inequality. It follows from equation (43) that
$$
\mathbb {E} \left[ Z _ {k} ^ {2} | \mathcal {B} \right] \leq 8 C _ {1} e ^ {(2 C _ {2} + \| v \| + 1) \sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right|} e ^ {2 C _ {2} ^ {2} \sigma_ {n} ^ {2}} = \hat {\Phi} (g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}) \cdot e ^ {2 C _ {2} ^ {2} \sigma_ {n} ^ {2}},
$$
where  $\hat{\Phi}(x^1, \dots, x^\ell) \coloneqq 8C_1e^{(2C_2 + ||v|| + 1)\sum_{i=1}^\ell |x_i|}$  is clearly a controllable function. It follows from inductive hypothesis and some basic properties of almost surely convergence in Lemma A.1 that
$$
\frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} \left[ Z _ {k} ^ {2} | \mathcal {B} \right] \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \hat {\Phi} (g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}) \cdot e ^ {2 C _ {2} ^ {2} \sigma_ {n} ^ {2}} \xrightarrow {a . s .} \mathbb {E} \left[ \hat {\Phi} (z ^ {1}, \dots , z ^ {\ell}) \right] \cdot e ^ {2 C _ {2} ^ {2} \sigma^ {2}}.
$$
As RHS is a deterministic constant, we have  $\frac{1}{n}\sum_{k = 1}^{n}\mathbb{E}\left[X_k^2 |\mathcal{B}\right]\in o(n^\rho)$  for all  $\rho >0$ . Then by using the same argument provided in Section B.3, we have  $A_{n}\stackrel {a.s.}{\to}0$
$$
\mathbf {I f} \sigma > 0
$$
In this subsection, we assume  $\sigma > 0$ . In addition, since  $\sigma_n \stackrel{a.s.}{\to} \sigma$ , we have  $\sigma_n \geq \sigma / 2 > 0$  almost surely for large enough  $n$ .
We can obtain the desired result  $B_{n} \stackrel{a.s.}{\rightarrow} 0$  by applying the same argument in Section B.3 to functions  $f_{k}$  and  $F_{k}$  redefined as follows
$$
f _ {k} (x) := \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, x\right), \quad F _ {k} (\sigma) := \mathbb {E} _ {z \sim \mathcal {N} \left(\mu_ {k, n}, \sigma^ {2}\right)} f _ {k} (z).
$$
By using Gaussian smoothing, for large enough  $n$ , we have
$$
\begin{array}{l} \left| B _ {n} \right| \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \left| F _ {k} \left(\sigma_ {n}\right) - F _ {k} (\sigma) \right| \\ \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \int_ {\sigma} ^ {\sigma_ {n}} | F _ {k} ^ {\prime} (t) | d t, \quad \text {a s s u m e} \sigma \leq \sigma_ {n} \\ \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \int_ {\sigma} ^ {\sigma_ {n}} t ^ {- 1} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \left| f _ {k} \left(\mu_ {k, n} + t z\right) \left(t ^ {2} - 1\right) \right| d t, \quad (a) \\ \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \int_ {\sigma} ^ {\sigma_ {n}} t ^ {- 1} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} C _ {1} e ^ {(C _ {2} + \| v \| + 1) \sum_ {i = 1} ^ {\ell} | g _ {k} ^ {i} | + C _ {2} t | z | + t} d t, \quad (b) \\ \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \int_ {\sigma} ^ {\sigma_ {n}} t ^ {- 1} C _ {1} e ^ {(C _ {2} + \| v \| + 1) \sum_ {i = 1} ^ {\ell} | g _ {k} ^ {i} | + C _ {2} t ^ {2} / 2 + t} d t, \quad (c) \\ = C _ {1} \left(\frac {1}{n} \sum_ {k = 1} ^ {n} e ^ {\left(C _ {2} + \| v \| + 1\right) \sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right|}\right) (\alpha (\sigma_ {n}) - \alpha (\sigma)), \\ \end{array}
$$
where  $(a)$  is by Lemma A.2,  $(b)$  is because  $f_{k}$  is controllable since  $\Phi$  is,  $(c)$  is by Lemma A.6 and equation (44), and  $\alpha(t)$  is the anti-derivative of the function  $\dot{\alpha}(t) = t^{-1}C_1e^{C_2t^2/2 + t}$ . Here,  $\dot{\alpha}(t)$  is continuous, so that  $\alpha(t)$  is well-defined and continuous. Since  $e^{C\sum_{i=1}^{\ell}|g_k^i|}$  is controllable for any constant  $C$ , it follows from the inductive hypothesis that
$$
\frac {1}{n} \sum_ {k = 1} ^ {n} e ^ {\left(C _ {2} + \| v \| + 1\right) \sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right|} \xrightarrow {a . s _ {;} \mathbb {E}} \left[ e ^ {\left(C _ {2} + \| v \| + 1\right) \sum_ {i = 1} ^ {\ell} \left| z _ {i} \right|} \right] <   \infty .
$$
$$
\left| B _ {n} \right| \leq C _ {1} \left(\frac {1}{n} \sum_ {k = 1} ^ {n} e ^ {\left(C _ {2} + \| v \| + 1\right) \sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right|}\right) (\alpha (\sigma_ {n}) - \alpha (\sigma)) \xrightarrow {a . s .} 0.
$$
$$
\mathbf {I f} \sigma = 0
$$
In this subsection, we consider when  $\sigma = 0$ . Note that the argument in the case  $\sigma > 0$  also holds if  $\sigma = 0$  and  $\sigma_{n} \neq 0$  (infinitely often), because the derivatives  $F_{k}^{\prime}(t)$  are well-defined if either  $\sigma > 0$  or  $\sigma_{n} > 0$ . Thus, we only need to analyze the case where  $\sigma = 0$  and  $\sigma_{n} = 0$  eventually.
For  $\sigma = 0$ , we have  $\Sigma(z^{\ell + 1}, z^{\ell + 1}) = \Sigma(z^{\ell + 1}, Z^{\ell}) \Sigma(Z^{\ell}, Z^{\ell})^{\dagger} \Sigma(Z^{\ell}, z^{\ell + 1})$ . By Lemma A.4, we have
$$
z ^ {\ell + 1} = \Sigma (z ^ {\ell + 1}, Z ^ {\ell}) \Sigma (Z ^ {\ell}, Z ^ {\ell}) ^ {\dagger} Z ^ {\ell} = v Z ^ {\ell}, \quad a. s.
$$
For controllable  $\Phi$ , we can show the function  $\hat{\Phi}:(g_k^1,\dots ,g_k^\ell)\mapsto \Phi (g_k^1,\dots ,g_k^\ell ,G_k^\ell v_n)$  is also controllable as follows
$$
\begin{array}{l} \left| \hat {\Phi} \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}\right) \right| = \left| \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, G _ {k} ^ {\ell} v _ {n}\right) \right| \\ \leq C _ {1} e ^ {C _ {2} \sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right| + C _ {2} \left| \sum_ {i = 1} ^ {\ell} v _ {n, i} g _ {k} ^ {i} \right|} \\ \leq C _ {1} e ^ {(2 C _ {2} + \| v \| + 1) \sum_ {i = 1} ^ {\ell} | g _ {k} ^ {i} |}, \\ \end{array}
$$
where the second inequality follows from equation (34). By using the inductive hypothesis, we obtain
$$
\begin{array}{l} \frac {1}{n} \sum_ {k = 1} ^ {n} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, G _ {k} ^ {\ell} v _ {n}\right) = \frac {1}{n} \sum_ {k = 1} ^ {n} \hat {\Phi} \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}\right) \\ \stackrel {a. s.} {\rightarrow} \mathbb {E} \left[ \hat {\Phi} (z ^ {1}, \dots , z ^ {\ell}) \right] \\ = \mathbb {E} \left[ \Phi (z ^ {1}, \dots , z ^ {\ell}, v Z ^ {\ell}) \right] \\ = \mathbb {E} \left[ \Phi \left(z ^ {1}, \dots , z ^ {\ell + 1}\right) \right]. \tag {45} \\ \end{array}
$$
Moreover, as we assume  $\sigma_{n} = 0$  for all large enough  $n$ , we obtain  $g_{k}^{\ell +1}|\mathcal{B} = G_{k}^{\ell}v_{n}$  almost surely. Then for large enough  $n$ , we obtain
$$
\frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} \left(\mu_ {k, n}, \sigma_ {n} ^ {2}\right)} \Phi \left(G _ {k} ^ {\ell}, z\right) = \frac {1}{n} \sum_ {k = 1} ^ {n} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, \mu_ {k, n}\right) = \frac {1}{n} \sum_ {k = 1} ^ {n} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, G _ {k} ^ {\ell} v _ {n}\right) \tag {46}
$$
Combining  $A_{n} \xrightarrow{a.s.} 0$  with equations (45) and (46) yields  $B_{n} \xrightarrow{a.s.} 0$ .
B.4.2  $C_n$  converges to 0 almost surely
As discussed in Section B.4.1, we can assume  $\sigma > 0$ . By using Gaussian smoothing again, we can easily show  $C_n \stackrel{a.s.}{\to} 0$  since  $\mu_{k,n} \stackrel{a.s.}{\to} \mu_k$ . Define functions
$$
f _ {k} (x) = \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, x\right), \quad F _ {k} (\mu) = \mathbb {E} _ {z \sim \mathcal {N} \left(\mu , \sigma^ {2}\right)} f _ {k} (z).
$$
It follows from Lemma A.2 that
$$
\begin{array}{l} | C _ {n} | \leq \frac {1}{n} \sum_ {k = 1} ^ {n} | F _ {k} (\mu_ {k, n}) - F _ {k} (\mu) | \\ \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \int_ {\mu_ {k}} ^ {\mu_ {k, n}} | F _ {k} ^ {\prime} (t) | d t, \quad \text {a s s u m e} \mu_ {k} \leq \mu_ {k, n} \\ \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \int_ {\mu_ {k}} ^ {\mu_ {k, n}} \frac {1}{\sigma} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} | f _ {k} (t + \sigma z) | | z | d t \\ \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \int_ {\mu_ {k}} ^ {\mu_ {k, n}} \frac {1}{\sigma} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} C _ {1} e ^ {C _ {2} \sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right| + C _ {2} t + (C _ {2} \sigma + 1) | z |} d t \\ \leq \frac {1}{\sigma} C _ {1} e ^ {(C _ {2} \sigma + 1) ^ {2} / 2} \cdot \frac {1}{n} \sum_ {k = 1} ^ {n} e ^ {C _ {2} \sum_ {i = 1} ^ {\ell} \left| g _ {k} ^ {i} \right|} \cdot \left[ \beta \left(\mu_ {k, n}\right) - \beta \left(\mu_ {k}\right) \right], \\ \end{array}
$$
where  $\beta (\mu)$  is the anti-derivative of the function  $\dot{\beta} (t) = e^{C_2t}$ . Here  $\beta$  is well-defined and continuous since  $\dot{\beta}$  is continuous. As  $\mu_{k,n}\stackrel {a.s.}{\rightarrow}\mu_k$ , it follows from inductive hypothesis and Lemma A.1 that  $C_n\stackrel {a.s.}{\rightarrow}0$ .
$D_{n}$  converges to 0 almost surely
In this section, we can show  $D_{n} \stackrel{a.s.}{\to} 0$  straightforward from the induction. Define functions
$$
\hat {\Phi} \left(z ^ {1}, \dots , z ^ {\ell}\right) := \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \left[ \Phi \left(z ^ {1}, \dots , z ^ {\ell}, \sum_ {i = 1} ^ {\ell} v _ {i} z _ {i} + \sigma z\right) \right].
$$
Here  $\hat{\Phi}$  is controllable as  $\Phi$  is. By applying the inductive hypothesis on  $\hat{\Phi}$ , we obtain
$$
\begin{array}{l} D _ {n} = \left| \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} (\mu_ {k}, \sigma^ {2})} \Phi (g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, z) - \mathbb {E} \left[ \Phi (z ^ {1}, \dots , z ^ {\ell + 1}) \right] \right| \\ = \left| \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, \mu_ {k} + \sigma z\right) - \mathbb {E} _ {z ^ {1}, \dots , z ^ {\ell}} \mathbb {E} _ {z ^ {\ell + 1} | z ^ {1}, \dots , z ^ {\ell}} \Phi \left(z ^ {1}, \dots , z ^ {\ell + 1}\right) \right| \\ = \left| \frac {1}{n} \sum_ {k = 1} ^ {n} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \Phi \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}, \mu_ {k} + \sigma z\right) - \mathbb {E} _ {z ^ {1}, \dots , z ^ {\ell}} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \Phi \left(z ^ {1}, \dots , \sigma z\right) \right| \\ = \left| \frac {1}{n} \sum_ {k = 1} ^ {n} \hat {\Phi} \left(g _ {k} ^ {1}, \dots , g _ {k} ^ {\ell}\right) - \mathbb {E} _ {z ^ {1}, \dots , z ^ {\ell}} \hat {\Phi} \left(z ^ {1}, \dots , z ^ {\ell}\right) \right| \\ \stackrel {a. s.} {\rightarrow} 0, \\ \end{array}
$$
where we use the fact  $\mu_{k} = G_{k}^{\ell}v = \sum_{i = 1}^{\ell}v_{i}g_{k}^{i}$
C Proof of Corollary 4.2
Define Gaussian random variables  $u^{\ell}(x)$  that is encoded by input  $x$  as follows for all  $\ell = [2, L - 1]$
$$
u ^ {1} (x) = z ^ {1} (x) \tag {47}
$$
$$
u ^ {\ell} (x) = z ^ {\ell} (x) + z ^ {1} (x). \tag {48}
$$
Then we can easily compute the corresponding covariance as follows for  $\ell \geq 2$
$$
\begin{array}{l} \operatorname {c o v} \left(u ^ {1} (x), u ^ {1} \left(x ^ {\prime}\right)\right) = \operatorname {c o v} \left(z ^ {1} (x), z ^ {1} \left(x ^ {\prime}\right)\right) \\ = \Sigma^ {1} (x, x ^ {\prime}) \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {c o v} \left(u ^ {\ell} (x), u ^ {\ell} \left(x ^ {\prime}\right)\right) = \operatorname {c o v} \left(z ^ {\ell} (x) + z ^ {1} (x), z ^ {\ell} \left(x ^ {\prime}\right) + z ^ {1} \left(x ^ {\prime}\right)\right) \\ = \operatorname {c o v} \left(z ^ {\ell} (x), z ^ {\ell} \left(x ^ {\prime}\right)\right) + \operatorname {c o v} \left(z ^ {1} (x), z ^ {1} \left(x ^ {\prime}\right)\right) \\ = \Sigma^ {\ell} (x, x ^ {\prime}) + \Sigma^ {1} (x, x ^ {\prime}) \\ \end{array}
$$
D Proof of Theorem 4.3
This section is deducted to prove the strict positive definiteness of  $\Sigma^L$ . We will prove it by using the notion of dual activation and Hermitian expansion.
Let  $x\sim \mathcal{N}(0,1)$  and  $f:\mathbb{R}\to \mathbb{R}$ . Then we can define an inner product
$$
\langle f, g \rangle := \mathbb {E} _ {x \sim \mathcal {N} (0, 1)} f (x) g (x).
$$
Thus, we define a Hilbert space of functions  $\mathcal{H}$ , that is,  $f\in \mathcal{H}$  if and only if
$$
\left\| f \right\| ^ {2} = \mathbb {E} _ {x \sim \mathcal {N} (0, 1)} \left| f (x) \right| ^ {2} <   \infty .
$$
Next, consider the function sequence  $1, x, x^2, \dots$ . Clearly, they are independent. Then apply Gram-Schmidt process to the function sequence w.r.t. the inner product we defined before, and we obtain  $\{h_n\}$  the (normalized) Hermite polynomial that is an orthonormal basis to the Hilbert space  $\mathcal{H}$ .
Now, we are ready to introduce dual activation. The dual activation  $\hat{\phi} : [-1, 1] \to \mathbb{R}$  of an activation  $\phi : \mathbb{R} \to \mathbb{R}$  is defined by
$$
\hat {\phi} (\rho) := \mathbb {E} _ {(X, Y) \sim \mathcal {N} _ {\rho}} \phi (X) \phi (Y). \tag {49}
$$
where  $\mathcal{N}_{\rho}$  is multidimensional Gaussian distribution with mean 0 and covariance matrix  $\begin{bmatrix} 1 & \rho \\ \rho & 1 \end{bmatrix}$ . Then the dual kernel  $k_{\phi}$  is given by
$$
k _ {\phi} (x, x ^ {\prime}) := \hat {\phi} (\langle x, x ^ {\prime} \rangle).
$$
If a function  $\phi \in \mathcal{H}$ , we not only can obtain an expansion by using the orthonormal basis of Hermitian polynomials but also an expansion to the dual activation  $\hat{\phi}$  by using the same Hermitian coefficients. As a consequence, the corresponding dual kernel  $k_{\phi}$  can be shown to be strict positive definite by using the Hermitian expansion.
Lemma D.1. [10, Lemma 12] If  $\phi \in \mathcal{H}$ , then
$$
\phi (x) = \sum_ {n = 0} ^ {\infty} a _ {n} h _ {n} (x), \tag {50}
$$
$$
\hat {\phi} (\rho) = \sum_ {n = 0} ^ {\infty} a _ {n} ^ {2} \rho^ {n}. \tag {51}
$$
where  $a_{n} \coloneqq \langle h_{n}, \phi \rangle$  is the Hermite coefficients, and the above is Hermitian expansion.
Theorem D.1. [23, Theorem 3][17, Theorem 1] For a function  $f: [-1,1] \to \mathbb{R}$  with  $f = \sum_{n=0}^{\infty} b_n h_n$ , the kernel  $K_f: S^{n_0-1} \times S^{n_0-1} \to \mathbb{R}$  defined by
$$
K _ {f} (x, x ^ {\prime}) := f \left(x ^ {T} x ^ {\prime}\right)
$$
is strictly positive definite for any  $n_0 \geq 1$  if and only if the coefficients  $b_n > 0$  for infinitely many even and odd integer  $n$ .
Now we are ready to prove the kernel or covariance function  $\Sigma^L$  is strictly positive definite by using Gaussian measure techniques on the existence of positive definiteness.
Lemma D.2. Suppose  $\phi$  is non-polynomial Lipschitz continuous. If  $\Sigma^{\ell}$  is strictly positive, then  $\Sigma^{\ell +1}$  is also strictly positive definite.
Proof. Assume the contrary. Then there exists a finite distinct collection  $\{x_{i}\}_{i = 1}^{n}$  and some constants  $\{c_i\}_{i = 1}^n$  such that
$$
0 = \sum_ {i, j = 1} ^ {n} c _ {i} c _ {j} \Sigma^ {\ell + 1} (x _ {i}, x _ {j}) = \mathbb {E} \left[ \sum_ {i = 1} ^ {n} c _ {i} \phi (u _ {i}) \right] ^ {2}.
$$
This indicates  $\sum_{i=1}^{n} c_i \phi(u_i) = 0$  almost surely. Note that we have the random variables  $(u_i, u_j)$  follows Gaussian distribution given by
$$
(u _ {i}, u _ {j}) \sim \mathcal {N} (0, A ^ {\ell} (x _ {i}, x _ {j})).
$$
WLOG, we can assume  $c_{1} \neq 0$ . Then for some  $\phi(u_{1}) \neq 0$ , we choose  $u_{1} = \dots = u_{n} = u_{2}$ . Then
$$
c _ {1} \phi (u _ {1}) + (c _ {2} + \dots + c _ {n}) \phi (u _ {1}) = 0,
$$
indicates  $c_{1} = -(c_{2} + \dots + c_{n})$ . Then for any  $u \neq u'$ , we have
$$
c _ {1} \phi (u) + (- c _ {1}) \phi \left(u ^ {\prime}\right) = 0
$$
This implies  $\phi(u) = \phi(u')$ , but it contradicts  $\phi$  is non-constant.
Lemma D.3. Suppose  $\phi$  is non-polynomial Lipschitz continuous. Then  $\Sigma^2$  is strictly positive definite.
Proof. For  $\ell = 2$ , we have
$$
\Sigma^ {2} (x, x ^ {\prime}) = \sigma_ {2} ^ {2} \mathbb {E} _ {(u, v) \sim \mathcal {N} \left(0, A ^ {1} \left(x, x ^ {\prime}\right)\right)} [ \phi (u) \phi (v) ],
$$
$$
A ^ {1} (x, x ^ {\prime}) = \left[ \begin{array}{c c} 1 & \langle x, x ^ {\prime} \rangle \\ \langle x ^ {\prime}, x \rangle & 1 \end{array} \right].
$$
$$
\Sigma^ {2} (x, x ^ {\prime}) = \sigma_ {2} ^ {2} \hat {\mu} (x ^ {T} x ^ {\prime})
$$
where  $\mu (x)\coloneqq \phi (x\sigma_{u})$
Clearly,  $\mu$  is Lipschitz continuous since  $\phi$  is. Let the expansion of  $\mu$  in Hermite polynomials  $\{h_n\}_{n = 0}^{\infty}$  to be given as  $\mu = \sum_{n = 0}^{\infty}a_{n}h_{n}$ . Then we can write  $\hat{\mu}$  as  $\hat{\mu} (\rho) = \sum_{n = 0}^{\infty}a_n^2\rho^n$ . Then we have
$$
\Sigma^ {2} (x, x ^ {\prime}) = \sigma_ {w} ^ {2} \hat {\mu} (x ^ {T} x ^ {\prime}) = \sigma_ {w} ^ {2} \sum_ {n = 0} ^ {\infty} a _ {n} ^ {2} (x ^ {T} x ^ {\prime}) ^ {n}.
$$
Since  $\phi$  is assumed non-polynomials,  $\mu$  is also non-polynomial, and so there are infinitely many number of nonzero  $a_{n}$  in the expansion. Thus,  $b_{n} := a_{n}^{2} > 0$  for infinitely many even and odd numbers. Since  $\sigma_w^2 > 0$ , we have  $\Sigma^2$  is strictly positive definite.
Then we obtain  $\Sigma^L$  is strict positive definite by combining Lemma D.2 and D.3
This section we show the limiting covariance function  $\Sigma^{*}$  is well defined. As each  $\Sigma^L$  satisfies Cauchy-Schwartz inequality, it suffices to show  $\Sigma^{*}(x,x)$  is well defined, which is given in Lemma E.1.
Lemma E.1. Choose  $\sigma_w > 0$  small for which  $\beta := \frac{\sigma_w^2}{2} \mathbb{E}|z|^2|z^2 - 1| < 1$ , where  $z$  is standard Gaussian random variable. Then we have for every  $x \in \mathbb{S}^{n_{in} - 1}$  and  $\ell \in [2, L]$
$$
\left| \Sigma^ {\ell + 1} (x, x) - \Sigma^ {\ell} (x, x) \right| \leq \beta \left| \Sigma^ {\ell} (x, x) - \Sigma^ {\ell - 1} (x, x) \right|. \tag {52}
$$
Therefore,  $\Sigma^{*}(x,x)\coloneqq \lim_{\ell \to \infty}\Sigma^{\ell}(x,x)$  exists uniquely and
$$
0 <   \Sigma^ {*} (x, x) \leq (1 + 1 / \beta) \Sigma^ {2} (x, x). \tag {53}
$$
Proof. Fix  $x$  and we denote  $\sigma_{\ell}^{2} \coloneqq \Sigma^{\ell}(x,x)$  to simplify the notation. Define function  $\Phi(\sigma) \coloneqq \mathbb{E}_{u \sim \mathcal{N}(0,\sigma^2)} \phi(u)^2$
$$
\begin{array}{l} \sigma_ {\ell + 1} ^ {2} - \sigma_ {\ell} ^ {2} = \sigma_ {w} ^ {2} \left(\mathbb {E} _ {u ^ {\ell + 1} \sim \mathcal {N} (0, \sigma_ {\ell} ^ {2} + \sigma_ {1} ^ {2})} \phi (u ^ {\ell + 1}) ^ {2} - \mathbb {E} _ {u ^ {\ell} \sim \mathcal {N} (0, \sigma_ {\ell - 1} ^ {2} + \sigma_ {1} ^ {2})} \phi (u ^ {\ell}) ^ {2}\right) \\ = \sigma_ {w} ^ {2} \left(\Phi \left(\sqrt {\sigma_ {\ell} ^ {2} + \sigma_ {1} ^ {2}}\right) - \Phi \left(\sqrt {\sigma_ {\ell - 1} ^ {2} + \sigma_ {1} ^ {2}}\right)\right) \\ = \sigma_ {w} ^ {2} \int_ {\sqrt {\sigma_ {\ell - 1} ^ {2} + \sigma_ {1} ^ {2}}} ^ {\sqrt {\sigma_ {\ell} ^ {2} + \sigma_ {1} ^ {2}}} \Phi^ {\prime} (t) d t \\ = \sigma_ {w} ^ {2} \int_ {\sqrt {\sigma_ {\ell - 1} ^ {2} + \sigma_ {1} ^ {2}}} ^ {\sqrt {\sigma_ {\ell} ^ {2} + \sigma_ {1} ^ {2}}} \frac {1}{t} \mathbb {E} _ {z} \phi (t z) ^ {2} (z ^ {2} - 1) d t \\ \leq \sigma_ {w} ^ {2} \int_ {\sqrt {\sigma_ {\ell - 1} ^ {2} + \sigma_ {1} ^ {2}}} ^ {\sqrt {\sigma_ {\ell} ^ {2} + \sigma_ {1} ^ {2}}} \frac {1}{t} \mathbb {E} _ {z} | t z | ^ {2} | z ^ {2} - 1 | d t \\ = \sigma_ {w} ^ {2} \mathbb {E} _ {z} | z | ^ {2} | z ^ {2} - 1 | \int_ {\sqrt {\sigma_ {\ell - 1} ^ {2} + \sigma_ {1} ^ {2}}} t d t \\ = \frac {\sigma_ {w} ^ {2} \mathbb {E} _ {z} | z | ^ {2} | z ^ {2} - 1 |}{2} \left| \sigma_ {\ell} ^ {2} - \sigma_ {\ell - 1} ^ {2} \right| \\ = \beta \left| \sigma_ {\ell} ^ {2} - \sigma_ {\ell - 1} ^ {2} \right|, \\ \end{array}
$$
where  $\beta := \frac{\sigma_w^2 \mathbb{E}_z |z|^2 |z^2 - 1|}{2}$ . As we choose  $\sigma_w$  small such that  $\beta < 1$ , then the mapping
$$
\sigma_ {\ell + 1} ^ {2} = \mathbb {E} _ {u \sim \mathcal {N} (0, \sigma_ {\ell} ^ {2} + \sigma_ {1} ^ {2})} [ \phi (u) ^ {2} ]
$$
is a contraction. Thus, it has unique fixed point  $\sigma_{*}$  such that
$$
\sigma_ {*} ^ {2} = \mathbb {E} _ {z \sim \mathcal {N} \left(0, \sigma_ {*} ^ {2} + \sigma_ {1} ^ {2}\right)} \phi (u) ^ {2}. \tag {54}
$$
In addition, let  $\tau_{\ell}^{2} = \sigma_{\ell}^{2} + \sigma_{1}^{2}$  and  $\tau_1^2 = \sigma_1^2$  , then we have
$$
\left| \tau_ {\ell + 1} ^ {2} - \tau_ {\ell} ^ {2} \right| = \left| \sigma_ {\ell + 1} ^ {2} - \sigma_ {\ell} ^ {2} \right| \leq \beta \left| \sigma_ {\ell} ^ {2} - \sigma_ {\ell - 1} ^ {2} \right| = \beta \left| \tau_ {\ell} ^ {2} - \tau_ {\ell - 1} ^ {2} \right|.
$$
Then we repeat this inequality for  $\ell$  times and obtain
$$
\left| \tau_ {\ell + 1} ^ {2} - \tau_ {\ell} ^ {2} \right| \leq \beta^ {\ell - 1} \left| \tau_ {2} ^ {2} - \tau_ {1} ^ {2} \right|.
$$
As LHS is  $\left|\sigma_{\ell +1}^{2} - \sigma_{\ell}^{2}\right|$  and RHS is  $\sigma_2^2$ , we obtain
$$
\left| \sigma_ {\ell + 1} ^ {2} - \sigma_ {\ell} ^ {2} \right| \leq \beta^ {\ell - 1} \sigma_ {2} ^ {2}.
$$
$$
\left| \sigma_ {\ell + 1} ^ {2} - \sigma_ {2} ^ {2} \right| \leq \sum_ {s = 2} ^ {\ell} \left| \sigma_ {s + 1} ^ {2} - \sigma_ {s} ^ {2} \right| \leq \sum_ {s = 2} ^ {\ell} \beta^ {s - 1} \sigma_ {2} ^ {2} \leq \frac {1}{\beta} \sigma_ {2} ^ {2}.
$$
$$
\sigma_ {\ell} ^ {2} \leq \left(1 + \frac {1}{\beta}\right) \sigma_ {2} ^ {2} <   \infty , \quad \forall \ell \geq 2.
$$
Now, suppose  $\sigma_{*} = 0$ , then we have equation
$$
\begin{array}{l} 0 = \sigma_ {*} ^ {2} \\ = \mathbb {E} _ {u \sim \mathcal {N} (0, \sigma_ {*} ^ {2} + \sigma_ {1} ^ {2})} \phi (u) ^ {2} \\ = \mathbb {E} _ {u \sim \mathcal {N} (0, \sigma_ {1} ^ {2})} \phi (u) ^ {2} \\ = \mathbb {E} _ {u \sim \mathcal {N} (0, 1)} \phi (u) ^ {2} \\ \end{array}
$$
where we use the fact  $\sigma_1^2 = 1$ . The equation above implies  $\phi(u) = 0$  almost surely, which is impossible since  $u$  follows standard Gaussian and  $\phi$  is nonconstant.
E.1  $\Sigma^{*}(x,x) = \Sigma^{*}(x^{\prime},x^{\prime})$
In this subsection, we will first show  $\Sigma^{\ell}(x,x) = \Sigma^{\ell}(x',x')$  for all  $x,x'$ . The desired result is obtained by letting  $\ell \to \infty$ .
Given  $x_{i}$  and  $x_{j}$ , let  $A_{ij}^{\ell} \coloneqq \Sigma^{\ell}(x_{i},x_{j})$ . We prove by induction. For the basic case, we have
$$
A _ {i i} ^ {1} = \mathbb {E} \left| \sigma (x _ {i} ^ {T} z) \right| ^ {2} = \mathbb {E} \left| \sigma (u _ {j} ^ {1}) \right| ^ {2} = \mathbb {E} \left| \sigma (u _ {j} ^ {1}) \right| ^ {2} = A _ {j j} ^ {1},
$$
where we use the fact  $u_{i}\stackrel {i.i.d.}{\sim}\mathcal{N}(0,1)$  due to  $\| x_i\| ^2 = 1$
Assume the result holds for  $\ell - 1$ . Then we will show the result for  $\ell$ . Note that
$$
\operatorname {V a r} \left(u _ {i} ^ {\ell - 1}\right) = A _ {i i} ^ {\ell - 1} + A _ {i i} ^ {1} = A _ {j j} ^ {\ell - 1} + A _ {j j} ^ {1} = \operatorname {V a r} \left(u _ {j} ^ {\ell - 1}\right),
$$
where the last equality holds from the inductive hypothesis. As each  $u_{i}^{\ell -1}$  is a centered Gaussian random variable, equal variance implies equal distribution. Then we obtain
$$
A _ {i i} ^ {\ell} = \mathbb {E} _ {u _ {i} ^ {\ell - 1} \sim \mathcal {N} (0, A _ {i i} ^ {\ell - 1} + A _ {i i} ^ {1})} \left| \sigma (u _ {i} ^ {\ell - 1}) \right| ^ {2} = \mathbb {E} _ {u _ {j} ^ {\ell - 1} \sim \mathcal {N} (0, A _ {j j} ^ {\ell - 1} + A _ {j j} ^ {1})} \left| \sigma (u _ {j} ^ {\ell}) \right| ^ {2} = A _ {j j} ^ {\ell}.
$$
Then let  $\ell \to \infty$  and we obtain the desired result.
F Proof of Lemma 4.1