SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| \boldsymbol {S} \| = \sup  _ {\| \boldsymbol {v} \| = 1} \operatorname {V a r} (\boldsymbol {v} ^ {\top} \boldsymbol {u} _ {i}) = \sup  _ {\| \boldsymbol {v} \| = 1, \boldsymbol {v} \perp \boldsymbol {\mu}} \operatorname {V a r} (\boldsymbol {v} ^ {\top} \boldsymbol {u} _ {i}) \\ \leq \sup  _ {\| \boldsymbol {v} \| = 1, \boldsymbol {v} \perp \boldsymbol {\mu}} \mathbb {E} [ (\boldsymbol {v} ^ {\top} \boldsymbol {u} _ {i}) ^ {2} ] = \sup  _ {\| \boldsymbol {v} \| = 1, \boldsymbol {v} \perp \boldsymbol {\mu}} \mathbb {E} [ (\boldsymbol {v} ^ {\top} \boldsymbol {P} \boldsymbol {z} _ {i}) ^ {2} ] \cdot \mathbb {E} [ \sigma_ {*} (\xi) ^ {2} ] \leq C, \\ \end{array}
$$
for some universal constant  $C > 0$ . Meanwhile,
$$
\begin{array}{l} \| \bar {u} _ {i} \| \leq \| P z _ {i} \| \cdot | \sigma_ {*} (\boldsymbol {\mu} ^ {\top} z _ {i}) | + \mathbb {E} [ \| P z _ {i} \| ] \mathbb {E} [ | \sigma_ {*} (\boldsymbol {\mu} ^ {\top} z _ {i}) | ] \\ \leq \left\| \boldsymbol {z} _ {i} \right\| \cdot \left| \sigma_ {*} \left(\boldsymbol {\mu} ^ {\top} \boldsymbol {z} _ {i}\right) \right| + \mathbb {E} \left[ \| \boldsymbol {z} _ {i} \| \right] \cdot \mathbb {E} \left[ \left| \sigma_ {*} (\xi) \right| \right] \\ \leq \left\| \boldsymbol {z} _ {i} \right\| \cdot \left| \sigma_ {*} \left(\boldsymbol {\mu} ^ {\top} \boldsymbol {z} _ {i}\right) \right| + C ^ {\prime} \sqrt {d}, \tag {B.13} \\ \end{array}
$$
where the inequality is due to the sub-Gaussian tail of the norm of the Gaussian random vector. Applying Theorem 3.1.1 of [Ver18], we know that  $\| z_i\| \leq c'\sqrt{d}$  almost surely, for some universal constant  $c' > 0$ . Since  $\sigma_{*}$  is a polynomial with degree  $p$ , we know that  $\| \sigma_{*}(\xi)\|_{\psi_{\alpha}}$  is uniformly bounded by some constant for  $\alpha = 1 / p$  and  $\xi \sim \mathcal{N}(0,1)$ . Hence,  $|\sigma_{*}(\xi)|\leq \log^{2p}(n)$  almost surely. Together with (B.13), we can conclude that  $\| \bar{\boldsymbol{u}}_i\| \leq C\log^{2p}(n)\sqrt{d}$  almost surely as  $n\to \infty$ . Therefore, by Theorem 5.44 in [Ver10], we have that with probability at least  $1 - n\exp (-cs^2)$ ,
$$
\frac {1}{\sqrt {n}} \| \boldsymbol {A} \| \leq (C + s \log^ {2 p} (n)) \sqrt {\frac {d}{n}}.
$$
Combining (B.12) and choosing  $q = \frac{1}{2}$  and  $t = \log^{4p}(n)\sqrt{\frac{d}{n}}$  in (B.11), one can easily obtain (B.9) for any  $\beta \in [0,1)$ . As a corollary of (B.9), applying Proposition 16, we can directly obtain (B.10). This completes the proof of the proposition.
B.3 Learning  $f_{*}$  with Neural Network Features
Univariate approximation. We first adapt an argument from [DLS22] showing that a ReLU nonlinearity can approximate a polynomial function in one dimension with the aid of random bias units.
Lemma 20 (Corollary 3 in [DLS22], adapted). Let  $\sigma(z) = \mathrm{ReLU}(z)$ ,  $s \sim \mathrm{Unif}(\{-1,1\})$  and  $b \sim \mathcal{N}(0,1)$ . Then given twice-differentiable  $f: \mathbb{R} \to \mathbb{R}$ , there exists some  $v(s,b)$  such that for any  $|z| \leq C$  with some constant  $C > 0$ ,
$$
\mathbb {E} [ v (s, b) \sigma (s z + b) ] = f (z), \quad a n d \sup  _ {s, b} | v (s, b) | \lesssim \exp (C).
$$
Proof. Let  $v(s, b) = \frac{e^{c^2 / 2}\sqrt{2\pi}}{1 - e^{-3c^2 / 2}}\mathbf{1}_{b \in [c, 2c]}$ , for any  $c > 0$ . Then we have
$$
\mathbb {E} [ v (s, b) \sigma (s z + b) ] = 1,
$$
for all  $|z|\leq c$ . Now let  $v(s,b) = \frac{s}{f(c)}\mathbf{1}_{b\in [c,2c]}$ , for any  $c > 0$ , where  $f(c)\coloneqq \int_{c}^{2c}\frac{1}{\sqrt{2\pi}} e^{-b^2 /2}db$ . For similar reasons, we may verify that
$$
\mathbb {E} [ v (s, b) \sigma (s z + b) ] = z,
$$
for all  $|z|\leq c$ . Let  $v(s,b) = 2\sqrt{2\pi} f^{\prime \prime}(-sb)e^{b^2 /2}\mathbf{1}_{b\in [0,c]}$ , for any  $c > 0$ , where  $f\in C^2 (\mathbb{R})$ . Then we can easily check that
$$
\mathbb {E} [ v (s, b) \sigma (s z + b) ] = f (z) + F (c) + G (c) z,
$$
for all  $|z|\leq c$  , where
$$
F (c) = c \left(f ^ {\prime} (- c) + f ^ {\prime} (c)\right) + f (0) - 2 f (c),
$$
$$
G (c) = f ^ {\prime} (0) - 2 f ^ {\prime} (c).
$$
Notice that this construction of  $v(s, b)$  approximates the function  $f(z)$  using some constant and linear function perturbations. Now recalling the first two cases, we know that we can approximate any constant and linear functions on this interval, and hence we obtain the desired claim by combining all three cases together.
B.3.1 Analysis of the Learned Feature Map
Recall the training procedure in Algorithm 1. The following lemma shows that after the first-layer parameters are optimized for one (normalized) gradient step, there exist some second-layer coefficients that achieve small prediction risk with high probability.
Proposition 21. Given fixed  $k \in \mathbb{N}$  and  $\beta > 1 - \frac{1}{k}$ , let  $W_{1}$  be the weight matrix optimized via one normalized gradient descent step defined in Algorithm 1, there exists some second-layer coefficients  $\tilde{\pmb{a}} \in \mathbb{R}^{N}$  such that  $\tilde{f}(\pmb{x}) = \frac{1}{\sqrt{N}} \sigma(W_{1}^{\top} \pmb{x} + \pmb{b}) \tilde{\pmb{a}} = \frac{1}{\sqrt{N}} \sum_{i=1}^{N} \tilde{a}_{i} \sigma(a_{i} \langle \pmb{x}, \pmb{w}_{i}^{(1)} \rangle + b_{i})$  achieves
$$
\left\| f _ {*} - \tilde {f} \right\| _ {L ^ {2}} ^ {2} \lesssim 1 / \log^ {C} d, a n d \| \tilde {\boldsymbol {a}} \| \lesssim \log^ {C} d,
$$
for some large constant  $C > 0$ , with probability  $1 - \exp \left(-c\log^2 N\right)$  when  $n, d \to \infty$  proportionally and  $N = \Omega(d^\epsilon)$ .
Proof. We follow an argument similar to  $\left[\mathrm{BES}^{+}22\right.$  (Appendix D] by selecting a suitable set of neurons that can approximate  $f_{*}$ . Throughout this proof, we take  $\varepsilon > 0$  to be some small nonvanishing constant and  $C > 0$  to be some large constant. At a high level, we select the neurons following the univariate approximation result in Lemma 20 to fit a polynomial target function. To this end, we need to show that after one gradient descent step, there exists a sufficiently large set of neurons  $\boldsymbol{w}_{i}^{(1)}$  that align with the signal direction, i.e.,  $\left|\sqrt{1 + \theta}\langle \boldsymbol{w}_{i}^{(1)},\boldsymbol{\mu}\rangle\right| \approx 1$ .
We first restrict ourselves to the "average-case" neurons around the equator  $\boldsymbol{w}_i$  where the indices  $i \in \mathcal{E}_{\tau}$ . Due to the Gaussian initialization, setting  $\tau = \log^2 d$  entails  $\mathbb{P}(i \notin \mathcal{E}_{\tau}) \lesssim \exp(-\log^2 d)$ .
Now we consider the subset of bias units that gives an "informative" gradient update, i.e., the  $k$ -th Hermite coefficient is not vanishing:  $\alpha_{k}^{b_{i}} = \Omega \left(\frac{1}{\log^{C}d}\right)$  for some large constant  $C > 0$ . Recall that Lemma 15 implies that for bias unit satisfying  $h_{p - 2}(b_i)\neq 0$ , the degree- $k$  Hermite coefficient of  $f(z) = \sigma (z + b_{i})$  is non-zero. Therefore, we define the index set
$$
\mathcal {B} _ {\varepsilon} = \left\{i \in [ N ] \mid \left| h _ {p - 2} \left(b _ {i}\right) \right| \geq \varepsilon , \text {i f} p \geq 2 \right\}, \tag {B.14}
$$
Notice that all the zeros of Hermite polynomials are real and distinct, see Theorem 3.3.1 in [Sze39]. Additionally, from Theorem 3.3.2 in [Sze39], the zeros of  $h_{p + 1}(x)$  are interlacing between the zeros of  $h_p(x)$ . From the recurrence relation of Hermite polynomials, namely,
$$
h _ {p + 1} (x) = x h _ {p} (x) - h _ {p} ^ {\prime} (x),
$$
we know that  $h_p'(x) \neq 0$  if  $x$  is a root of  $h_p$ . Thus, applying the implicit function theorem, we can deduce that by choosing  $\varepsilon \asymp \frac{1}{\log^C d}$  for some large  $C > 0$ , we attain  $\alpha_k^{b_i} = \Omega \left( \frac{1}{\log^C d} \right)$ , for any  $i \in \mathcal{B}_{\varepsilon}$ . Due to the Gaussian initialization of  $b_i$ , we know what the expected size of the set  $\mathbb{E}|\mathcal{B}_{\varepsilon}| = \Omega (N^{1 - \varepsilon'})$  for some small  $\varepsilon' > 0$ . Therefore, Hoeffding's inequality implies that for large enough  $N = \Omega (d^\epsilon)$ , we have  $\left| \frac{|\mathcal{B}_{\varepsilon}|}{\mathbb{E}|\mathcal{B}_{\varepsilon}|} - 1 \right| \leq N^{-1/2 + \varepsilon'}$  with probability  $1 - \exp(-c\log^2 N)$ .
Gradient approximation. For the subset of neurons with indices  $i \in \mathcal{B}_{\varepsilon}$ , we introduce a sequence of approximations to the gradient update given as
$$
g _ {n} (\pmb {w} _ {i} ^ {(0)}) = \frac {1}{n} \pmb {X} ^ {\top} \big (f _ {\mathrm {N N}} ^ {(0)} (\pmb {X}) - f _ {*} (\pmb {X}) \big) \odot \sigma^ {\prime} (\pmb {X} \pmb {w} _ {i} ^ {(0)} + b _ {i} \pmb {1} _ {n}),
$$
where we dropped the factor  $\frac{a_i}{\sqrt{N}}$  for convenience. First recall that by Lemma 18, we can ignore the initial NN output as follows:
$$
\left\| g _ {n} \left(\boldsymbol {w} _ {i} ^ {(0)}\right) - \frac {1}{n} \boldsymbol {X} ^ {\top} \left(f _ {*} (\boldsymbol {X}) \odot \sigma^ {\prime} \left(\boldsymbol {X} \boldsymbol {w} _ {i} ^ {(0)} + b _ {i} \mathbf {1} _ {n}\right)\right) \right\| \lesssim \frac {\sqrt {1 + \theta}}{d}, \tag {B.15}
$$
with probability  $1 - \exp \left(-c\log^2 d\right)$  for some constant  $c > 0$ . Note that since  $\beta < 1$ , the RHS can always be upper-bounded by  $d^{-\varepsilon}$  for some small  $\varepsilon > 0$ . Next, we take into account the concentration and truncation error in Propositions 19 and 16, respectively. Define the population gradient (again omitting the scalar  $\frac{a_i}{\sqrt{N}}$ ),
$$
g \left(\boldsymbol {w} _ {i} ^ {(0)}\right) = \mathbb {E} _ {\boldsymbol {x}} \left[ \boldsymbol {x} f _ {*} (\boldsymbol {x}) \sigma^ {\prime} \left(\langle \boldsymbol {x}, \boldsymbol {w} _ {i} ^ {(0)} \rangle + b _ {i}\right) \right].
$$
(B.10) in Proposition 19 implies that for any fixed  $p \in \mathbb{N}$ ,  $\beta > 1 - \frac{1}{k}$ , and large enough  $n, d$ , with probability  $1 - \exp(-c \log^2 d)$ , we have
$$
\left\| \frac {1}{n} \boldsymbol {X} ^ {\top} \left(f _ {*} (\boldsymbol {X}) \odot \sigma^ {\prime} \left(\boldsymbol {X} \boldsymbol {w} _ {i} ^ {(0)} + b _ {i} \mathbf {1} _ {n}\right)\right) - g \left(\boldsymbol {w} _ {i} ^ {(0)}\right) \right\| \lesssim d ^ {- \varepsilon} \left\| g \left(\boldsymbol {w} _ {i} ^ {(0)}\right) \right\|, \tag {B.16}
$$
for some  $\varepsilon > 0$ . As for the population counterpart, Corollary 17 implies that for neurons with indices  $i \in \mathcal{E}_{\tau} \cap \mathcal{B}_{\varepsilon}$ , if  $\beta > 1 - \frac{1}{k}$ , then we have
$$
\left\| g \left(\boldsymbol {w} _ {i} ^ {(0)}\right) - \alpha_ {k} ^ {b _ {i}} \alpha_ {k} ^ {*} \sqrt {1 + \theta} \left\langle \sqrt {1 + \theta} \boldsymbol {\mu}, \boldsymbol {w} _ {i} ^ {(0)} \right\rangle^ {k - 1} \boldsymbol {\mu} \right\| \lesssim d ^ {- \varepsilon} \left(1 + \left\| g \left(\boldsymbol {w} _ {i} ^ {(0)}\right) \right\|\right), \tag {B.17}
$$
where  $\left\| g(\pmb{w}_i^{(0)})\right\| \asymp (1 + \theta)^{\frac{k}{2}}\left|\left\langle \pmb{w}_i^{(0)},\pmb {\mu}\right\rangle^{k - 1}\right| = \tilde{\Theta}\Big((1 + \theta)^{\frac{k}{2}}d^{\frac{1 - k}{2}}\Big)$  due to the condition  $i\in \mathcal{E}_{\tau}$  for  $\tau = \log^C d$  . Combining (B.15), (B.16), and (B.17), we know that the gradient vector  $g_{n}(\pmb{w}_{i}^{(0)})$  achieves significant alignment with the target direction  $\pmb{\mu}$  when  $\beta >1 - \frac{1}{k}$
Normalization step. Now we consider the normalization procedure (B.2). When  $\beta > 1 - \frac{1}{k}$ , (B.15), (B.16), and (B.17) imply that for neurons  $\boldsymbol{w}_i$  in the subset  $i \in \mathcal{E}_{\tau} \cap \mathcal{B}_{\varepsilon}$ , we have
$$
\begin{array}{l} \boldsymbol {w} _ {i} ^ {(1)} = \boldsymbol {w} _ {i} ^ {(0)} - \eta a _ {i} \cdot \frac {1}{n} \boldsymbol {X} ^ {\top} \left(f _ {\mathrm {N N}} ^ {(0)} (\boldsymbol {X}) - f _ {*} (\boldsymbol {X})\right) \odot \sigma^ {\prime} \left(\boldsymbol {X} \boldsymbol {w} _ {i} ^ {(0)} + b _ {i} \mathbf {1} _ {n}\right) \\ \stackrel {(i)} {=} \boldsymbol {w} _ {i} ^ {(0)} + \eta a _ {i} \cdot \left(\alpha_ {k} ^ {b _ {i}} \alpha_ {k} ^ {*} \sqrt {1 + \theta} \left\langle \sqrt {1 + \theta} \boldsymbol {\mu}, \boldsymbol {w} _ {i} ^ {(0)} \right\rangle^ {k - 1} (\boldsymbol {\mu} + \boldsymbol {\Delta})\right), \\ \end{array}
$$
where  $\| \pmb{\Delta} \| \lesssim d^{-\varepsilon}$  for some  $\varepsilon > 0$ , with probability  $1 - \exp \left(-c \log^2 d\right)$ . Now recall that we select a sufficiently large learning rate  $\eta = \Omega(N^{1/2 + \varepsilon})$  for some small  $\varepsilon > 0$ . Since  $a_i \sim \mathcal{N}(0, 1/N)$ ,
$$
\left\| \boldsymbol {w} _ {i} ^ {(1)} - \kappa_ {i} \boldsymbol {\mu} \right\| \lesssim d ^ {- \varepsilon} | \kappa_ {i} |, \tag {B.18}
$$
where  $\kappa_{i} = \eta a_{i}\alpha_{k}^{b_{i}}\alpha_{k}^{*}\sqrt{1 + \theta}\left\langle \sqrt{1 + \theta}\boldsymbol {\mu},\boldsymbol{w}_{i}^{(0)}\right\rangle^{k - 1} = \tilde{\Theta}\Big(N^{\varepsilon}(1 + \theta)^{\frac{k}{2}}d^{\frac{1 - k}{2}}\Big)$  by the choice of  $\tau \asymp \log^C d$  and  $\alpha_{k}^{b_{i}}\asymp \frac{1}{\log^{C}d}$ . Define the normalization factor
$$
\chi_ {i} = \left(\frac {1}{n} \sum_ {j = 1} ^ {n} \langle \boldsymbol {x} _ {j}, \boldsymbol {w} _ {i} ^ {(1)} \rangle^ {2}\right) ^ {1 / 2} = \frac {1}{\sqrt {n}} \left\| \boldsymbol {X} \boldsymbol {w} _ {i} ^ {(1)} \right\|,
$$
by (B.18) we know that
$$
\begin{array}{l} \left| \chi_ {i} - \kappa_ {i} \| \boldsymbol {X} \boldsymbol {\mu} \| \right| = \frac {1}{\sqrt {n}} \left| \left\| \boldsymbol {X} \boldsymbol {w} _ {i} ^ {(1)} \right\| - \kappa_ {i} \| \boldsymbol {X} \boldsymbol {\mu} \| \right| \\ \leq \frac {1}{\sqrt {n}} \left\| \boldsymbol {X} \left(\boldsymbol {w} _ {i} ^ {(1)} - \kappa_ {i} \boldsymbol {\mu}\right) \right\| \lesssim d ^ {- \varepsilon} (1 + \theta) ^ {1 / 2} | \kappa_ {i} |, \\ \end{array}
$$
with high probability, where we used the proportional scaling of  $n$ ,  $d$  in the last inequality. Now note that  $\frac{1}{\sqrt{n}}\|X\pmb{\mu}\| \to \sqrt{1 + \theta}$  almost surely when  $n, d \to \infty$  proportionally, we have
$$
\left| \chi_ {i} ^ {- 1} - \left(\kappa_ {i} \sqrt {1 + \theta}\right) ^ {- 1} \right| = \frac {\left| \chi_ {i} - \kappa_ {i} (1 + \theta) ^ {1 / 2} \right|}{\left| \chi_ {i} \kappa_ {i} (1 + \theta) ^ {1 / 2} \right|} \lesssim d ^ {- \varepsilon} (1 + \theta) ^ {- 1 / 2} \left| \kappa_ {i} ^ {- 1} \right|.
$$
Combining the above calculations and recalling  $N = \Omega(d^{\epsilon})$ , we arrive at the following characterization of the normalized weight vector:
$$
\begin{array}{l} \left\| \left(\frac {1}{n} \sum_ {j = 1} ^ {n} \langle \boldsymbol {x} _ {j}, \boldsymbol {w} _ {i} ^ {(1)} \rangle^ {2}\right) ^ {- 1 / 2} \cdot \boldsymbol {w} _ {i} ^ {(1)} - \operatorname {s i g n} (\kappa_ {i}) (1 + \theta) ^ {- 1 / 2} \boldsymbol {\mu} \right\| \\ \leq \chi_ {i} ^ {- 1} \left\| \boldsymbol {w} _ {i} ^ {(1)} - \kappa_ {i} \boldsymbol {\mu} \right\| + \left| \operatorname {s i g n} \left(\kappa_ {i}\right) (1 + \theta) ^ {- 1 / 2} - \kappa_ {i} \chi_ {i} ^ {- 1} \right| \lesssim d ^ {- \varepsilon} (1 + \theta) ^ {- 1 / 2}, \tag {B.19} \\ \end{array}
$$
with probability  $1 - \exp \left(-c\log^2 d\right)$ . Observe that (B.19) entails that after the normalization step:
$$
\pmb {w} _ {i} ^ {(1)} \leftarrow \left(\frac {1}{n} \sum_ {j = 1} ^ {n} \langle \pmb {x} _ {j}, \pmb {w} _ {i} ^ {(1)} \rangle^ {2}\right) ^ {- 1 / 2} \cdot \pmb {w} _ {i} ^ {(1)},
$$
the updated weight vector with index  $i \in \mathcal{E}_{\tau} \cap \mathcal{B}_{\varepsilon}$  approximates the target direction in the sense that  $\left\| \sqrt{1 + \theta} \cdot \boldsymbol{w}_i^{(1)} - \mathrm{sign}(\kappa_i)\boldsymbol{\mu} \right\| \lesssim d^{-\varepsilon}$  with high probability, for some small  $\varepsilon > 0$ .
Approximation of link function. Now we construct certain second-layer coefficients  $\tilde{a}_i$  on a subset of neurons with indices  $i\in \mathcal{I}$  such that the (sub)network approximates the link function  $\sigma_{*}$ . Roughly speaking, we wish to find some collection of neurons that utilizes the univariate approximation result in Lemma 20:
$$
\tilde {f} (\boldsymbol {x}) := \sum_ {i \in \mathcal {I}} \frac {\tilde {a} _ {i}}{\sqrt {N}} \sigma (\langle \boldsymbol {x}, \boldsymbol {w} _ {i} ^ {(1)} \rangle + b _ {i}) \approx \mathbb {E} [ v (s, b) \cdot \sigma (s \langle \boldsymbol {x}, (1 + \theta) ^ {- 1 / 2} \boldsymbol {\mu} \rangle + b) ].
$$
First, we consider the ideal setting where  $\pmb{w}_i$  is exactly in the direction of  $\pmb{\mu}$  up to sign flip, that is, we may write  $\pmb{w}_i = s_i(1 + \theta)^{-1/2}\pmb{\mu}$  for  $s_i \sim \mathrm{Unif}(\{+1, -1\})$ . This allows us to reparameterize  $\langle \pmb{x}, (1 + \theta)^{-1/2}\pmb{\mu} \rangle \eqqcolon z \in \mathbb{R}$ , and choose the second-layer coefficients  $\tilde{a}$  according to Lemma 20 on domain  $|z| \leq \log \log d$ . This implies that  $|\upsilon(s, b)| = \mathcal{O}(\exp(\log \log d)) = \mathcal{O}(\log d)$ . The corresponding "infinite-width" neural network is written as
$$
f _ {\infty} (z) = \mathbb {E} _ {s \sim \operatorname {U n i f} (\{+ 1, - 1 \}), b \sim \mathcal {N} (0, 1)} [ v (s, b) \cdot \sigma (s z + b) ].
$$
From Lemma 20 we know that for  $|z|\leq \log \log d$ , we have  $f_{\infty}(z) = \sigma_*(z)$ .
Now we control the difference between the ideal model and the finite-width network of interest. First we restrict ourselves to  $b \in \mathcal{B}_{\varepsilon}^{\prime}$  defined as a "continuous" analogue of (B.14):  $\mathcal{B}_{\varepsilon}^{\prime} = \left\{b \mid |h_{p - 2}(b)| \geq \varepsilon, \text{ if } p \geq 2\right\}$ , with  $\varepsilon \asymp \frac{1}{\log^{C} d}$  for large  $C > 0$ , and write the corresponding infinite-width model as
$$
f _ {\infty} ^ {\varepsilon} (z) = \mathbb {E} _ {s, b} [ v (s, b) \sigma (s z + b) \cdot \mathbf {1} _ {b \in \mathcal {B} _ {\varepsilon} ^ {\prime}} ].
$$
By the Lipschitz property of  $\sigma$ , we have  $|f_{\infty}(z) - f_{\infty}^{\varepsilon}(z)| \lesssim \sup |v(s, b)| \cdot \frac{1}{\log^{C} d} = o_{d}(1)$ .
Next, we bound the error due to truncation at the interval  $|z| \leq \log \log d$ . Recall that  $\langle \pmb{x}, (1 + \theta)^{-1/2} \pmb{\mu} \rangle \eqqcolon z \sim \mathcal{N}(0, 1)$ ; hence we have the following evaluation
$$
\begin{array}{l} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \left[ \left(\sigma_ {*} (z) - f _ {\infty} ^ {\varepsilon} (z)\right) ^ {2} \right] \leq \mathbb {E} _ {z} \left[ \left(\sigma_ {*} (z) - f _ {\infty} (z)\right) ^ {2} \right] + \mathbb {E} _ {z} \left[ \left(f _ {\infty} (z) - f _ {\infty} ^ {\varepsilon} (z)\right) ^ {2} \right] \\ \lesssim \frac {1}{\log^ {C} d} + \mathbb {E} _ {z} \left[ \left(\sigma_ {*} (z) - f _ {\infty} (z)\right) ^ {2} \mathbf {1} _ {| z | \geq \log \log d} \right]. \tag {B.20} \\ \end{array}
$$
To upper-bound the above expectation, observe that
$$
\mathbb {E} _ {z} \left[ \sigma_ {*} (z) ^ {2} \mathbf {1} _ {| z | \geq \log \log d} \right] \lesssim \int_ {| z | > \log \log d} \sigma_ {*} (z) ^ {2} \exp (- z ^ {2} / 2) \mathrm {d} z \lesssim \frac {1}{\log^ {C} d}, \tag {B.21}
$$
for large  $C > 0$ , where we used the fact that  $\sigma_{*}$  is a degree-  $p$  polynomial with bounded  $L^2$  norm in the last inequality. On the other hand, by Cauchy-Schwarz we have
$$
| f _ {\infty} (z) | \leq \sqrt {\mathbb {E} [ v (s , b) ^ {2} ] \cdot \mathbb {E} [ \sigma (s z + b) ^ {2} ]} \lesssim \sup  | v (s, b) | \cdot \sqrt {\mathbb {E} [ \sigma (s z + b) ^ {2} ]}.
$$
Therefore, by the Lipschitz property of  $\sigma$  and the fact that  $\sup |v(s,b)|\asymp \log d$
$$
\mathbb {E} _ {z} \left[ f _ {\infty} (z) ^ {2} \mathbf {1} _ {| z | \geq \log \log d} \right] \lesssim \sqrt {\mathbb {E} \left[ f _ {\infty} (z) ^ {4} \right] \cdot \mathbb {E} \left[ \mathbf {1} _ {| z | \geq \log \log d} \right]} \lesssim \frac {1}{\log^ {C} d}. \tag {B.22}
$$
Combining (B.20), (B.21), and (B.22) yields  $\mathbb{E}_{z\sim \mathcal{N}(0,1)}\left[\left(\sigma_{*}(z) - f_{\infty}^{\varepsilon}(z)\right)^{2}\right] \lesssim \frac{1}{\log^{C}d} = o_{d}(1)$  for some large  $C > 0$ . Now consider the finite-width model
$$
\hat {f} (z) = \sum_ {i \in \mathcal {I}} \frac {\tilde {a} _ {i}}{\sqrt {N}} \sigma (s _ {i} z + b _ {i}) \mathbf {1} _ {i \in \mathcal {B} _ {\varepsilon}},
$$
where  $s_i \stackrel{\mathrm{i.i.d.}}{\sim} \operatorname{Unif}(\{+1, -1\})$  and  $b_i \stackrel{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0, 1)$ . Similar to above, we set the second-layer  $\tilde{a}$  according to Lemma 20, that is,  $\tilde{a}_i = \frac{\sqrt{N}}{|I|} v(s_i, b_i)$ .
To establish the concentration of  $\hat{f}$  around  $f_{\infty}^{\varepsilon}$ , we first restrict  $|z| \leq \log d$ . Let  $\phi(s, b) = \sigma(sz + b)$ . Then  $\|\nabla \phi(s, b)\|_{\infty} \leq 1 + \log d$ . Hence, applying Lipschitz concentration inequality for  $s$  and  $b$ , we have  $\|\phi(s, b)\|_{\psi_2} \leq 1 + \log d$ . Since uniformly  $\|v(s, b)\|_{\infty} \leq \log d$ , we know that the subexponential norm  $\|v(s, b)\phi(s, b)\|_{\psi_1} \leq 2\log^2 d$ . Thus, utilizing the Bernstein's inequality [Ver18, Theorem 2.8.1], we know that
$$
\begin{array}{l} \mathbb {P} \left(\left| \frac {1}{| \mathcal {T} |} \sum_ {i \in \mathcal {I}} v \left(s _ {i}, b _ {i}\right) \sigma \left(z s _ {i} + b _ {i}\right) \mathbf {1} _ {i \in \mathcal {B} _ {\varepsilon}} - f _ {\infty} ^ {\varepsilon} (z) \right| > t \mid | z | \leq \log d\right) \\ \leq 2 \exp \left(- \frac {c | \mathcal {I} |}{\log^ {4} d} \min  \{t ^ {2}, t \}\right), \\ \end{array}
$$
for any  $t > 0$ . By construction, we can construct a subset of neurons  $\mathcal{I} \subset \mathcal{E}_{\tau} \cap \mathcal{B}_{\varepsilon}$  with  $|\mathcal{I}| = \Theta(N)$ , with probability at least  $1 - \exp(-c\log^2 d)$ . Since we choose  $N = \Omega(d^\epsilon)$ , we may take  $t \asymp \frac{1}{\log^C d}$  for large  $C$ , and a union bound yields  $\left|\hat{f}(z) - f_{\infty}^\varepsilon(z)\right| \lesssim \frac{1}{\log^C d}$  with probability  $1 - \exp(-c\log^2 d)$  as  $d, N \to \infty$ , under the event that  $|z| \leq \log d$ . Then by the tail bound of the standard Gaussian random variable  $z$ , we have
$$
\begin{array}{l} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \left[ (\hat {f} (z) - f _ {\infty} ^ {\varepsilon} (z)) ^ {2} \right] \\ \lesssim \mathbb {E} _ {z} \left[ (\hat {f} (z) - f _ {\infty} ^ {\varepsilon} (z)) ^ {2} \mathbf {1} _ {| z | \leq \log d} \right] + \mathbb {E} _ {z} \left[ (\hat {f} (z) ^ {2} + f _ {\infty} ^ {\varepsilon} (z) ^ {2}) \mathbf {1} _ {| z | > \log d} \right] \lesssim \frac {1}{\log^ {C} d}, \tag {B.23} \\ \end{array}
$$
with probability  $1 - \exp \left(-c\log^2 d\right)$  when  $d,N\to \infty$
To control the deviation from the ideal setting, we consider neurons  $\pmb{w}_i^{(1)}$  where  $i\in \mathcal{E}_{\tau}\cap \mathcal{B}_{\varepsilon}$  for  $\tau \asymp \log^C d$  and  $\varepsilon \asymp \frac{1}{\log^Cd}$ . Recall that for this subset of neurons, with at least probability  $1 - \exp (-c\log^2 d)$  we have  $\left\| \sqrt{1 + \theta}\cdot \pmb{w}_i^{(1)} - \mathrm{sign}(\kappa_i)\pmb {\mu}\right\| \lesssim d^{-\varepsilon}$ , where  $\mathrm{sign}(\kappa_i)\sim \mathrm{Unif}(\{+1, - 1\})$  due to the Gaussian initialization of  $a_{i}$  (note that  $\pmb {w}_i,a_i,b_i$  are independent). Therefore,
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}} \left(\sum_ {i \in \mathcal {I}} \frac {1}{\sqrt {N}} \tilde {a} _ {i} \sigma \left(\left\langle \boldsymbol {x}, \boldsymbol {w} _ {i} ^ {(1)} \right\rangle + b _ {i}\right) - \sum_ {i \in \mathcal {I}} \frac {1}{\sqrt {N}} \tilde {a} _ {i} \sigma \left(\operatorname {s i g n} \left(\kappa_ {i}\right) \left\langle \boldsymbol {x}, (1 + \theta) ^ {- 1 / 2} \boldsymbol {\mu} \right\rangle + b _ {i}\right)\right) ^ {2} \\ \leq \frac {1}{N} \| \tilde {\boldsymbol {a}} \| ^ {2} \cdot \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {i \in \mathcal {I}} \left\langle \boldsymbol {x}, \boldsymbol {w} _ {i} ^ {(1)} - \operatorname {s i g n} \left(\kappa_ {i}\right) (1 + \theta) ^ {- 1 / 2} \boldsymbol {\mu} \right\rangle^ {2} \right] \lesssim \frac {\| \tilde {\boldsymbol {a}} \| ^ {2} | \mathcal {I} |}{N d ^ {\varepsilon}}, \tag {B.24} \\ \end{array}
$$