SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To focus on the sample complexity and avoid width-related bottlenecks, we directly simulate the infinite width limit  $(m_2\to \infty)$  of Algorithm 1 by computing the kernel  $\mathbb{K}$  in closed form. Finally, we run each trial with 5 random seeds and report the min, median, and max values in Figure 1.
Comparison Between Two and Three-Layer Networks We also show that the sample complexity separation between two and three layer networks persists in standard training settings. In Figure 2, we train both a two and three-layer neural network on the target  $f^{*}(x) = \mathrm{ReLU}(x^{T}Ax)$ , where  $A$  is symmetric with random entries, as described above. Both networks are initialized using the  $\mu \mathrm{P}$  parameterization [57] and are trained using SGD with momentum on all layers simultaneously. The input dimension is  $d = 32$ , and the widths are chosen to be 256 for the three-layer network and 2048 for the two-layer network, so that the parameter counts are approximately equal. Figure 2 plots the average test loss over 5 random seeds against sample size; here, we see that the three-layer network has a better sample complexity than the two-layer network.
Figure 2: Three-layer neural networks learn the target  $f^{*}(x) = \mathrm{ReLU}(x^{T}Ax)$  with better sample complexity than two-layer networks.
Experimental Details. Our experiments were written in JAX [14], and were run on a single NVIDIA RTX A6000 GPU.
B.1 Asymptotic Notation
Throughout the proof we will let  $C$  be a fixed but sufficiently large constant.
Definition 4 (high probability events). Let  $\iota = C\log(dnm_1m_2)$ . We say that an event happens with high probability if it happens with probability at least  $1 - \mathrm{poly}(d,n,m_1,m_2)e^{-\iota}$ .
Example 5. If  $z \sim N(0,1)$  then  $|z| \leq \sqrt{2\iota}$  with high probability.
Note that high probability events are closed under union bounds over sets of size  $\mathrm{poly}(d,n,m_1,m_2)$ . We will also assume throughout that  $\iota \leq C^{-1}d$ .
B.2 Tensor Notation
For a  $k$ -tensor  $T \in (\mathbb{R}^d)^{\otimes k}$ , let Sym be the symmetrization of  $T$  across all  $k$  axes, i.e
$$
\operatorname {S y m} (T) _ {i _ {1}, \dots , i _ {k}} = \frac {1}{k !} \sum_ {\sigma \in S _ {k}} T _ {i _ {\sigma (1)}, \dots , i _ {\sigma (k)}}
$$
Next, given tensors  $A \in (\mathbb{R}^d)^{\otimes a}$ ,  $B \in (\mathbb{R}^d)^{\otimes b}$ , we let  $A \otimes B \in (\mathbb{R}^d)^{\otimes a + b}$  be their tensor product.
Definition 5 (Symmetric tensor product). Given tensors  $A \in (\mathbb{R}^d)^{\otimes a}$ ,  $B \in (\mathbb{R}^d)^{\otimes b}$ , we define their symmetric tensor product  $A\tilde{\otimes} B \in (\mathbb{R}^d)^{\otimes a + b}$  as
$$
A \tilde {\otimes} B := \operatorname {S y m} (A \otimes B).
$$
We note that  $\tilde{\otimes}$  satisfies associativity.
Definition 6 (Tensor contraction). Given a symmetric  $k$ -tensor  $T \in (\mathbb{R}^d)^{\otimes k}$  and an  $a$ -tensor  $A \in (\mathbb{R}^d)^{\otimes a}$ , where  $k \geq a$ , we define the tensor contraction  $T(A)$  to be the  $k - a$  tensor given by
$$
T (A) _ {i _ {1}, \dots , i _ {k - a}} = \sum_ {(j _ {1}, \dots , j _ {a}) \in [ d ] ^ {a}} T _ {j _ {1}, \dots , j _ {a}, i _ {1}, \dots , i _ {k - a}} A _ {j _ {1}, \dots , j _ {a}}.
$$
When  $A$  is also a  $k$ -tensor, then  $\langle T, A \rangle \coloneqq T(A) \in \mathbb{R}$  denotes their Euclidean inner product. We further define  $\|T\|_F^2 \coloneqq \langle T, T \rangle$ .
C Univariate Approximation
Throughout this section, let  $\mu(b) := \frac{\exp(-b^2/2)}{\sqrt{2\pi}}$  denote the PDF of a standard Gaussian.
Lemma 1. Let  $a \sim \mathrm{Unif}(\{-1,1\})$  and let  $b \sim N(0,1)$ . Then there exists  $v(a,b)$  supported on  $\{-1,1\} \times [0,2]$  such that for any  $|x| \leq 1$ ,
$$
\mathbb {E} _ {a, b} [ v (a, b) \sigma (a x + b) ] = 1 \quad a n d \quad \sup  _ {a, b} | v (a, b) | \lesssim 1.
$$
Proof. Let  $v(a,b) = c\mathbf{1}_{b \in [1,2]}$  where  $c = \frac{1}{\mu(1) - \mu(2)}$ . Then for  $|x| \leq 1$ ,
$$
\begin{array}{l} \mathbb {E} _ {a, b} [ v (a, b) \sigma (a x + b) ] = c \int_ {1} ^ {2} \frac {1}{2} [ \sigma (x + b) + \sigma (- x + b) ] \mu (b) d b \\ = c \int_ {1} ^ {2} b \mu (b) \\ = 1. \\ \end{array}
$$
Lemma 2. Let  $a \sim \mathrm{Unif}(\{-1,1\})$  and let  $b \sim N(0,1)$ . Then there exists  $v(a,b)$  supported on  $\{-1,1\} \times [0,2]$  such that for any  $|x| \leq 1$ ,
$$
\mathbb {E} _ {a, b} [ v (a, b) \sigma (a x + b) ] = x \quad a n d \quad \sup  _ {a, b} | v (a, b) | \lesssim 1.
$$
Proof. Let  $v(a, b) = ca\mathbf{1}_{b \in [1,2]}$  where  $c = \frac{1}{\int_0^1 \mu(b)db}$ . Then for  $|x| \leq 1$ ,
$$
\begin{array}{l} \mathbb {E} _ {a, b} [ v (a, b) \sigma (a x + b) ] = c \int_ {1} ^ {2} \frac {1}{2} [ \sigma (x + b) - \sigma (- x + b) ] \mu (b) d b \\ = c x \int_ {1} ^ {2} \mu (b) d b \\ = x. \\ \end{array}
$$
Lemma 3. Let  $f:[-1,1]\to \mathbb{R}$  be any twice differentiable function. Then there exists  $v(a,b)$  supported on  $\{-1,1\} \times [0,2]$  such that for any  $|x|\leq 1$ ,
$$
\mathbb{E}_{a,b}[v(a,b)\sigma (ax + b)] = f(x) \quad and \quad \sup_{a,b}|v(a,b)|\lesssim \sup_{\substack{x\in [-1,1]\\ k\in \{0,1,2\}}} |f^{(k)}(x)|.
$$
Proof. First consider  $v(a, b) = \frac{\mathbf{1}_{b \in [0,1]}}{\mu(b)} 2f''(-ab)$ . Then when  $x \geq 0$  we have by integration by parts:
$$
\begin{array}{l} \mathbb {E} _ {a b} [ v (a, b) \sigma (a x + b) ] \\ = \int_ {0} ^ {1} [ f ^ {\prime \prime} (- b) \sigma (x + b) + f ^ {\prime \prime} (b) \sigma (- x + b) ] d b \\ = f ^ {\prime} (0) x - f ^ {\prime} (- 1) (x + 1) + f ^ {\prime} (1) (- x + 1) + \int_ {0} ^ {1} f ^ {\prime} (- b) d b - \int_ {x} ^ {1} f ^ {\prime} (b) d b \\ = f (x) + c _ {1} + c _ {2} x \\ \end{array}
$$
where  $c_{1} = f(0) - f(1) - f(-1) + f'(1) - f'(-1)$  and  $c_{2} = f'(0) - f'(1) - f'(-1)$ . In addition when  $x < 0$ ,
$$
\begin{array}{l} \mathbb {E} _ {a b} [ v (a, b) \sigma (a x + b) ] \\ = \int_ {0} ^ {1} [ f ^ {\prime \prime} (- b) \sigma (x + b) + f ^ {\prime \prime} (b) \sigma (- x + b) ] d b \\ = f ^ {\prime} (0) x - f ^ {\prime} (- 1) (x + 1) + f ^ {\prime} (1) (- x + 1) + \int_ {- x} ^ {1} f ^ {\prime} (- b) d b - \int_ {0} ^ {1} f ^ {\prime} (b) d b \\ = f (x) + c _ {1} + c _ {2} x \\ \end{array}
$$
so this equality is true for all  $x$ . By Lemmas 1 and 2 we can subtract out the constant and linear terms so there exists  $v(a, b)$  such that
$$
\mathbb {E} _ {a, b} [ v (a, b) \sigma (a x + b) ] = f (x).
$$
In addition, using that  $\mu (b)\gtrsim 1$  for  $b\in [0,1]$  gives that this  $v(a,b)$  satisfies
$$
\sup_{a,b}|v(a,b)|\lesssim |c_{1}| + |c_{2}| + \max_{x\in [-1,1]}|f^{\prime \prime}(x)|\lesssim \sup_{\substack{x\in [-1,1]\\ k\in \{0,1,2\}}} |f^{(k)}(x)|.
$$
D Proofs for Section 3
The following is a formal restatement of Theorem 1.
Theorem 6. Select  $q \in \mathcal{W}^{2,\infty}([-1,1])$ . Let  $\eta_1 = \frac{m_1}{m_2}\overline{\eta}$ , and  $n \gtrsim \| \mathbb{K}f^*\|_{L^2}^{-2}\iota^{2\ell +2\alpha_\sigma +1}$ ,  $m_1 \gtrsim \| \mathbb{K}f^*\|_{L^2}^{-2}\iota$ ,  $m_2 \gtrsim \| \mathbb{K}f^*\|_{L^2}^{-2}\iota^{2\alpha_\sigma +1}$ . There exists a choice of  $\overline{\eta} = \Theta (\iota^{-\chi}\| Kf^*\|_{L^2}^{-1})$ ,  $T = \text{poly}(d,n,m_1,m_2,\| q\|_{2,\infty})$ , and  $\lambda, \eta_2$  such that with high probability the output  $\hat{\theta}$  of Algorithm 1 satisfies the population  $L^2$  loss bound
$$
\begin{array}{l} \mathbb {E} _ {x} \left[ \left(f (x; \hat {\theta}) - f ^ {*} (x)\right) ^ {2} \right] \lesssim \| q \| _ {1, \infty} ^ {2} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 2} \cdot \left(\frac {\iota^ {2 \ell + 2 \alpha_ {\sigma} + 2 \chi + 1}}{n} + \frac {\iota^ {2 \alpha_ {\sigma} + 2 \chi + 1}}{m _ {2}} + \frac {\iota}{m _ {1}}\right) \\ + \| q \circ (\bar {\eta} \cdot \mathbb {K} f ^ {*}) - f ^ {*} \| _ {L ^ {2}} ^ {2} + \frac {\| q \| _ {2 , \infty} ^ {2} \iota}{m _ {1}} + \sqrt {\frac {\| q \| _ {2 , \infty} ^ {4} \iota + \iota^ {4 \ell + 1}}{n}} \\ \end{array}
$$
Proof of Theorem 6. The gradient with respect to  $w_{j}$  is
$$
\begin{array}{l} \nabla_ {w _ {j}} L _ {1} \left(\theta^ {(0)}\right) = \frac {1}{n} \sum_ {i} ^ {n} \left(f \left(x _ {i}; \theta^ {(0)}\right) - f ^ {*} \left(x _ {i}\right)\right) \nabla_ {w _ {j}} f \left(x _ {i}; \theta^ {(0)}\right) \\ = \sigma_ {1} ^ {\prime} \left(b _ {j} ^ {(0)}\right) \cdot \frac {1}{n} \sum_ {i = 1} ^ {n} \left(f \left(x _ {i}; \theta^ {(0)}\right) - f ^ {*} \left(x _ {i}\right)\right) \frac {a _ {j} ^ {(0)}}{m _ {1}} h ^ {(0)} \left(x _ {i}\right) \\ = \mathbf {1} _ {b _ {j} ^ {(0)} > 0} \cdot \frac {1}{n} \sum_ {i = 1} ^ {n} \left(f (x _ {i}; \theta^ {(0)}) - f ^ {*} (x _ {i})\right) \frac {a _ {j} ^ {(0)}}{m _ {1}} h ^ {(0)} (x _ {i}) \\ \end{array}
$$
$$
\begin{array}{l} w _ {j} ^ {(1)} = - \eta_ {1} \nabla_ {w _ {j}} L _ {1} \left(\theta^ {(0)}\right) \\ = \mathbf {1} _ {b _ {j} ^ {(0)} > 0} \cdot \frac {\overline {{\eta}}}{m _ {2}} a _ {j} ^ {(0)} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(f ^ {*} (x _ {i}) - f (x _ {i}; \theta^ {(0)})\right) h ^ {(0)} (x _ {i}). \\ \end{array}
$$
One then has
$$
\begin{array}{l} f (x; (a, W ^ {(1)}, b ^ {(1)}, V ^ {(0)})) \\ = \sum_ {j = 1} ^ {m _ {1}} \frac {a _ {j}}{m _ {1}} \sigma_ {1} \left(\mathbf {1} _ {b _ {j} ^ {(0)} > 0} \cdot a _ {j} ^ {(0)} \cdot \frac {\overline {{\eta}}}{m _ {2}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(f ^ {*} (x _ {i}) - f (x _ {i}; \theta^ {(0)})\right) \langle h ^ {(0)} (x _ {i}), h ^ {(0)} (x) \rangle + b _ {j}\right) \\ = \sum_ {j = 1} ^ {m _ {1}} \frac {a _ {j}}{m _ {1}} \sigma_ {1} \left(a _ {j} ^ {(0)} \cdot \bar {\eta} \phi (x) + b _ {j}\right) \cdot \mathbf {1} _ {b _ {j} ^ {(0)} > 0}, \\ \end{array}
$$
where  $\phi (x)\coloneqq \frac{1}{m_2n}\sum_{i = 1}^{n}\bigl (f^{\ast}(x_i) - (x_i;\theta^{(0)})\bigr)\langle h^{(0)}(x_i),h^{(0)}(x)\rangle .$
The second stage of Algorithm 1 is equivalent to random feature regression. The next lemma shows that there exists  $a^*$  with small norm that achieves low empirical loss on the dataset  $\mathcal{D}_2$ .
Lemma 4. There exists  $a^*$  with  $\| a^* \|_2 \lesssim \| q \|_{2,\infty} \sqrt{m_1}$  such that  $\theta^* = (a^*, W^{(1)}, b^{(0)}, V^{(0)})$  satisfies
$$
\begin{array}{l} L _ {2} \left(\theta^ {*}\right) \lesssim \| q \| _ {1, \infty} ^ {2} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 2} \cdot \left(\frac {\iota^ {2 \ell + 2 \alpha_ {\sigma} + 2 \chi + 1}}{n} + \frac {\iota^ {2 \alpha_ {\sigma} + 2 \chi + 1}}{m _ {2}} + \frac {\iota}{m _ {1}}\right) \\ + \frac {\iota^ {2 \ell + 1}}{n} + \frac {\| q \| _ {2 , \infty} ^ {2} \iota}{m _ {1}} + \| q \circ (\bar {\eta} \cdot \mathbb {K} f ^ {*}) - f ^ {*} \| _ {L ^ {2}} ^ {2} \\ \end{array}
$$
The proof of Lemma 4 is deferred to Appendix D.1. We first show that  $\phi$  is approximately proportional to  $\mathbb{K}f^{*}$ , and then invoke the ReLU random feature expressivity results from Appendix C.
Next, set
$$
\begin{array}{l} \lambda = \| a ^ {*} \| _ {2} ^ {- 2} \left(\| q \| _ {1, \infty} ^ {2} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 2} \cdot \left(\frac {\iota^ {2 \ell + 2 \alpha_ {\sigma} + 2 \chi + 1}}{n} + \frac {\iota^ {2 \alpha_ {\sigma} + 2 \chi + 1}}{m _ {2}} + \frac {\iota}{m _ {1}}\right) \right. \\ \left. + \frac {\iota^ {2 \ell + 1}}{n} + \frac {\| q \| _ {2 , \infty} ^ {2} \iota}{m _ {1}} + \| q \circ (\bar {\eta} (\mathbb {K} f ^ {*})) - f ^ {*} \| _ {L ^ {2}} ^ {2}\right), \\ \end{array}
$$
so that  $L_{2}(\theta^{*})\lesssim \| a^{*}\|_{2}^{2}\lambda$
Define the regularized loss to be  $\hat{L}(a) \coloneqq L_2\big((a, W^{(1)}, b^{(0)}, V^{(0)})\big) + \frac{\lambda}{2} \|a\|_2^2$ . Let  $a^{(\infty)} = \arg \min_a \hat{L}(a)$ , and  $a^{(t)}$  be the predictor from running gradient descent for  $t$  steps initialized at  $a^{(0)}$ . We first note that
$$
\hat {L} \left(a ^ {(\infty)}\right) \leq \hat {L} \left(a ^ {*}\right) \lesssim \left\| a ^ {*} \right\| _ {2} ^ {2} \lambda .
$$
Next, we remark that  $\hat{L}(a)$  is  $\lambda$ -strongly convex. Additionally, we can write  $f(x; (a, W^{(1)}, b^{(0)}, V^{(0)})) = a^T \psi(x)$  where  $\psi(x) = \operatorname{Vec}\left(\frac{1}{m_1} \sigma_1(a_j^{(0)}) \overline{\eta} \phi(x) + b_j^{(0)}\right) \cdot \mathbf{1}_{b_j^{(0)} > 0}$ . In Appendix D.2 we show  $\sup_{x \in \mathcal{D}_2} \| \psi(x) \| \lesssim \frac{1}{m_1}$ . Therefore
$$
\lambda_ {m a x} \Big (\nabla_ {a} ^ {2} \hat {L} \Big) \leq \frac {2}{n} \sum_ {x \in \mathcal {D} _ {2}} \| \psi (x) \| ^ {2} \lesssim \frac {1}{m _ {1}},
$$
so  $\hat{L}$  is  $O\left(\frac{1}{m_1}\right) + \lambda$  smooth. Choosing a learning rate  $\eta_{2} = \Theta (m_{1})$ , after  $T = \tilde{O}\left(\frac{1}{\lambda m_1}\right) = \mathrm{poly}(d,n,m_1,m_2,\| q\|_{2,\infty})$  steps we reach an iterate  $\hat{a} = a^{(T)}$ ,  $\hat{\theta} = (\hat{a},W^{(1)},b^{(0)},V^{(0)})$  satisfying
$$
L _ {2} (\hat {\theta}) \lesssim L _ {2} \left(\theta^ {*}\right) \quad \text {a n d} \quad \| \hat {a} \| _ {2} \lesssim \| a ^ {*} \| _ {2}.
$$
For  $\tau > 0$ , define the truncated loss  $\ell_{\tau}$  by  $\ell_{\tau}(z) = \min(z^2, \tau^2)$ . We have that  $\ell_{\tau}(z) \leq z^2$ , and thus
$$
\frac {1}{n} \sum_ {x \in \mathcal {D} _ {2}} \ell_ {\tau} \Big (f (x; \hat {\theta}) - f ^ {*} (x) \Big) \leq L _ {2} (\hat {\theta}).
$$
Consider the function class
$$
\mathcal {F} (B _ {a}) := \left\{f (\cdot ; (a, W ^ {(1)}, b ^ {(0)}, V ^ {(0)}): \| a \| _ {2} \leq B _ {a}) \right\}
$$
The following lemma bounds the empirical Rademacher complexity of this function class
Lemma 5. Given a dataset  $\mathcal{D}$ , recall that the empirical Rademacher complexity of  $\mathcal{F}$  is defined as
$$
\mathcal {R} _ {\mathcal {D}} (\mathcal {F}) := \mathbb {E} _ {\sigma \in \{\pm 1 \} ^ {n}} \left[ \sup  _ {f \in \mathcal {F}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} f (x _ {i}) \right].
$$
Then with high probability
$$
\mathcal {R} _ {\mathcal {D} _ {2}} (\mathcal {F} (B _ {a})) \lesssim \sqrt {\frac {B _ {a} ^ {2}}{n m _ {1}}}.
$$
Since  $\ell_{\tau}$  loss is  $2\tau$ -Lipschitz, the above lemma with  $B_{a} = O(\| a^{*}\|_{2}) = O\Big(\| q\|_{2,\infty}\sqrt{m_{1}}\Big)$  along with the standard empirical Rademacher complexity bound yields
$$
\begin{array}{l} \mathbb {E} _ {x} \ell_ {\tau} \Big (f (x; \hat {\theta}) - f ^ {*} (x) \Big) \lesssim \frac {1}{n} \sum_ {x \in \mathcal {D} _ {2}} \ell_ {\tau} \Big (f (x; \hat {\theta}) - f ^ {*} (x) \Big) + \tau \cdot \mathcal {R} _ {\mathcal {D} _ {2}} (\mathcal {F}) + \tau^ {2} \sqrt {\frac {\iota}{n}} \\ \lesssim \| q \| _ {1, \infty} ^ {2} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 2} \cdot \left(\frac {\iota^ {2 \ell + 2 \alpha_ {\sigma} + 2 \chi + 1}}{n} + \frac {\iota^ {2 \alpha_ {\sigma} + 2 \chi + 1}}{m _ {2}} + \frac {\iota}{m _ {1}}\right) \\ + \| q \circ (\bar {\eta} \cdot \mathbb {K} f ^ {*}) - f ^ {*} \| _ {L ^ {2}} ^ {2} + \frac {\iota^ {2 \ell + 1}}{n} + \frac {\| q \| _ {2 , \infty} ^ {2} \iota}{m _ {1}} \\ + \tau \sqrt {\frac {\left\| q \right\| _ {2 , \infty} ^ {2}}{n}} + \tau^ {2} \sqrt {\frac {\iota}{n}}. \\ \end{array}
$$
Finally, we relate the  $\ell_{\tau}$  population loss to the  $\ell^2$  population loss.
Lemma 6. Let  $\tau = \Theta (\max (t^{\ell},\| q\|_{2,\infty}))$ . Then with high probability over  $\hat{\theta}$
$$
\mathbb {E} _ {x} \left[ \left(f (x; \hat {\theta}) - f ^ {*} (x)\right) ^ {2} \right] \leq \mathbb {E} _ {x} \ell_ {\tau} \Big (f (x; \hat {\theta}) - f ^ {*} (x) \Big) + \frac {\| q \| _ {2 , \infty} ^ {2}}{m _ {1}}.
$$
Plugging in  $\tau$  and applying Lemma 6 yields
$$
\begin{array}{l} \mathbb {E} _ {x} \left[ \left(f (x; \hat {\theta}) - f ^ {*} (x)\right) ^ {2} \right] \lesssim \| q \| _ {1, \infty} ^ {2} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 2} \cdot \left(\frac {\iota^ {2 \ell + 2 \alpha_ {\sigma} + 2 \chi + 1}}{n} + \frac {\iota^ {2 \alpha_ {\sigma} + 2 \chi + 1}}{m _ {2}} + \frac {\iota}{m _ {1}}\right) \\ + \| q \circ (\overline {{\eta}} \cdot \mathbb {K} f ^ {*}) - f ^ {*} \| _ {L ^ {2}} ^ {2} + \frac {\| q \| _ {2 , \infty} ^ {2} \iota}{m _ {1}} + \sqrt {\frac {\| q \| _ {2 , \infty} ^ {4} \iota + \iota^ {4 \ell + 1}}{n}} \\ \end{array}
$$
D.1 Proof of Lemma 4
We require three auxiliary lemmas, all of whose proofs are deferred to Appendix D.4. The first lemma bounds the error between the population learned feature  $\mathbb{K}f^{*}$  and the finite sample learned feature  $\phi$  over the dataset  $\mathcal{D}_2$ .
Lemma 7. With high probability
$$
\sup _ {x \in \mathcal {D} _ {2}} | \phi (x) - (\mathbb {K} f ^ {*}) (x) | \lesssim \sqrt {\frac {\iota^ {2 \ell + 2 \alpha_ {\sigma} + 1}}{n}} + \sqrt {\frac {\iota^ {2 \alpha_ {\sigma} + 1}}{m _ {2}}} + \sqrt {\frac {\iota}{m _ {1}}}.
$$
The second lemma shows that with appropriate choice of  $\eta$ , the quantity  $\eta \phi(x)$  is small.
Lemma 8. Let  $n \gtrsim \| \mathbb{K}f^{*}\|_{L^2}^{-2}\iota^{2\ell +2\alpha_\sigma +1},m_1 \gtrsim \| \mathbb{K}f^{*}\|_{L^2}^{-2}\iota,$  and  $m_2 \gtrsim \| \mathbb{K}f^{*}\|_{L^2}^{-2}\iota^{2\alpha_\sigma +1}$ . There exists  $\overline{\eta} = \tilde{\Theta} (\| \mathbb{K}f^{*}\|_{L^2}^{-2})$  such that with high probability,  $\sup_{x\in \mathcal{D}_2}|\overline{\eta}\phi (x)|\leq 1$  and  $\sup_{x\in \mathcal{D}_2}|\overline{\eta}\cdot \mathbb{K}f^* (x)|\leq 1$
The third lemma expresses the compositional function  $q \circ \overline{\eta} \phi$  via an infinite width network.
Lemma 9. Assume that  $n, m_1, m_2 = \tilde{\Omega}(\| \mathbb{K}f^*\|_{L^2}^{-2})$ . There exists  $v: \{\pm 1\} \times \mathbb{R} \to \mathbb{R}$  such that  $\sup_{a,b}|v(a,b)| \leq \| q\|_{2,\infty}$  and, with high probability over  $\mathcal{D}_2$ , the infinite width network
$$
f _ {v} ^ {\infty} (x) := \mathbb {E} _ {a, b} [ v (a, b) \sigma_ {1} (a \bar {\eta} \phi (x) + b) \mathbf {1} _ {b > 0} ]
$$
$$
f _ {v} ^ {\infty} (x) = q (\bar {\eta} \phi (x))
$$
for all  $x\in \mathcal{D}_2$
Proof of Lemma 4. Let  $v$  be the infinite width construction defined in Lemma 9. Define  $a^{*} \in \mathbb{R}^{m_{1}}$  to be the vector with  $a_{j}^{*} = v(a_{j}^{(0)}, b_{j}^{(1)})$ . We can decompose
$$
\begin{array}{l} L _ {2} \left(\theta^ {*}\right) = \frac {1}{n} \sum_ {x \in \mathcal {D} _ {2}} \left(f (x; \theta^ {*}) - f ^ {*} (x)\right) ^ {2} \\ \lesssim \frac {1}{n} \sum_ {x \in \mathcal {D} _ {2}} \left(f (x; \theta^ {*}) - f _ {v} ^ {\infty} (x)\right) ^ {2} + \frac {1}{n} \sum_ {x \in \mathcal {D} _ {2}} \left(f _ {v} ^ {\infty} (x) - q (\overline {{\eta}} \phi (x))\right) ^ {2} \\ + \frac {1}{n} \sum_ {x \in \mathcal {D} _ {2}} (q (\overline {{\eta}} \phi (x)) - q (\overline {{\eta}} (\mathbb {K} f ^ {*}) (x))) ^ {2} + \frac {1}{n} \sum_ {x \in \mathcal {D} _ {2}} (q (\overline {{\eta}} (\mathbb {K} f ^ {*}) (x)) - f ^ {*}) ^ {2} \\ \end{array}
$$
Take  $m_1 \gtrsim \iota$ . The first term is the error between the infinite width network  $f_v^\infty(x)$  and the finite width network  $f(x; \theta^*)$ . This error can be controlled via standard concentration arguments: by Corollary 2 we have that with high probability  $\|a^*\| \lesssim \frac{\|q\|_{2,\infty}}{\sqrt{m_1}}$ , and by Lemma 17 we have
$$
\frac {1}{n} \sum_ {x \in \mathcal {D} _ {2}} (f (x; \theta^ {*}) - f _ {v} ^ {\infty} (x)) ^ {2} \lesssim \frac {\| q \| _ {2 , \infty} ^ {2} \iota}{m _ {1}}.
$$
Next, by Lemma 9 we get that the second term is zero with high probability. We next turn to the third term. Since  $q$  is  $\| q \|_{1,\infty}$ -Lipschitz on  $[-1,1]$ , and  $\sup_{x \in \mathcal{D}_2} |\overline{\eta}\phi(x)| \leq 1$ ,  $\sup_{x \in \mathcal{D}_2} |(\overline{\eta} \cdot \mathbb{K}f^*)(x)| \leq 1$  by Lemma 8, we can apply Lemma 7 to get
$$
\begin{array}{l} \sup  _ {x \in \mathcal {D} _ {2}} | q (\bar {\eta} \phi (x)) - q (\bar {\eta} (\mathbb {K} f ^ {*}) (x)) | \leq \| q \| _ {1, \infty} \sup  _ {x \in \mathcal {D} _ {2}} | \bar {\eta} \phi (x) - \bar {\eta} (\mathbb {K} f ^ {*}) (x) | \\ \lesssim \| q \| _ {1, \infty} \overline {{\eta}} \left(\sqrt {\frac {\iota^ {2 \ell + 2 \alpha_ {\sigma} + 1}}{n}} + \sqrt {\frac {\iota^ {2 \alpha_ {\sigma} + 1}}{m _ {2}}} + \sqrt {\frac {\iota}{m _ {1}}}\right) \\ \lesssim \| q \| _ {1, \infty} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1} \left(\sqrt {\frac {\iota^ {2 \ell + 2 \alpha_ {\sigma} + 1}}{n}} + \sqrt {\frac {\iota^ {2 \alpha_ {\sigma} + 1}}{m _ {2}}} + \sqrt {\frac {\iota}{m _ {1}}}\right) \\ \end{array}
$$
$$
\frac {1}{n} \sum_ {x \in \mathcal {D} _ {2}} \left(q (\overline {{\eta}} \phi (x)) - q (\overline {{\eta}} (\mathbb {K} f ^ {*}) (x))\right) ^ {2} \lesssim \| q \| _ {1, \infty} ^ {2} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 2} \cdot \left(\frac {\iota^ {2 \ell + 2 \alpha_ {\sigma} + 2 \chi + 1}}{n} + \frac {\iota^ {2 \alpha_ {\sigma} + 2 \chi + 1}}{m _ {2}} + \frac {\iota}{m _ {1}}\right).
$$
Finally, we must relate the empirical error between  $q \circ \overline{\eta} \cdot \mathbb{K}f^{*}$  and  $f^{*}$  to the population error. This can be done via standard concentration arguments: in Lemma 19, we show
$$
\frac {1}{n} \sum_ {x \in \mathcal {D} _ {2}} \left(q (\overline {{\eta}} (\mathbb {K} f ^ {*}) (x)) - f ^ {*} (x)\right) ^ {2} \lesssim \| q \circ (\overline {{\eta}} (\mathbb {K} f ^ {*})) - f ^ {*} \| _ {L ^ {2}} ^ {2} + \frac {\| q \| _ {\infty} ^ {2} \iota + \iota^ {2 \ell + 1}}{n}.
$$
$$
\begin{array}{l} L _ {2} (\theta^ {*}) \lesssim \| q \| _ {1, \infty} ^ {2} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 2} \cdot \left(\frac {\iota^ {2 \ell + 2 \alpha_ {\sigma} + 2 \chi + 1}}{n} + \frac {\iota^ {2 \alpha_ {\sigma} + 2 \chi + 1}}{m _ {2}} + \frac {\iota}{m _ {1}}\right) + \frac {\| q \| _ {\infty} ^ {2} \iota + \iota^ {2 \ell + 1}}{n} + \frac {\| q \| _ {2 , \infty} ^ {2} \iota}{m _ {1}} \\ + \left\| q \circ (\bar {\eta} (\mathbb {K} f ^ {*})) - f ^ {*} \right\| _ {L ^ {2}} ^ {2} \\ \lesssim \| q \| _ {1, \infty} ^ {2} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 2} \cdot \left(\frac {\iota^ {2 \ell + 2 \alpha_ {\sigma} + 2 \chi + 1}}{n} + \frac {\iota^ {2 \alpha_ {\sigma} + 2 \chi + 1}}{m _ {2}} + \frac {\iota}{m _ {1}}\right) + \frac {\iota^ {2 \ell + 1}}{n} + \frac {\| q \| _ {2 , \infty} ^ {2} \iota}{m _ {1}} \\ + \left\| q \circ (\bar {\eta} (\mathbb {K} f ^ {*})) - f ^ {*} \right\| _ {L ^ {2}} ^ {2} \\ \end{array}
$$
since  $\| \mathbb{K}\|_{op}\lesssim 1$  and thus  $\| \mathbb{K}f^{*}\|_{L^2}\gtrsim 1$
Proof. The standard linear Rademacher bound states that for functions of the form  $x \mapsto w^T \psi(x)$  with  $\| w \|_2 \lesssim B_w$ , the empirical Rademacher complexity is upper bounded by
$$
\frac {B _ {w}}{n} \sqrt {\sum_ {x \in \mathcal {D}} \| \psi (x) \| ^ {2}}.
$$
We have that  $f(x;\theta) = a^T\psi (x)$ , where
$$
\psi (x) = \operatorname {V e c} \left(\frac {1}{m _ {1}} \sigma_ {1} \left(a _ {j} ^ {(0)} \bar {\eta} \phi (x) + b _ {j} ^ {(0)}\right) \cdot \mathbf {1} _ {b _ {j} ^ {(0)} > 0}\right).
$$
By Lemma 8, with high probability,  $\sup_{x\in \mathcal{D}_2}|\overline{\eta}\phi (x)|\leq 1$  . Thus for  $x\in \mathcal{D}_2$
$$
\left\| \psi (x) \right\| ^ {2} \leq \frac {1}{m _ {1} ^ {2}} \left(1 + | b _ {j} |\right) ^ {2} \lesssim \frac {1}{m _ {1}}
$$
with high probability by Lemma 10. Altogether, we have
$$
\mathcal {R} _ {\mathcal {D} _ {2}} (\mathcal {F} (B _ {a})) \lesssim \sqrt {\frac {B _ {a} \cdot n \cdot 1 / m _ {1}}{n}} = \sqrt {\frac {B _ {a} ^ {2}}{n m _ {1}}}.
$$