SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Additionally, with probability at least  $1 - \frac{1}{d^{\Omega(\log d)}}$  over the initialization and dataset, we have  $B_{t} \lesssim (\hat{\sigma}_{2,d}^{2} + \hat{\sigma}_{4,d}^{2}) \cdot \left( \frac{d(\log d)^{O(1)}}{\sqrt{m}} + \overline{\Delta}_{t} \right) \| \delta_{t} \|_{2}, \mathbb{E}[B_{t}] \lesssim \frac{(\hat{\sigma}_{2,d}^{2} + \hat{\sigma}_{4,d}^{2}) d(\log d)^{O(1)}}{\sqrt{m}} \overline{\Delta}_{t} + (\hat{\sigma}_{2,d}^{2} + \hat{\sigma}_{4,d}^{2}) \overline{\Delta}_{t}^{3}$  and  $C_{t} \lesssim (\hat{\sigma}_{2,d}^{2} + \hat{\sigma}_{4,d}^{2})(\log d)^{O(1)} \cdot \left( \sqrt{\frac{d}{n}} \| \delta_{t} \|_{2} + \frac{d^{2-2\psi}}{n} \overline{\Delta} \| \delta_{t} \|_{2}^{2} + \frac{d^{2.5-2\psi}}{n} \overline{\Delta}^{2} \| \delta_{t} \|_{2} + \frac{d^{4-4\psi}}{n} \overline{\Delta}^{2} \| \delta_{t} \|_{2}^{2} \right)$ .
The proof is in Appendix E. We now give an explanation for each of these bounds. For  $A_{t}$ , we establish two separate bounds, one which holds during Phase 1, and one which holds during Phases 2 and 3. Intuitively, the signal part in a neuron (i.e.  $w^{2}$ ) grows at a rate  $4\hat{\sigma}_{2,d}^{2}|D_{2,t}|$  during Phase 1 (which follows from Lemma 4.4) and in Eq. (5.2) we show that the growth rate of the coupling error due to the growth in the signal is also at most  $4\hat{\sigma}_{2,d}^{2}|D_{2,t}|$ . Intuitively, the growth rates match because the signal parts of the neurons follow similar dynamics to a power method during Phase 1. For example, in the worst case when  $\delta_{t}$  is mostly in the direction of  $e_1$ , the factor by which  $\delta_{t}$  grows is the same as that of  $w$ . More mathematically, this is due to the fact that the first-order term in Lemma 4.4 is the dominant term in  $v(w) - v(\hat{w})$  for any two particles  $w, \hat{w}$  (where  $v(w)$  is the one-dimensional velocity defined in the previous section). To get a sample complexity better than NTK, the precise constant factor 4 in the growth rate  $4\hat{\sigma}_{2,d}^{2}|D_{2,t}|w$  is important — a larger constant would lead to  $\delta_{T_1}$  being larger by a poly(d) factor, thus increasing the sample complexity by  $\mathrm{poly}(d)$ .
The same bound for  $A_{t}$  no longer applies during Phases 2 and 3. This is because the dynamics of  $w$  are no longer similar to a power method, and the higher-order terms may make a larger contribution to  $v(w) - v(\hat{w})$  than to the growth of  $w$ , or vice versa. Thus we use a looser bound  $O(1) \cdot (\hat{\sigma}_{2,d}^{2} + \hat{\sigma}_{4,d}^{2})\gamma_{2}\left\| \delta_{t}\right\|^{2}$  in Eq. (5.2). However, since the remaining running time after Phase 1,  $T_{*,\epsilon} - T_{1}$ , is very short (only poly(log log  $d$ ) — this corresponds to the second term of the running time in Theorem 3.3), the total growth of the coupling error is at most  $\exp (\mathrm{poly}(\log \log d))$ , which is sub-polynomial and does not contribute to the sample complexity. Note that if the running time during Phases 2 and 3 is longer (e.g.  $O(\log d)$ ) then the coupling error would grow by an additional  $d^{O(1)}$  factor during Phases 2 and 3, which would make our sample complexity worse than NTK.
Our upper bounds on  $B_{t}$  and  $C_t$  capture the growth of coupling error due to the finite width and samples. We establish a stronger bound for  $\mathbb{E}[B_t]$  than for  $B_{t}$ . In our proof, we use the bounds for  $B_{t}$  and  $\mathbb{E}[B_t]$  to prove the inductive hypothesis for  $\overline{\Delta}_t$  and  $\Delta_{\mathrm{max},t}$  respectively. We prove the bound for  $C_t$  by expanding the error due to finite samples into second-order and fourth-order polynomials and applying concentration inequalities for higher moments (Lemma I.7).
All of these bounds together control the growth of  $\| \hat{u}_t - \bar{u}_t\|$  at time  $t$ . Using Lemma 5.4 we can show that for some properly chosen  $\phi$  and  $\psi$ , the inductive hypothesis in Assumption 5.3 holds until  $T_{*,\epsilon}$  (see Lemma E.3 for the statement), which naturally leads to the bound in Lemma 5.2. The full proof can be found in Appendix E.
In this paper, we prove a clear sample complexity separation between vanilla gradient flow and kernel methods with any inner product kernel, including NTK. Our work leads to several directions for future research. The first question is to generalize our results to the ReLU activation. Another question is whether gradient descent can achieve less than  $d^3$  sample complexity in our setting. The work of Arous et al. [11] shows that if the population loss has information exponent 2, then a sample complexity of  $d$  (up to logarithmic factors) can be achieved with a single-neuron student network — it is an open question if a similar sample complexity could be obtained in our setting. A final open question is whether two-layer neural networks can be shown to attain arbitrarily small generalization error  $\epsilon$  — one limitation of our analysis is that we require  $\epsilon \geq \mathrm{poly}(1 / \log \log d)$ .
The authors would like to thank Zhiyuan Li for helpful discussions. The authors would like to thank the support of NSF IIS 2045685, NSF Grant CCF-1844628 and a Sloan Research Fellowship.
[1] Emmanuel Abbe, Enric Boix-Adsera, Matthew S Brennan, Guy Bresler, and Dheeraj Nagaraj. The staircase property: How hierarchical structure can guide deep learning. Advances in Neural Information Processing Systems, 34:26989-27002, 2021.
[2] Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, pages 4782-4887. PMLR, 2022.
[3] Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. arXiv preprint arXiv:2302.11055, 2023.
[4] Radosław Adamczak, Alexander E. Litvak, Alain Pajor, and Nicole Tomczak-Jaegermann. Quantitative estimates of the convergence of the empirical covariance matrix in log-concave ensembles. Journal of the American Mathematical Society, 23(2):535-561, oct 2009. doi: 10.1090/s0894-0347-09-00650-x. URL https://doi.org/10.1090%2Fs0894-0347-09-00650-x.
[5] Zeyuan Allen-Zhu and Yuanzhi Li. What can resnet learn efficiently, going beyond kernels? arXiv preprint arXiv:1905.10337, 2019.
[6] Zeyuan Allen-Zhu and Yuanzhi Li. Backward feature correction: How deep learning performs deep learning. arXiv preprint arXiv:2001.04413, 2020.
[7] Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems, 32, 2019.
[8] Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, and Bruno Loureiro. From high-dimensional & mean-field dynamics to dimensionless odes: A unifying approach to sgd in two-layers networks, 2023.
[9] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. arXiv preprint arXiv:1904.11955, 32, 2019. URL https://proceedings.neurips.cc/paper/2019/file/dbc4d84bfcfe2284ba11beffb853a8c4-Paper.pdf.
[10] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint arXiv:1901.08584, pages 322-332, 2019.
[11] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. The Journal of Machine Learning Research, 22(1):4788–4838, 2021.
[12] Kendall Atkinson and Weimin Han. Spherical harmonics and approximations on the unit sphere: an introduction, volume 2044. Springer Science & Business Media, 2012.
[13] Drumi D Bainov and Pavel S Simeonov. Integral inequalities and applications, volume 57. Springer Science & Business Media, 1992.
[14] Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems, 35:21750-21764, 2022.
[15] Gérard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Algorithmic thresholds for tensor pca. 2020.
[72] Gal Vardi, Gilad Yehudai, and Ohad Shamir. Learning a single neuron with bias using gradient descent. Advances in Neural Information Processing Systems, 34:28690-28700, 2021.
[73] Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027, 2010.
[74] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[75] Colin Wei, Jason D. Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets v.s. their induced kernel. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9709-9721, 2019. URL https://proceedings.neurips.cc/paper/2019/bitnet/8744cf92c88433f8cb04a02e6db69a0d-Abtract.html.
[76] Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. In Advances in Neural Information Processing Systems, pages 9709-9721, 2019.
[77] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. arXiv preprint arXiv:2002.09277, 2020.
[78] Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Sham M. Kakade. Learning high-dimensional single-neuron relu networks with finite samples. arXiv preprint arXiv:2303.02255, 2023.
[79] Weihang Xu and Simon S Du. Over-parameterization exponentially slows down gradient descent for learning a single neuron. arXiv preprint arXiv:2302.10034, 2023.
[80] Xiao Zhang, Yaodong Yu, Lingxiao Wang, and Quanquan Gu. Learning one-hidden-layer relu networks via gradient descent. In The 22nd international conference on artificial intelligence and statistics, pages 1524-1534. PMLR, 2019.
List of Appendices
A Additional Related Works 17
B Simulations for Population Dynamics 17
C Background on Spherical Harmonics 19
D Proofs for Population Dynamics 23
D.1 Proof of Lemma 4.2 23
D.2 Proof of Lemma 4.3 and Lemma 4.4 24
D.3 Analysis of Phase 1 27
D.3.1 Summary of Phase 1 29
D.4 Analysis of Phase 2 30
D.4.1 Summary of Phase 2 32
D.5 Phase 3, Case 1 33
D.6 Phase 3, Case 2 46
D.7 Summary of Phase 3 62
D.8 Proof of Main Theorem for Population Case 62
D.9 Additional Lemmas 62
D.10Helper Lemmas 63
E Proofs for Finite-Samples and Finite-Width Dynamics 64
E.1 Upper bound on  $A_{t}$  69
E.2 Upper Bound on  $B_{t}$  73
E.3 Upper Bound on  $C_t$  77
E.4 Additional Lemmas 80
F Proof of Theorem 3.5 84
G Missing Proofs in Section 3 94
G.1 Proof of Lemma 3.1 94
H Proofs for Sample Complexity Lower Bounds for Kernel Method 97
H.1 Proof of Theorem 3.6 97
H.2 Proof of Lemma H.1 98
I Toolbox 100
I.1 Rademacher Complexity and Generalization Bounds 100
I.2 Concentration Lemmas 102
I.3 Helper Lemmas 112
A Additional Related Works
In addition to the related works on the NTK approach, mean-field analyses, and other works analyzing the training dynamics of neural networks that we discussed in Section 1, we also discuss the following additional related works.
There are several works which study mean-field Langevin dynamics [19, 68]. These works show "uniform-in-time propagation of chaos" estimates, i.e. their results imply bounds on the coupling error between finite-width and infinite-width trajectories which do not grow exponentially in the time  $T$ . However, it is likely non-trivial to apply these analyses to directly obtain good test error and sample complexity in the setting of two-layer neural networks, since mean-field Langevin dynamics may require many iterations to obtain good population loss. More concretely, Theorem 4 of Mei et al. [56] suggests that the inverse temperature  $\lambda$  for mean-field Langevin dynamics has to be at least proportional to the dimension  $d$  in order for Langevin dynamics to achieve low population loss. This would cause the log-Sobolev constant in Suzuki et al. [68] to be on the order of  $e^{-d}$ , leading to a running time of  $e^{-d}$ . In comparison, we show in Theorem 3.5 that projected gradient descent with poly  $(d)$  iterations can achieve a low population loss. We also note that Chen et al. [19] do not study the impact of finite samples on the coupling error.
We also note that there are several works which reduce the population dynamics to a lower-dimensional dynamics [2, 37, 8], as we do in Section 4.1. The focus of our work is on the analysis of the population dynamics (Section 4.2) and coupling error between the empirical and population dynamics (Section 5), rather than the dimension-free dynamics.
Finally, we note that our setting goes beyond that of Abbe et al. [2]: our target function does not satisfy the merged-staircase property required by Abbe et al. [2], since the lowest-degree term in our target function has degree 2. The work of Abbe et al. [3] studies functions which have higher "leap complexity," meaning that terms of the target function can introduce more than one new coordinate at once, or introduce a new coordinate using a Hermite polynomial with degree greater than 1 (in the case where the inputs are Gaussian). However, Abbe et al. [3] use a non-standard algorithm which separates the coordinates based on whether they are large or small, and applying different projections to each subset of coordinates.
B Simulations for Population Dynamics
Here we include some simulations in a setting which is a simplified version of our 1-dimensional dynamics, to illustrate the different phases. Here, the goal is to find a weight vector  $w \in \mathbb{R}^d$ , where  $d = 1000$ , that minimizes the following loss:
$$
L (w) = \left(\frac {1}{d} \sum_ {i = 1} ^ {d} w _ {i} ^ {2} - \gamma_ {2}\right) ^ {2} + \left(\frac {1}{d} \sum_ {i = 1} ^ {d} w _ {i} ^ {4} - \gamma_ {4}\right) ^ {2} \tag {B.1}
$$
where  $\gamma_{2} = 0.8$  and  $\gamma_{4} = 0.7$ . Intuitively, the  $w_{i}$  play the role of the particles  $w$  in the 1-dimensional dynamics of Section 4.2. We initialize  $w$  as follows — each coordinate is initialized according to a standard Gaussian distribution with mean 0 and standard deviation 0.0001. We then train  $w$  using gradient descent on  $L(w)$  with learning rate 0.1, for 100,000 iterations. We define  $D_{2} = \frac{1}{d}\sum_{i=1}^{d}w_{i}^{2} - \gamma_{2}$  and  $D_{4} = \frac{1}{d}\sum_{i=1}^{d}w_{i}^{4} - \gamma_{4}$ .
The dynamics are illustrated by Figure 1, Figure 2, Figure 3 and Figure 4. At initialization, since the standard deviation is 0.0001, all of the  $w_{i}$  are very small, as shown in Figure 1. Now, as shown in Figure 4, while both  $D_{2}$  and  $D_{4}$  grow rapidly after a certain point,  $D_{4}$  reaches 0 first. At the point where  $D_{4}$  reaches 0, the distribution of the coordinates of  $w$  is shown in Figure 2. At this point, the coordinates  $w_{i}$  have grown substantially, with a substantial fraction of coordinates around 0.5 or larger. This illustrates the intuition that all of the coordinates grow by a large factor during Phases 1 and 2. The weight histogram after many iterations have passed during Phase 3 is shown in Figure 3. In our infinite-width 1-dimensional population dynamics, when  $D_{4} > 0$  and  $D_{2} < 0$ , the particles  $w$  which are larger than the root  $r$  of the velocity will move towards 0, while the particles  $w$  which are smaller than the root  $r$  of the velocity will move away from 0. This intuition is reflected in Figure 3, where not as many particles are concentrated around 0 compared to at the end of Phase 2, but the particles have not grown larger than 1 or -1 in absolute value.
C Background on Spherical Harmonics
In the following, we summarize the background needed for this paper based on Atkinson and Han [12, Section 2].
Spherical Harmonics. Spherical harmonics are the eigenfunctions of any inner product kernels on the unit sphere  $\mathbb{S}^{d - 1}$ . The eigenfunctions corresponding to the  $k$ -th eigenvalue are degree- $k$  polynomials, and form a Hilbert space  $\mathbb{Y}_k^d$  with inner product  $\langle f,g\rangle = \mathbb{E}_{x\sim \mathbb{S}^{d - 1}}[f(x)g(x)]$ . The dimension of  $\mathbb{Y}_k^d$  is  $N_{k,d} = \binom{d + k - 1}{d - 1} - \binom{d + k - 3}{d - 1}$ . For any universal constant  $k$ ,  $N_{k,d} \asymp d^k$ . Spherical harmonics with different degrees are orthogonal to each other, and their linear combinations can represent all square-integrable functions over  $\mathbb{S}^{d - 1}$ .
The following theorem states that  $\mathbb{Y}_k^d$  is the eigenspace corresponding to the  $k$ -th eigenvalue for any inner product kernel on the sphere.
Theorem C.1 (Funk-Hecke formula [12]). Let  $h: [-1, 1] \to \mathbb{R}$  be any one-dimensional function with  $\int_{-1}^{1} |h(t)| \mu_d(t) \mathrm{d}t < \infty$ , and  $\lambda_k = N_{k,d}^{-1/2} \mathbb{E}_{t \sim \mu_d} [h(t) \overline{P}_{k,d}(t)]$ . Then for any function  $Y_k \in \mathbb{Y}_{k,d}$ ,
$$
\forall x \in \mathbb {S} ^ {d - 1}, \quad \underset {z \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ h (\langle x, z \rangle) Y _ {k} (z) ] = \lambda_ {k} Y _ {k} (x). \tag {C.1}
$$
Legendre Polynomials. The un-normalized Legendre polynomial is defined recursively by
$$
P _ {0, d} (t) = 1, \quad P _ {1, d} (t) = t, \tag {C.2}
$$
$$
P _ {k, d} (t) = \frac {2 k + d - 4}{k + d - 3} t P _ {k - 1, d} (t) - \frac {k - 1}{k + d - 3} P _ {k - 2, d} (t), \quad \forall k \geq 2. \tag {C.3}
$$
In particular, we have
$$
P _ {2, d} (t) = \frac {d}{d - 1} t ^ {2} - \frac {1}{d - 1}, \quad P _ {4, d} (t) = \frac {(d + 2) (d + 4)}{d ^ {2} - 1} t ^ {4} - \frac {6 d + 1 2}{d ^ {2} - 1} t ^ {2} + \frac {3}{d ^ {2} - 1}. \tag {C.4}
$$
For any  $u \in \mathbb{S}^{d-1}$ ,  $P_{k,d}(\langle u, x \rangle)$  is a spherical harmonic of degree  $k$  and, for every  $k, k' \geq 0$  and  $u_1, u_2 \in \mathbb{S}^{d-1}$ ,  $\mathbb{E}_{x \sim \mathbb{S}^{d-1}}[\overline{P}_{k,d}(\langle u_1, x \rangle) \overline{P}_{k', d}(\langle u_2, x \rangle)] = 1$ $[k = k'] P_{k,d}(\langle u_1, u_2 \rangle)$  (Lemma C.5).
Projection Operator. Let  $\Pi_k$  be the projection operator to  $\mathbb{Y}_k^d$ . For any function  $f: \mathbb{S}^{d-1} \to \mathbb{R}$ , the projection operator  $\Pi_k$  is given by
$$
\left(\Pi_ {k} f\right) (x) = \sqrt {N _ {k , d}} \underset {\xi \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \bar {P} _ {k, d} (\langle x, \xi \rangle) f (\xi) \right]. \tag {C.5}
$$
In addition, if  $f$  is a function of the form  $f(x) = h(\langle u,x\rangle)$  for some fixed vector  $u\in \mathbb{S}^{d - 1}$  and one-dimensional function  $h:[-1,1]\to \mathbb{R}$ , the Funk-Hecke formula (Theorem C.1) implies
$$
\left(\Pi_ {k} f\right) (x) = \hat {h} _ {k, d} \bar {P} _ {k, d} (\langle u, x \rangle), \quad \forall x \in \mathbb {S} ^ {d - 1}, \tag {C.6}
$$
where  $\hat{h}_{k,d} = \mathbb{E}_{t\sim \mu_d}[h(t)\overline{P}_{k,d}(t)]$  is the  $k$ -th coefficient in the Legendre decomposition of  $h$ .
Note that by the orthogonality of Legendre polynomials, we have
$$
\forall x \in \mathbb {S} ^ {d - 1}, \quad \underset {u \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ \bar {P} _ {k, d} (\langle u, x \rangle) ] = \underset {t \sim \mu_ {d}} {\mathbb {E}} [ \bar {P} _ {k, d} (t) ] = \underset {t \sim \mu_ {d}} {\mathbb {E}} [ \bar {P} _ {k, d} (t) \bar {P} _ {0, d} (t) ] = \mathbf {1} [ k = 0 ]. \tag {C.7}
$$
The following lemma computes the expectation of  $\overline{P}_{k,d}(\langle u,x\rangle)$  with respect to a distribution on  $u$  which is rotationally invariant on the last  $(d - 1)$  coordinates.
Lemma C.2. Let  $\rho$  be a distribution on  $\mathbb{S}^{d - 1}$  that is rotationally invariant as in Definition 4.1. Then, for any  $x\in \mathbb{S}^{d - 1}$  we have  $\mathbb{E}_{u\sim \rho}[P_{k,d}(\langle u,x\rangle)] = \mathbb{E}_{u\sim \rho}[P_{k,d}(\langle e_1,u\rangle)]P_{k,d}(\langle e_1,x\rangle)$ .
Proof. We prove this lemma by invoking Eq. (2.167) of Atkinson and Han [12], which states that for every  $s, t \in [-1, 1]$  and  $k \geq 0, d \geq 3$ ,
$$
\int_ {- 1} ^ {1} P _ {k, d} \left(s t + \left(1 - s ^ {2}\right) ^ {1 / 2} \left(1 - t ^ {2}\right) ^ {1 / 2} \xi\right) \mu_ {d - 1} (\xi) \mathrm {d} \xi = P _ {k, d} (s) P _ {k, d} (t). \tag {C.8}
$$
In the following, for  $x \in \mathbb{S}^{d - 1}$ , we write  $x = (\langle e_1,x\rangle ,\zeta)$ , and for  $u \sim \rho$  we write  $u = (w,z)$ . Note that when  $\rho$  is symmetric and  $u \sim \rho$ , conditioned on  $u_{1}$  we have  $\langle z,\zeta \rangle \sim (1 - u_1^2)^{1 / 2}(1 - x_1^2)^{1 / 2}\mu_{d - 1}$  for any fixed  $x$ . As a result, plugging in  $t = w$ ,  $s = \langle e_1,x\rangle$  and  $\xi = (1 - u_1^2)^{-1 / 2}(1 - x_1^2)^{-1 / 2}\langle z,\zeta \rangle$  we get
$$
\underset {u \sim \rho} {\mathbb {E}} \left[ P _ {k, d} (\langle u, x \rangle) \mid u _ {1} \right] = P _ {k, d} (w) P _ {k, d} (\langle e _ {1}, x \rangle). \tag {C.9}
$$
Finally, taking expectation over  $w$  we prove the desired result.
Additional Useful Lemmas. In the following, we present some useful lemmas about spherical harmonics and Legendre polynomials.
The following proposition computes the coefficient in the Legendre polynomial decomposition of ReLU activation.
Proposition C.3 (Lemma C.2 of Dong and Ma [27]). Let  $\sigma(t) = \max\{t, 0\}$  be the ReLU activation. For every  $d \geq 3$  and even  $k$ , we have
$$
\left| \underset {t \sim \mu_ {d}} {\mathbb {E}} [ \sigma (t) \bar {P} _ {k, d} (t) ] \right| \asymp d ^ {1 / 4} k ^ {- 5 / 4} (d + k) ^ {- 3 / 4}. \tag {C.10}
$$
As a corollary, when  $k$  is an absolute constant we have
$$
\left| \underset {t \sim \mu_ {d}} {\mathbb {E}} [ \sigma (t) \bar {P} _ {k, d} (t) ] \right| \asymp d ^ {- 1 / 2}. \tag {C.11}
$$
Theorem C.4 (Addition Theorem [12]). Let  $\{Y_{k,j}:1\leq j\leq N_{k,d}\}$  be an orthonormal basis of  $\mathbb{Y}_{k,d}$ , that is,
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ Y _ {k, j} (x) Y _ {k, j ^ {\prime}} (x) \right] = \mathbf {1} [ j = j ^ {\prime} ]. \tag {C.12}
$$
Then for all  $x,z\in \mathbb{S}^{d - 1}$
$$
\sum_ {j = 1} ^ {N _ {k, d}} Y _ {k, j} (x) Y _ {k, j} (z) = N _ {k, d} P _ {k, d} (\langle x, z \rangle). \tag {C.13}
$$
Lemma C.5. For every  $k, k' \geq 0, d \geq 3$  and  $u, v \in \mathbb{S}^{d-1}$ , we have
$$
\underset {\xi \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \bar {P} _ {k, d} (\langle u, \xi \rangle) \bar {P} _ {k ^ {\prime}, d} (\langle v, \xi \rangle) \right] = \mathbf {1} [ k = k ^ {\prime} ] P _ {k, d} (\langle u, v \rangle). \tag {C.14}
$$
Proof. Let  $\{Y_{k,j}:1\leq j\leq N_{k,d}\}$  be an orthonormal basis of  $\mathbb{Y}_{k,d}$  with
$$
\underset {\xi \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ Y _ {k, j} (\xi) Y _ {k, j ^ {\prime}} (\xi) \right] = \mathbf {1} [ j = j ^ {\prime} ]. \tag {C.15}
$$
Similarly define  $\{Y_{k',j}:1\leq j\leq N_{k',d}\}$  for  $\mathbb{Y}_{k',d}$ . The addition theorem (Theorem C.4) implies that
$$
P _ {k, d} (\langle u, \xi \rangle) = \frac {1}{N _ {k , d}} \sum_ {j = 1} ^ {N _ {k, d}} Y _ {k, j} (u) Y _ {k, j} (\xi). \tag {C.16}
$$
$$
P _ {k ^ {\prime}, d} (\langle u, \xi \rangle) = \frac {1}{N _ {k ^ {\prime} , d}} \sum_ {j = 1} ^ {N _ {k ^ {\prime}, d}} Y _ {k ^ {\prime}, j} (u) Y _ {k ^ {\prime}, j} (\xi). \tag {C.17}
$$
Recall that for  $k \neq k'$ ,  $\mathbb{Y}_{k,d}$  and  $\mathbb{Y}_{k',d}$  are orthonormal subspaces. Consequently,