SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A.2 Hermite Polynomials and Hermite Tensors
We provide a brief review of the properties of Hermite polynomials and Hermite tensors.
Definition 10. We define the kth Hermite polynomial  $H e_{k}(x)$  by
$$
H e _ {k} (x) := (- 1) ^ {k} \frac {\nabla^ {k} \mu (x)}{\mu (x)}
$$
where  $\mu(x) := \frac{e^{-\frac{\|x\|^2}{2}}}{(2\pi)^{d/2}}$  is the PDF of a standard Gaussian in  $d$  dimensions. Note that when  $d = 1$ , this definition reduces to the standard univariate Hermite polynomials.
We begin with the classical properties of the scalar Hermite polynomials:
Lemma 4 (Properties of Hermite Polynomials). When  $d = 1$
- Orthogonality:
$$
\mathbb {E} _ {x \sim N (0, 1)} [ H e _ {j} (x) H e _ {k} (x) ] = k! \delta_ {j k}
$$
- Derivatives:
$$
\frac {d}{d x} H e _ {k} (x) = k H e _ {k - 1} (x)
$$
Correlations: If  $x, y \sim N(0,1)$  are correlated Gaussians with correlation  $\alpha \coloneqq \mathbb{E}[xy]$ ,
$$
\mathbb {E} _ {x, y} [ H e _ {j} (x) H e _ {k} (y) ] = k! \delta_ {j k} \alpha^ {k}.
$$
- Hermite Expansion: If  $f \in L^{2}(\mu)$  where  $\mu$  is the PDF of a standard Gaussian,
$$
f (x) \overset {L ^ {2} (\mu)} {=} \sum_ {k \geq 0} \frac {c _ {k}}{k !} H e _ {k} (x) \quad \text {w h e r e} \quad c _ {k} = \mathbb {E} _ {x \sim N (0, 1)} [ f (x) H e _ {k} (x) ].
$$
These properties also have tensor analogues:
Lemma 5 (Hermite Polynomials in Higher Dimensions).
- Relationship to Univariate Hermite Polynomials: If  $\| w\| = 1$
$$
H e _ {k} (w \cdot x) = \left\langle H e _ {k} (x), w ^ {\otimes k} \right\rangle
$$
Orthogonality:
$$
\mathbb {E} _ {x \sim N (0, I _ {d})} [ H e _ {j} (x) \otimes H e _ {k} (x) ] = \delta_ {j k} k! \operatorname {S y m} _ {k}
$$
or equivalently, for any  $j$  tensor  $A$  and  $k$  tensor  $B$ :
$$
\mathbb {E} _ {x \sim N (0, I _ {d})} \left[ \langle H e _ {j} (x), A \rangle \langle H e _ {k} (x), B \rangle \right] = \delta_ {j k} k! \langle \operatorname {S y m} (A), \operatorname {S y m} (B) \rangle .
$$
- Hermite Expansions: If  $f: \mathbb{R}^d \to \mathbb{R}$  satisfies  $\mathbb{E}_{x \sim N(0, I_d)}[f(x)^2] < \infty$ ,
$$
f = \sum_ {k \geq 0} \frac {1}{k !} \left\langle H e _ {k} (x), C _ {k} \right\rangle \quad \text {w h e r e} \quad C _ {k} = \mathbb {E} _ {x \sim N \left(0, I _ {d}\right)} [ f (x) H e _ {k} (x) ].
$$
The proof of Theorem 1 is divided into four parts. First, Appendix B.1 introduces some notation that will be used throughout the proof. Next, Appendix B.2 computes matching upper and lower bounds for the gradient of the smoothed population loss. Similarly, Appendix B.3 concentrates the empirical gradient of the smoothed loss. Finally, Appendix B.4 combines the bounds in Appendix B.2 and Appendix B.3 with a standard online SGD analysis to arrive at the final rate.
B.1 Additional Notation
Throughout the proof we will assume that  $w \in S^{d - 1}$  so that  $\nabla_w$  denotes the spherical gradient of  $w$ . In particular,  $\nabla_w g(w) \perp w$  for any  $g: S^{d - 1} \to \mathbb{R}$ . We will also use  $\alpha$  to denote  $w \cdot w^\star$  so that we can write expressions such as:
$$
\nabla_ {w} \alpha = P _ {w} ^ {\perp} w ^ {\star} = w ^ {\star} - \alpha w.
$$
We will use the following assumption on  $\lambda$  without reference throughout the proof:
Assumption 2.  $\lambda^2\leq d / C$  for a sufficiently large constant  $C$
We note that this is satisfied for the optimal choice of  $\lambda = d^{1/4}$ .
We will use  $\tilde{O}(\cdot)$  to hide  $\mathrm{polylog}(d)$  dependencies. Explicitly,  $X = \tilde{O}(1)$  if there exists  $C_1, C_2 > 0$  such that  $|X| \leq C_1 \log(d)^{C_2}$ . We will also use the following shorthand for denoting high probability events:
Definition 11. We say an event  $E$  happens with high probability if for every  $k \geq 0$  there exists  $d(k)$  such that for all  $d \geq d(k)$ ,  $\mathbb{P}[E] \geq 1 - d^{-k}$ .
Note that high probability events are closed under polynomially sized union bounds. As an example, if  $X \sim N(0,1)$  then  $X \leq \log(d)$  with high probability because
$$
\mathbb {P} [ x > \log (d) ] \leq \exp \left(- \log (d) ^ {2} / 2\right) \ll d ^ {- k}
$$
for sufficiently large  $d$ . In general, Lemma 24 shows that if  $X$  is mean zero and has polynomial tails, i.e. there exists  $C$  such that  $E[X^p]^{1/p} \leq p^C$ , then  $X = \tilde{O}(1)$  with high probability.
B.2 Computing the Smoothed Population Gradient
$$
\sigma (x) \stackrel {L ^ {2} (\mu)} {=} \sum_ {k \geq 0} \frac {c _ {k}}{k !} H e _ {k} (x) \quad \text {w h e r e} \quad c _ {k} := \mathbb {E} _ {x \sim N (0, 1)} [ \sigma (x) H e _ {k} (x) ].
$$
In addition, because we assumed that  $\mathbb{E}_{x\sim N(0,1)}[\sigma (x)^2 ] = 1$  we have Parseval's identity:
$$
1 = \mathbb {E} _ {x \sim N (0, 1)} [ \sigma (x) ^ {2} ] = \sum_ {k \geq 0} \frac {c _ {k} ^ {2}}{k !}.
$$
This Hermite decomposition immediately implies a closed form for the population loss:
Lemma 6 (Population Loss). Let  $\alpha = w\cdot w^{\star}$  . Then,
$$
L (w) = \sum_ {k \geq 0} \frac {c _ {k} ^ {2}}{k !} [ 1 - \alpha^ {k} ].
$$
Lemma 6 implies that to understand the smoothed population  $L_{\lambda}(w) = (\mathcal{L}_{\lambda}L)(w)$ , it suffices to understand  $\mathcal{L}_{\lambda}(\alpha^{k})$  for  $k \geq 0$ . First, we will show that the set of single index models is closed under smoothing operator  $\mathcal{L}_{\lambda}$ :
Lemma 7. Let  $g:[-1,1]\to \mathbb{R}$  and let  $u\in S^{d - 1}$ . Then
$$
\mathcal {L} _ {\lambda} (g (w \cdot u)) = g _ {\lambda} (w \cdot u)
$$
$$
g _ {\lambda} (\alpha) := \mathbb {E} _ {z \sim \mathrm {U n i f} (S ^ {d - 2})} \left[ g \left(\frac {\alpha + \lambda z _ {1} \sqrt {1 - \alpha^ {2}}}{\sqrt {1 + \lambda^ {2}}}\right) \right]
$$
and  $z_{1}$  denotes the first coordinate of  $z$
Proof. Expanding the definition of  $\mathcal{L}_{\lambda}$  gives:
$$
\begin{array}{l} \mathcal {L} _ {\lambda} (g (w \cdot u)) = \mathbb {E} _ {z \sim \mu_ {w}} \left[ g \left(\frac {w + \lambda z}{\| w + \lambda z \|} \cdot u\right) \right] \\ = \mathbb {E} _ {z \sim \mu_ {w}} \left[ g \left(\frac {w \cdot u + \lambda z \cdot u}{\sqrt {1 + \lambda^ {2}}}\right) \right]. \\ \end{array}
$$
Now I claim that when  $z \sim \mu_w, z \cdot u \stackrel{d}{=} z_1 \sqrt{1 - (w \cdot u)^2}$  where  $z \operatorname{Unif}(\sim \operatorname{Unif}(S^{d-2}))$  which would complete the proof. To see this, note that we can decompose  $R^d$  into  $\operatorname{span}\{w\} \oplus \operatorname{span}\{w\}^\perp$ . Under this decomposition we have the polyspherical decomposition  $z \stackrel{d}{=} (0, z')$  where  $z' \sim \operatorname{Unif}(S^{d-2})$ . Then
$$
z \cdot u = z ^ {\prime} \cdot P _ {w} ^ {\perp} u \stackrel {d} {=} z _ {1} \| P _ {w} ^ {\perp} u \| = z _ {1} \sqrt {1 - (w \cdot u) ^ {2}}.
$$
Of central interest are the quantities  $\mathcal{L}_{\lambda}(\alpha^{k})$  as these terms show up when smoothing the population loss (see Lemma 6). We begin by defining the quantity  $s_k(\alpha; \lambda)$  which will provide matching upper and lower bounds on  $\mathcal{L}_{\lambda}(\alpha^{k})$  when  $\alpha \geq 0$ :
Definition 12. We define  $s_k(\alpha; \lambda)$  by
$$
s _ {k} (\alpha ; \lambda) := \frac {1}{(1 + \lambda^ {2}) ^ {k / 2}} \left\{ \begin{array}{l l} \alpha^ {k} & \alpha^ {2} \geq \frac {\lambda^ {2}}{d} \\ \left(\frac {\lambda^ {2}}{d}\right) ^ {\frac {k}{2}} & \alpha^ {2} \leq \frac {\lambda^ {2}}{d} a n d k i s e v e n \\ \alpha \left(\frac {\lambda^ {2}}{d}\right) ^ {\frac {k - 1}{2}} & \alpha^ {2} \leq \frac {\lambda^ {2}}{d} a n d k i s o d d \end{array} . \right.
$$
Lemma 8. For all  $k \geq 0$  and  $\alpha \geq 0$ , there exist constants  $c(k), C(k)$  such that
$$
c (k) s _ {k} (\alpha ; \lambda) \leq \mathcal {L} _ {\lambda} \left(\alpha^ {k}\right) \leq C (k) s _ {k} (\alpha ; \lambda).
$$
Proof. Using Lemma 7 we have that
$$
\begin{array}{l} \mathcal {L} _ {\lambda} (\alpha^ {k}) = \mathbb {E} _ {z \sim \operatorname {U n i f} (S ^ {d - 2})} \left[ \left(\frac {\alpha + \lambda z _ {1} \sqrt {1 - \alpha^ {2}}}{\sqrt {1 + \lambda^ {2}}}\right) ^ {k} \right] \\ = (1 + \lambda^ {2}) ^ {- k / 2} \sum_ {j = 0} ^ {k} \binom {k} {j} \alpha^ {k - j} \lambda^ {j} (1 - \alpha^ {2}) ^ {j / 2} \mathbb {E} _ {z \sim \mathrm {U n i f} (S ^ {d - 2})} [ z _ {1} ^ {j} ]. \\ \end{array}
$$
Now note that when  $j$  is odd,  $\mathbb{E}_{z\sim \mathrm{Unif}(S^{d - 2})}[z_1^j ] = 0$  so we can re-index this sum to get
$$
\begin{array}{l} \mathcal{L}_{\lambda}(\alpha^{k}) = (1 + \lambda^{2})^{-k / 2}\sum_{j = 0}^{\lfloor \frac{k}{2}\rfloor}\binom {k}{2j}\alpha^{k - 2j}\lambda^{2j}(1 - \alpha^{2})^{j}\mathbb{E}_{z\sim \operatorname {Unif}(S^{d - 2})}[z_{1}^{2j}] \\ = (1 + \lambda^ {2}) ^ {- k / 2} \sum_ {j = 0} ^ {\lfloor \frac {k}{2} \rfloor} \binom {k} {2 j} \alpha^ {k - 2 j} \lambda^ {2 j} (1 - \alpha^ {2}) ^ {j} \nu_ {j} ^ {(d - 1)}. \\ \end{array}
$$
Note that every term in this sum is non-negative. Now we can ignore constants depending on  $k$  and use that  $\nu_{j}^{(d - 1)}\asymp d^{-j}$  to get
$$
\mathcal {L} _ {\lambda} (\alpha^ {k}) \asymp \left(\frac {\alpha}{\sqrt {1 + \lambda^ {2}}}\right) ^ {k} \sum_ {j = 0} ^ {\lfloor \frac {k}{2} \rfloor} \left(\frac {\lambda^ {2} (1 - \alpha^ {2})}{\alpha^ {2} d}\right) ^ {j}.
$$
Now when  $\alpha^2 \geq \frac{\lambda^2}{d}$ , this is a decreasing geometric series which is dominated by the first term so  $\mathcal{L}_{\lambda}(\alpha^k) \asymp \left(\frac{\alpha}{\lambda}\right)^k$ . Next, when  $\alpha^2 \leq \frac{\lambda^2}{d}$  we have by Assumption 2 that  $\alpha \leq \frac{1}{C}$  so  $1 - \alpha^2$  is bounded
away from 0. Therefore the geometric series is dominated by the last term which is
$$
\frac {1}{(1 + \lambda^ {2}) ^ {- k / 2}} \left\{ \begin{array}{l l} \left(\frac {\lambda^ {2}}{d}\right) ^ {\frac {k}{2}} & k \text {i s e v e n} \\ \alpha \left(\frac {\lambda^ {2}}{d}\right) ^ {\frac {k - 1}{2}} & k \text {i s o d d} \end{array} \right.
$$
Next, in order to understand the population gradient, we need to understand how the smoothing operator  $\mathcal{L}_{\lambda}$  commutes with differentiation. We note that these do not directly commute because the smoothing distribution  $\mu_w$  depends on  $w$  so this term must be differentiated as well. However, smoothing and differentiation almost commute, which is described in the following lemma:
Lemma 9. Define the dimension-dependent univariate smoothing operator by:
$$
\mathcal {L} _ {\lambda} ^ {(d)} g (\alpha) := \mathbb {E} _ {z \sim \mathrm {U n i f} (S ^ {d - 2})} \left[ g \left(\frac {\alpha + \lambda z _ {1} \sqrt {1 - \alpha^ {2}}}{\sqrt {1 + \lambda^ {2}}}\right) \right].
$$
$$
\frac {d}{d \alpha} \mathcal {L} _ {\lambda} ^ {(d)} (g (\alpha)) = \frac {\mathcal {L} _ {\lambda} ^ {(d)} (g ^ {\prime} (\alpha))}{\sqrt {1 + \lambda^ {2}}} - \frac {\lambda^ {2} \alpha}{(1 + \lambda^ {2}) (d - 1)} \mathcal {L} _ {\lambda} ^ {(d + 2)} (g ^ {\prime \prime} (\alpha)).
$$
Proof. Directly differentiating the definition for  $\mathcal{L}_{\lambda}^{(d)}$  gives
$$
\frac {d}{d \alpha} \mathcal {L} _ {\lambda} ^ {(d)} = \frac {\mathcal {L} _ {\lambda} ^ {(d)} (g ^ {\prime} (\alpha))}{\sqrt {1 + \lambda^ {2}}} - \mathbb {E} _ {z \sim \mathrm {U n i f} (S ^ {d - 2})} \left[ \frac {\alpha \lambda z _ {1}}{\sqrt {(1 + \lambda^ {2}) (1 - \alpha^ {2})}} g ^ {\prime} \left(\frac {\alpha + \lambda z _ {1} \sqrt {1 - \alpha^ {2}}}{\sqrt {1 + \lambda^ {2}}}\right) \right].
$$
By Lemma 25, this is equal to
$$
\begin{array}{l} \frac {d}{d \alpha} \mathcal {L} _ {\lambda} ^ {(d)} = \frac {\mathcal {L} _ {\lambda} ^ {(d)} (g ^ {\prime} (\alpha))}{\sqrt {1 + \lambda^ {2}}} - \frac {\lambda^ {2} \alpha}{(1 + \lambda^ {2}) (d - 1)} \mathbb {E} _ {z \sim \mathrm {U n i f} (S ^ {d})} \left[ g ^ {\prime \prime} \left(\frac {\alpha + \lambda z _ {1} \sqrt {1 - \alpha^ {2}}}{\sqrt {1 + \lambda^ {2}}}\right) \right] \\ = \frac {\mathcal {L} _ {\lambda} ^ {(d)} (g ^ {\prime} (\alpha))}{\sqrt {1 + \lambda^ {2}}} - \frac {\lambda^ {2} \alpha}{(1 + \lambda^ {2}) (d - 1)} \mathcal {L} _ {\lambda} ^ {(d + 2)} (g ^ {\prime \prime} (\alpha)). \\ \end{array}
$$
Now we are ready to analyze the population gradient:
Lemma 10.
$$
\nabla_ {w} L _ {\lambda} (w) = - \left(w ^ {\star} - \alpha w\right) c _ {\lambda} (\alpha)
$$
where for  $\alpha \geq C^{-1 / 4}d^{-1 / 2}$
$$
c _ {\lambda} (\alpha) \asymp \frac {s _ {k ^ {*} - 1} (\alpha ; \lambda)}{\sqrt {1 + \lambda^ {2}}}.
$$
$$
L (w) = 1 - \sum_ {k \geq 0} \frac {c _ {k} ^ {2}}{k !} \alpha^ {k}.
$$
Because  $k^{\star}$  is the index of the first nonzero Hermite coefficient, we can start this sum at  $k = k^{\star}$ . Smoothing and differentiating gives:
$$
\nabla_ {w} L _ {\lambda} (w) = - \left(w ^ {\star} - \alpha w\right) c _ {\lambda} (\alpha) \quad \text {w h e r e} \quad c _ {\lambda} (\alpha) := \sum_ {k \geq k ^ {\star}} \frac {c _ {k} ^ {2}}{k !} \frac {d}{d \alpha} \mathcal {L} _ {\lambda} \left(\alpha^ {k}\right).
$$
We will break this into the  $k = k^{\star}$  term and the  $k > k^{\star}$  tail. First when  $k = k^{\star}$  we can use Lemma 9 and Lemma 8 to get:
$$
\frac {c _ {k ^ {\star}} ^ {2}}{(k ^ {\star}) !} \frac {d}{d \alpha} \mathcal {L} _ {\lambda} (\alpha^ {k ^ {\star}}) = \frac {c _ {k ^ {\star}} ^ {2}}{(k ^ {\star}) !} \left[ \frac {k ^ {\star} \mathcal {L} _ {\lambda} (\alpha^ {k ^ {\star} - 1})}{\sqrt {1 + \lambda^ {2}}} - \frac {k ^ {\star} (k ^ {\star} - 1) \lambda^ {2} \alpha \mathcal {L} _ {\lambda} ^ {(d + 2)} (\alpha^ {k ^ {\star} - 2})}{(1 + \lambda^ {2}) (d - 1)} \right].
$$
The first term is equal up to constants to  $\frac{s_{k^{\star} - 1}(\alpha;\lambda)}{\sqrt{1 + \lambda^{2}}}$  while the second term is equal up to constants to  $\frac{\lambda^2\alpha}{(1 + \lambda^2)d} s_{k^{\star} - 2}(\alpha ;\lambda)$ . However, we have that
$$
\frac {\frac {\lambda^ {2} \alpha}{(1 + \lambda^ {2}) d} s _ {k ^ {\star} - 2} (\alpha ; \lambda)}{\frac {s _ {k ^ {\star} - 1} (\alpha ; \lambda)}{\sqrt {1 + \lambda^ {2}}}} = \left\{ \begin{array}{l l} \frac {\lambda^ {2}}{d} & \alpha^ {2} \geq \frac {\lambda^ {2}}{d} \\ \frac {\lambda^ {2}}{d} & \alpha^ {2} \leq \frac {\lambda^ {2}}{d} \text {a n d} k ^ {\star} \text {i s e v e n} \leq \frac {\lambda^ {2}}{d} \leq \frac {1}{C}. \\ \alpha^ {2} & \alpha^ {2} \leq \frac {\lambda^ {2}}{d} \text {a n d} k ^ {\star} \text {i s o d d} \end{array} \right.
$$
Therefore the  $k = k^{\star}$  term in  $c_{\lambda}(\alpha)$  is equal up to constants to  $\frac{s_{k^{\star} - 1}(\alpha;\lambda)}{\sqrt{1 + \lambda^2}}$
Next, we handle the  $k > k^{\star}$  tail. By Lemma 9 this is equal to
$$
\sum_ {k > k ^ {*}} \frac {c _ {k} ^ {2}}{k !} \left[ \frac {k \mathcal {L} _ {\lambda} (\alpha^ {k - 1})}{\sqrt {1 + \lambda^ {2}}} - \frac {k (k - 1) \lambda^ {2} \alpha \mathcal {L} _ {\lambda} ^ {(d + 2)} (\alpha^ {k - 2})}{(1 + \lambda^ {2}) (d - 1)} \right].
$$
Now recall that from Lemma 8,  $\mathcal{L}_{\lambda}(\alpha^{k})$  is always non-negative so we can use  $c_k^2 \leq k!$  to bound this tail in absolute value by
$$
\begin{array}{l} \sum_ {k > k ^ {\star}} \frac {k \mathcal {L} _ {\lambda} (\alpha^ {k - 1})}{\sqrt {1 + \lambda^ {2}}} + \frac {k (k - 1) \lambda^ {2} \alpha \mathcal {L} _ {\lambda} ^ {(d + 2)} (\alpha^ {k - 2})}{(1 + \lambda^ {2}) (d - 1)} \\ \lesssim \frac {1}{\sqrt {1 + \lambda^ {2}}} \mathcal {L} _ {\lambda} \left(\sum_ {k > k ^ {\star}} k \alpha^ {k - 1}\right) + \frac {\lambda^ {2} \alpha}{(1 + \lambda^ {2}) d} \mathcal {L} _ {\lambda} ^ {(d + 2)} \left(\sum_ {k > k ^ {\star}} k (k - 1) \alpha^ {k - 2}\right) \\ \lesssim \frac {1}{\sqrt {1 + \lambda^ {2}}} \mathcal {L} _ {\lambda} \left(\frac {\alpha^ {k ^ {\star}}}{(1 - \alpha) ^ {2}}\right) + \frac {\lambda^ {2} \alpha}{(1 + \lambda^ {2}) d} \mathcal {L} _ {\lambda} ^ {(d + 2)} \left(\frac {\alpha^ {k ^ {\star} - 1}}{(1 - \alpha) ^ {3}}\right). \\ \end{array}
$$
Now by Corollary 3, this is bounded for  $d \geq 5$  by
$$
\frac {s _ {k ^ {\star}} (\alpha ; \lambda)}{\sqrt {1 + \lambda^ {2}}} + \frac {\lambda^ {2} \alpha}{(1 + \lambda^ {2}) d} s _ {k ^ {\star} - 1} (\alpha ; \lambda).
$$
$$
\frac {s _ {k ^ {\star}} (\alpha ; \lambda)}{s _ {k ^ {\star} - 1} (\alpha ; \lambda)} = \left\{ \begin{array}{l l} \frac {\alpha}{\lambda} & \alpha^ {2} \geq \frac {\lambda^ {2}}{d} \\ \frac {\lambda}{d \alpha} & \alpha^ {2} \leq \frac {\lambda^ {2}}{d} \text {a n d} k ^ {\star} \text {i s e v e n} \leq C ^ {- 1 / 4}. \\ \frac {\alpha}{\lambda} & \alpha^ {2} \leq \frac {\lambda^ {2}}{d} \text {a n d} k ^ {\star} \text {i s o d d} \end{array} \right.
$$
The second term is trivially bounded by
$$
\frac {\lambda^ {2} \alpha}{(1 + \lambda^ {2}) d} \leq \frac {\lambda^ {2}}{(1 + \lambda^ {2}) d} \leq \frac {1}{C (1 + \lambda^ {2})} \leq \frac {1}{C \sqrt {1 + \lambda^ {2}}}
$$
B.3 Concentrating the Empirical Gradient
We cannot directly apply Lemma 7 to  $\sigma (w\cdot x)$  as  $\| x\| \neq 1$ . Instead, we will use the properties of the Hermite tensors to directly smooth  $\sigma (w\cdot x)$ .
$$
\mathcal {L} _ {\lambda} \left(H e _ {k} (w \cdot x)\right) = \langle H e _ {k} (x), T _ {k} (w) \rangle
$$
$$
T_{k}(w) = (1 + \lambda^{2})^{-\frac{k}{2}}\sum_{j = 0}^{\lfloor \frac{k}{2}\rfloor}\binom {k}{2j}w^{\otimes (k - 2j)}\widetilde{\otimes} (P_{w}^{\perp})^{\widetilde{\otimes} j}\lambda^{2j}\nu_{j}^{(d - 1)}.
$$
Proof. Using Lemma 5, we can write
$$
\mathcal {L} _ {\lambda} \left(H e _ {k} (w \cdot x)\right) = \left\langle H e _ {k} (x), \mathcal {L} _ {\lambda} \left(w ^ {\otimes k}\right) \right\rangle .
$$
$$
\begin{array}{l} T _ {k} (w) = \mathcal {L} _ {\lambda} \left(w ^ {\otimes k}\right) \\ = \mathbb {E} _ {z \sim \mu_ {w}} \left[ \left(\frac {w + \lambda z}{\sqrt {1 + \lambda^ {2}}}\right) ^ {\otimes k} \right] \\ = (1 + \lambda^ {2}) ^ {- \frac {k}{2}} \sum_ {j = 0} ^ {k} \binom {k} {j} w ^ {\otimes (k - j)} \widetilde {\otimes} \mathbb {E} _ {z \sim \mu_ {w}} [ z ^ {\otimes j} ] \lambda^ {j}. \\ \end{array}
$$
Now by Lemma 2, this is equal to
$$
T_{k}(w) = (1 + \lambda^{2})^{-\frac{k}{2}}\sum_{j = 0}^{\lfloor \frac{k}{2}\rfloor}\binom {k}{2j}w^{\otimes (k - 2j)}\widetilde{\otimes} (P_{w}^{\perp})^{\widetilde{\otimes} j}\lambda^{2j}\nu_{j}^{(d - 1)}
$$
Lemma 12. For any  $u \in S^{d-1}$  with  $u \perp w$ ,
$$
\begin{array}{l} \mathbb {E} _ {x \sim N (0, I _ {d})} \left[ \left(u \cdot \nabla_ {w} \mathcal {L} _ {\lambda} \left(H e _ {k} (w \cdot x)\right)\right) ^ {2} \right] \\ \lesssim k! \left(\frac {k ^ {2}}{1 + \lambda^ {2}} \mathcal {L} _ {\lambda} \big (\alpha^ {k - 1} \big) + \frac {\lambda^ {4} k ^ {4}}{(1 + \lambda^ {2}) ^ {2} d ^ {2}} \mathcal {L} _ {\lambda} ^ {(d + 2)} \big (\alpha^ {k - 2} \big)\right) \Bigg | _ {\alpha = \frac {1}{\sqrt {1 + \lambda^ {2}}}}. \\ \end{array}
$$
Proof. Recall that by Lemma 11 we have
$$
T_{k}(w):= (1 + \lambda^{2})^{-\frac{k}{2}}\sum_{j = 0}^{\lfloor \frac{k}{2}\rfloor}\binom {k}{2j}w^{\otimes (k - 2j)}\widetilde{\otimes} (P_{w}^{\perp})^{\widetilde{\otimes} j}\lambda^{2j}\nu_{j}^{(d - 1)}.
$$
Differentiating this with respect to  $w$  gives
$$
u \cdot \nabla_ {w} \mathcal {L} _ {\lambda} (H e _ {k} (w \cdot x)) = \left\langle H e _ {k} (x), \nabla_ {w} T _ {k} (w) [ u ] \right\rangle .
$$
Now note that by Lemma 5:
$$
\begin{array}{l} \mathbb {E} _ {x \sim N (0, I _ {d})} \left[ \left(u \cdot \nabla_ {w} \mathcal {L} _ {\lambda} \left(H e _ {k} (w \cdot x)\right)\right) ^ {2} \right] = \mathbb {E} _ {x \sim N (0, I _ {d})} \left[ \left\langle H e _ {k} (x), \nabla_ {w} T _ {k} (w) [ u ] \right\rangle^ {2} \right] \\ = k! \| \nabla_ {w} T _ {k} (w) [ u ] \| _ {F} ^ {2}. \\ \end{array}
$$
Therefore it suffices to compute the Frobenius norm of  $\nabla_w T_k(w)[u]$ . We first explicitly differentiate  $T_k(w)$ :
$$
\begin{array}{l} \nabla_ {w} T _ {k} (w) [ u ] = (1 + \lambda^ {2}) ^ {- \frac {k}{2}} \sum_ {j = 0} ^ {\lfloor \frac {k}{2} \rfloor} \binom {k} {2 j} (k - 2 j) u \widetilde {\otimes} w ^ {\otimes k - 2 j - 1} \widetilde {\otimes} (P _ {w} ^ {\perp}) ^ {\widetilde {\otimes} j} \lambda^ {2 j} \nu_ {j} ^ {(d - 1)} \\ - (1 + \lambda^ {2}) ^ {- \frac {k}{2}} \sum_ {j = 0} ^ {\lfloor \frac {k}{2} \rfloor} \binom {k} {2 j} (2 j) u \widetilde {\otimes} w ^ {\otimes k - 2 j + 1} \widetilde {\otimes} (P _ {w} ^ {\perp}) ^ {\widetilde {\otimes} (j - 1)} \lambda^ {2 j} \nu_ {j} ^ {(d - 1)} \\ = \frac {k}{\left(1 + \lambda^ {2}\right) ^ {\frac {k}{2}}} \sum_ {j = 0} ^ {\lfloor \frac {k - 1}{2} \rfloor} \binom {k - 1} {2 j} u \widetilde {\otimes} w ^ {\otimes k - 1 - 2 j} \widetilde {\otimes} (P _ {w} ^ {\perp}) ^ {\widetilde {\otimes} j} \lambda^ {2 j} \nu_ {j} ^ {(d - 1)} \\ -\frac{\lambda^{2}k(k - 1)}{(d - 1)(1 + \lambda^{2})^{\frac{k}{2}}}\sum_{j = 0}^{\lfloor \frac{k - 2}{2}\rfloor}\binom {k - 2}{2j}u\widetilde{\otimes} w^{\otimes k - 1 - 2j}\widetilde{\otimes} (P_{w}^{\perp})^{\widetilde{\otimes} j}\lambda^{2j}\nu_{j}^{(d + 1)}. \\ \end{array}
$$
Taking Frobenius norms gives
$$
\begin{array}{l} \left\| \nabla_ {w} T _ {k} (w) [ u ] \right\| _ {F} ^ {2} \\ \lesssim \frac{k^{2}}{(1 + \lambda^{2})^{k}}\left\| \sum_{j = 0}^{\lfloor \frac{k - 1}{2}\rfloor}\binom {k - 1}{2j}u\widetilde{\otimes} w^{\otimes k - 1 - 2j}\widetilde{\otimes}(P_{w}^{\perp})^{\widetilde{\otimes} j}\lambda^{2j}\nu_{j}^{(d - 1)}\right\|_{F}^{2} \\ + \frac {\lambda^ {4} k ^ {4}}{(d - 1) ^ {2} (1 + \lambda^ {2}) ^ {k}} \left\| \sum_ {j = 0} ^ {\lfloor \frac {k - 2}{2} \rfloor} \binom {k - 2} {2 j} u \widetilde {\otimes} w ^ {\otimes k - 1 - 2 j} \widetilde {\otimes} (P _ {w} ^ {\perp}) ^ {\widetilde {\otimes} j} \lambda^ {2 j} \nu_ {j} ^ {(d + 1)} \right\| _ {F} ^ {2}. \\ \end{array}
$$
Now we can use Lemma 1 to pull out  $u$  and get:
$$
\begin{array}{l} \left\| \nabla_ {w} T _ {k} (w) [ u ] \right\| _ {F} ^ {2} \\ \lesssim \frac{k^{2}}{(1 + \lambda^{2})^{k}}\left\| \sum_{j = 0}^{\lfloor \frac{k - 1}{2}\rfloor}\binom {k - 1}{2j}w^{\otimes k - 1 - 2j}\widetilde{\otimes}(P_{w}^{\perp})^{\widetilde{\otimes} j}\lambda^{2j}\nu_{j}^{(d - 1)}\right\|_{F}^{2} \\ + \frac {\lambda^ {4} k ^ {4}}{d ^ {2} (1 + \lambda^ {2}) ^ {k}} \left\| \sum_ {j = 0} ^ {\lfloor \frac {k - 2}{2} \rfloor} \binom {k - 2} {2 j} w ^ {\otimes k - 1 - 2 j} \widetilde {\otimes} (P _ {w} ^ {\perp}) ^ {\widetilde {\otimes} j} \lambda^ {2 j} \nu_ {j} ^ {(d + 1)} \right\| _ {F} ^ {2}. \\ \end{array}
$$
Now note that the terms in each sum are orthogonal as at least one  $w$  will need to be contracted with a  $P_w^\perp$ . Therefore this is equivalent to:
$$
\begin{array}{l} \left\| \nabla_ {w} T _ {k} (w) [ u ] \right\| _ {F} ^ {2} \\ \lesssim \frac{k^{2}}{(1 + \lambda^{2})^{k}}\sum_{j = 0}^{\lfloor \frac{k - 1}{2}\rfloor}\binom {k - 1}{2j}^{2}\lambda^{4j}(\nu_{j}^{(d - 1)})^{2}\bigg\|w^{\otimes k - 1 - 2j}\widetilde{\otimes} (P_{w}^{\perp})^{\widetilde{\otimes} j}\bigg\|_{F}^{2} \\ + \frac {\lambda^ {4} k ^ {4}}{d ^ {2} (1 + \lambda^ {2}) ^ {k}} \sum_ {j = 0} ^ {\lfloor \frac {k - 2}{2} \rfloor} \binom {k - 2} {2 j} ^ {2} \lambda^ {4 j} (\nu_ {j} ^ {(d + 1)}) ^ {2} \Big \| w ^ {\otimes k - 1 - 2 j} \widetilde {\otimes} (P _ {w} ^ {\perp}) ^ {\widetilde {\otimes} j} \Big \| _ {F} ^ {2}. \\ \end{array}
$$
Next, note that for any  $k$  tensor  $A$ ,  $\| \mathrm{Sym}(A)\| _F^2 = \frac{1}{k!}\sum_{\pi \in S_k}\langle A,\pi (A)\rangle$ . When  $A = w^{\otimes k - 2j}\widetilde{\otimes} (P_w^\perp)^{\widetilde{\otimes} j}$ , the only permutations that don't give 0 are the ones which pair up all of the  $ws$  of which there are  $(2j)!(k - 2j)!$ . Therefore, by Lemma 3,
$$
\left\| w ^ {\otimes k - 2 j} \widetilde {\otimes} (P _ {w} ^ {\perp}) ^ {\widetilde {\otimes} j} \right\| _ {F} ^ {2} = \frac {1}{\binom {k} {2 j}} \left\| (P _ {w} ^ {\perp}) ^ {\widetilde {\otimes} j} \right\| _ {F} ^ {2} = \frac {1}{\nu_ {j} ^ {(d - 1)} \binom {k} {2 j}}.
$$
Plugging this in gives:
$$
\begin{array}{l} \left\| \nabla_ {w} T _ {k} (w) [ u ] \right\| _ {F} ^ {2} \\ \lesssim \frac{k^{2}}{(1 + \lambda^{2})^{k}}\sum_{j = 0}^{\lfloor \frac{k - 1}{2}\rfloor}\binom {k - 1}{2j}\lambda^{4j}\nu_{j}^{(d - 1)} \\ + \frac {\lambda^ {4} k ^ {4}}{d ^ {2} (1 + \lambda^ {2}) ^ {k}} \sum_ {j = 0} ^ {\lfloor \frac {k - 2}{2} \rfloor} \binom {k - 2} {2 j} \lambda^ {4 j} \nu_ {j} ^ {(d + 1)}. \\ \end{array}
$$
$$
\mathcal{L}_{\lambda}(\alpha^{k})\bigg|_{\alpha = \frac{1}{\sqrt{1 + \lambda^{2}}}} = \frac{1}{(1 + \lambda^{2})^{k}}\sum_{k = 0}^{\lfloor \frac{k}{2}\rfloor}\binom {k}{2j}\lambda^{4j}\nu_{j}^{(d - 1)}
$$