SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\prod_ {i = 1} ^ {P} | \langle X, u _ {i} \rangle | ^ {p _ {i}} - \prod_ {i = 1} ^ {P} \left| \varphi_ {B} \left(\langle X, u _ {i} \rangle\right) \right| ^ {p _ {i}}, \tag {I.105}
$$
if we expand the first product using Eq. (I.104), then we obtain terms of the form  $\prod_{i=1}^{P} F_i$ , where  $F_i$  is either  $|\varphi_B(\langle X, u_i \rangle)|^{p_i}$  or  $(|\langle X, u_i \rangle|^{p_i} - B^{p_i}) 1(|\langle X, u_i \rangle| \geq B)$ . Observe that the term where  $F_i = |\varphi_B(\langle X, u_i \rangle)|^{p_i}$  for all  $i \in [P]$  cancels with the second term in Eq. (I.105). Thus, in order to obtain an upper bound on
$$
\underset {X} {\mathbb {E}} \left[ \prod_ {i = 1} ^ {P} | \langle X, u _ {i} \rangle | ^ {p _ {i}} - \prod_ {i = 1} ^ {P} \left| \varphi_ {B} \left(\langle X, u _ {i} \rangle\right) \right| ^ {p _ {i}} \right], \tag {I.106}
$$
it suffices to upper bound the terms  $\prod_{i=1}^{P} F_i$  where at least one of the  $F_i$  is  $(|\langle X, u_i \rangle|^{p_i} - B^{p_i}) 1(|\langle X, u_i \rangle| \geq B)$ . Observe however that
$$
\left(| \langle X, u _ {i} \rangle | ^ {p _ {i}} - B ^ {p _ {i}}\right) 1 \left(| \langle X, u _ {i} \rangle | \geq B\right) \leq | \langle X, u _ {i} \rangle | ^ {p _ {i}} 1 \left(| \langle X, u _ {i} \rangle | \geq B\right). \tag {I.107}
$$
Thus, because  $|\varphi_{B}(\langle X,u_{i}\rangle)|\leq |\langle X,u_{i}\rangle|$ , in order to upper bound the terms  $\prod_{i = 1}^{P}F_{i}$  where at least one of the  $F_{i}$  is  $(|\langle X,u_i\rangle |^{p_i} - B^{p_i})1(|\langle X,u_i\rangle \geq B)$ , it suffices to obtain an upper bound on the expected value of
$$
1 \left(\left| \langle X, u _ {i} \rangle \right| \geq B \text {f o r s o m e} i \in [ P ]\right) \prod_ {i = 1} ^ {P} \left| \langle X, u _ {i} \rangle \right| ^ {p _ {i}} \leq \sum_ {i = 1} ^ {P} 1 \left(\left| \langle X, u _ {i} \rangle \right| \geq B\right) \prod_ {j = 1} ^ {P} \left| \langle X, u _ {j} \rangle \right| ^ {p _ {i}}. \tag {I.108}
$$
Without loss of generality, let us bound the expected value of the first term in the sum above:
$$
\mathbb {E} _ {X} \left[ 1 (| \langle X, u _ {1} \rangle | \geq B) \prod_ {i = 1} ^ {P} | \langle X, u _ {j} \rangle | ^ {p _ {i}} \right] \leq \mathbb {E} _ {X} [ 1 (| \langle X, u _ {1} \rangle | \geq B) ] ^ {1 / 2} \mathbb {E} _ {X} \left[ \prod_ {i = 1} ^ {P} | \langle X, u _ {j} \rangle | ^ {2 p _ {i}} \right] ^ {1 / 2}
$$
(By the Cauchy-Schwarz Inequality)
$$
\leq \underset {X} {\mathbb {E}} [ 1 (| \langle X, u _ {1} \rangle | \geq B) ] ^ {1 / 2} \cdot \prod_ {i = 1} ^ {P} \underset {X} {\mathbb {E}} [ | \langle X, u _ {j} \rangle | ^ {2 ^ {i + 1} p _ {i}} ] ^ {\frac {1}{2 ^ {i + 1}}}
$$
(By applying the Cauchy-Schwarz inequality  $P - 1$  times)
$$
\leq C _ {p _ {1}, \dots , p _ {P}} \underset {X} {\mathbb {E}} [ 1 (| \langle X, u _ {1} \rangle | \geq B ] ^ {1 / 2}, \tag {I.109}
$$
where the last inequality is by Lemma I.10 and Proposition 2.7.1 of Vershynin [74]. Finally, since the sub-exponential norm of  $\langle X,u_1\rangle$  is at most an absolute constant by Lemma I.10, we have that
$$
\underset {X} {\mathbb {E}} [ 1 (| \langle X, u _ {1} \rangle | \geq B ] = \mathbb {P} (| \langle X, u _ {1} \rangle | \geq B) \lesssim e ^ {- c B} \tag {I.110}
$$
for some absolute constant  $c > 0$ . Thus, by Eq. (I.109), we obtain
$$
\mathbb {E} _ {X} \left[ 1 (| \langle X, u _ {1} \rangle | \geq B) \prod_ {i = 1} ^ {P} | \langle X, u _ {j} \rangle | ^ {p _ {j}} \right] \lesssim C _ {p _ {1}, \dots , p _ {P}} e ^ {- c B} \tag {I.111}
$$
for some absolute constant  $c > 0$ . This completes the proof.
Lemma I.9. Let  $N, d \in \mathbb{N}$ . Let  $X_1, \ldots, X_N$  i.i.d. random vectors drawn uniformly from  $\sqrt{d}\mathbb{S}^{d-1}$ . Additionally, let  $\epsilon_1, \ldots, \epsilon_N$  be i.i.d. Rademacher random variables. Then,
$$
\underset {X, \epsilon_ {i}} {\mathbb {E}} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} X _ {i} \right\| _ {2} \leq \sqrt {\frac {d}{N}}. \tag {I.112}
$$
Proof of Lemma I.9. We first bound the second moment of the norm:
$$
\begin{array}{l} \underset {X, \epsilon_ {i}} {\mathbb {E}} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} X _ {i} \right\| _ {2} ^ {2} = \frac {1}{N ^ {2}} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {N} \underset {\epsilon_ {i}, X _ {i}} {\mathbb {E}} \epsilon_ {i} \epsilon_ {j} \langle X _ {i}, X _ {j} \rangle (I.113) \\ = \frac {1}{N ^ {2}} \sum_ {i = 1} ^ {N} \mathbb {E} _ {X _ {i}} \| X _ {i} \| _ {2} ^ {2} \quad (\mathrm {B . c .} \mathbb {E} [ \epsilon_ {i} \epsilon_ {j} ] = 0 \text {f o r} i \neq j) \\ = \frac {1}{N ^ {2}} \sum_ {i = 1} ^ {N} d \quad (\text {B . c .} \| X _ {i} \| _ {2} = \sqrt {d} \text {w i t h p r o b a b i l i t y} 1) \\ = \frac {N d}{N ^ {2}} (I.114) \\ = \frac {d}{N}. (I.115) \\ \end{array}
$$
Thus,  $\mathbb{E}_{X,\epsilon_i}\left\| \frac{1}{N}\sum_{i = 1}^N\epsilon_iX_i\right\| _2\leq \left(\mathbb{E}_{X,\epsilon_i}\left\| \frac{1}{N}\sum_{i = 1}^N\epsilon_iX_i\right\| _2^2\right)^{1 / 2}\leq \sqrt{\frac{d}{N}}$  , as desired.
Lemma I.10 (Sub-Exponential Norm of Uniform Distribution on  $\sqrt{d}\mathbb{S}^{d - 1}$ ). Assume  $d\geq C$  for some sufficiently large universal constant  $C$ . Then there exists an absolute constant  $K > 0$  such that, if  $X$  is a random vector drawn uniformly from  $\sqrt{d}\mathbb{S}^{d - 1}$  and  $v\in \mathbb{S}^{d - 1}$  is a fixed vector, then  $\| \langle X,v\rangle \|_{\psi_1}\leq K$ , where  $\| \cdot \|_{\psi_1}$  denotes the sub-exponential norm of a random variable. (See Definition 2.7.5 of Vershynin [74].) As a corollary, for any  $p\geq 1$ , there exists a constant  $C_p > 0$  depending on  $p$  such that for any  $v\in \mathbb{S}^{d - 1}$ ,  $\mathbb{E}_{X\sim \sqrt{d}\mathbb{S}^{d - 1}}[|\langle X,v\rangle |^p ]\leq C_p$ .
Proof of Lemma 1.10. Let  $X$  be drawn uniformly at random from the uniform distribution on  $\sqrt{d}\mathbb{S}^{d - 1}$ , and let  $v\in \mathbb{S}^{d - 1}$ . Observe that the distribution of  $\frac{1}{\sqrt{d}}\langle X,v\rangle$  is  $\mu_d$ . Thus, the density of  $\frac{1}{\sqrt{d}}\langle X,v\rangle$  is  $p(t) = \frac{\Gamma(d / 2)}{\sqrt{\pi}\Gamma((d - 1) / 2)} (1 - t^2)^{\frac{d - 3}{2}}$  by Eqs. (1.16) and (1.18) of Atkinson and Han [12]. By a change of variables, the density of  $\langle X,v\rangle$  is
$$
p (t) = \frac {\Gamma (d / 2)}{\sqrt {\pi} \Gamma ((d - 1) / 2)} \left(1 - \frac {t ^ {2}}{d}\right) ^ {\frac {d - 3}{2}} \cdot \frac {1}{\sqrt {d}}. \tag {I.116}
$$
Thus, for any  $t \geq 0$ ,
$$
\begin{array}{l} \mathbb {P} (| \langle X, v \rangle | \geq t) \lesssim 2 \int_ {t} ^ {\infty} p (s) d s (I.117) \\ \lesssim 2 \int_ {t} ^ {\infty} \frac {\Gamma (d / 2)}{\sqrt {\pi} \Gamma ((d - 1) / 2)} \left(1 - \frac {s ^ {2}}{d}\right) ^ {\frac {d - 3}{2}} \cdot \frac {1}{\sqrt {d}} d s (I.118) \\ \lesssim \int_ {t} ^ {\infty} \left(1 - \frac {s ^ {2}}{d}\right) ^ {\frac {d - 3}{2}} d s \quad (\text {B y S t i r l i n g ' s f o r m u l a}) \\ \lesssim \int_ {t} ^ {\infty} e ^ {- \frac {d - 3}{2} \cdot \frac {s ^ {2}}{d}} d s \quad (\mathrm {B . c .} 1 + x \leq e ^ {x}) \\ \lesssim \int_ {t} ^ {\infty} e ^ {- \frac {s ^ {2}}{3}} d s \quad (\text {B . c .} \frac {d - 3}{2 d} \geq \frac {1}{3} \text {f o r} d \text {s u f f i c i e n t l y l a r g e}) \\ \end{array}
$$
Thus, for some absolute constant  $C > 0$  and all  $t \geq 0$ ,
$$
\mathbb {P} (| \langle X, v \rangle | \geq t) \leq C \int_ {t} ^ {\infty} e ^ {- \frac {s ^ {2}}{3}} d s \leq C \int_ {t} ^ {\infty} e ^ {- \frac {s t}{3}} d s = C \left(- \frac {3}{t} e ^ {- \frac {s t}{3}}\right) \Big | _ {t} ^ {\infty} = \frac {3 C}{t} e ^ {- \frac {t ^ {2}}{3}} \leq e ^ {- \frac {t ^ {2}}{3} + \log \frac {3 C}{t}} \tag {I.119}
$$
Suppose  $t \geq (18C)^{1/3}$ . Then,  $\frac{t^2}{6} \geq \log \frac{3C}{t}$ , meaning that  $-\frac{t^2}{3} + \log \frac{3C}{t} \leq -\frac{t^2}{6}$ , and therefore
$$
\mathbb {P} (| \langle X, v \rangle | \geq t) \leq e ^ {- \frac {t ^ {2}}{6}} \tag {I.120}
$$
Thus, by the definition of sub-Gaussian norm (Definition 2.5.6 and Proposition 2.5.2 of Vershynin [74]), the sub-Gaussian norm of  $\langle X,v\rangle$  is bounded by a constant, i.e. for any  $v\in \mathbb{S}^{d - 1}$ , we have  $\| \langle X,v\rangle \|_{\psi_2}\leq K$  for some absolute constant  $K > 0$  (here  $\| \cdot \|_{\psi_2}$  denotes the sub-Gaussian norm of a random variable). By Propositions 2.5.2 and 2.7.1, and Definition 2.7.5, of Vershynin [74], this implies that  $\| \langle X,v\rangle \|_{\psi_1}\lesssim 1$ , since sub-Gaussian random variables are always sub-exponential (specifically, using item (ii) of Proposition 2.5.2 and item (b) of Proposition 2.7.1 of Vershynin [74]). The last statement of the lemma follows directly from Proposition 2.7.1, item (b) of Vershynin [74].
Lemma I.11 (Modification of Theorem 3.6 of Adamczak et al. [4]). Let  $d \geq C$  for some absolute constant  $C$ , and  $1 \leq N \leq e^{\sqrt{d}}$ . Let  $X_{1}, \ldots, X_{N}$  be i.i.d. random vectors sampled from the uniform distribution on  $\sqrt{d}\mathbb{S}^{d-1}$ . Finally, let  $t \geq 1$ . Then, for some absolute constant  $c > 0$ , with probability at least  $1 - e^{-ct\sqrt{d}}$ , for all  $m \in [N]$ ,
$$
\sup  _ { \begin{array}{c} z \in \mathbb {S} ^ {d - 1} \\ \text {s u p p} z \mid \leq m \end{array} } \left\| \sum_ {i = 1} ^ {N} z _ {i} X _ {i} \right\| _ {2} \lesssim t \left(\sqrt {d} + \sqrt {m} \log \left(\frac {2 N}{m}\right)\right) \tag {I.121}
$$
Here, we use supp  $z$  to denote the set of nonzero coordinates of  $z$ .
Proof of Lemma I.11. The proof is the same as that of Theorem 3.6 of Adamczak et al. [4] — while Theorem 3.6 of Adamczak et al. [4] is shown for the case where  $X_{1},\ldots ,X_{N}$  are sampled from an isotropic, log-concave probability distribution, the proof can be generalized to the case where the  $X_{i}$  are sampled from  $\sqrt{d}\mathbb{S}^{d - 1}$ . To see why, observe that the proof only uses the fact that the distribution of the  $X_{i}$  is log-concave in the following places: (i) when applying Lemma 3.1 (also of Adamczak et al. [4]), of which the conclusion is that  $\max_{i\leq N}\| X_i\| _2\leq C_0K\sqrt{d}$  with probability at least  $1 - e^{-K\sqrt{d}}$ , for some absolute constant  $C_0$ , (ii) when applying Lemmas 3.3 and 3.4 (also of Adamczak et al. [4]) the proof requires that  $\| \langle X_i,y\rangle \|_{\psi_1}\leq \psi$  for some absolute constant  $\psi >0$  and any  $y\in \mathbb{S}^{d - 1}$ . However, property (i) trivially holds when the  $X_{i}$  are on  $\sqrt{d}\mathbb{S}^{d - 1}$  since  $\| X_i\| _2 = \sqrt{d}$ , and property (ii) holds by Lemma I.10.
I.3 Helper Lemmas
Lemma I.12. Let  $f:[0,\infty) \to \mathbb{R}_{\geq 0}$  such that
$$
\frac {d}{d t} f (t) ^ {2} \leq p _ {t} f (t) + q _ {t} f (t) ^ {2} \tag {I.122}
$$
where  $p_t, q_t > 0$  for  $t \in [0, \infty)$ . Then, for  $T > 0$  we have
$$
f (T) ^ {2} \leq \left(f (0) + \frac {\max  _ {t} p _ {t}}{\min  _ {t} q _ {t}}\right) ^ {2} \exp \left(\int_ {0} ^ {T} q _ {t} d t\right). \tag {I.123}
$$
$$
\frac {d f (t)}{d t} = \frac {1}{2 f (t)} \frac {d f (t) ^ {2}}{d t} \leq \frac {p _ {t}}{2} + \frac {q _ {t}}{2} f (t) \leq \frac {\max  _ {t} p _ {t}}{2} + \frac {q _ {t}}{2} f (t). \tag {I.124}
$$
$$
\frac {d}{d t} \left(f (t) + \frac {\operatorname* {m a x} _ {t} p _ {t}}{\operatorname* {m i n} _ {t} q _ {t}}\right) \leq \frac {q _ {t}}{2} \left(f (t) + \frac {\operatorname* {m a x} _ {t} p _ {t}}{\operatorname* {m i n} _ {t} q _ {t}}\right). \tag {I.125}
$$
As a result, by Fact I.13,
$$
f (T) \leq \left(f (0) + \frac {\max  _ {t} p _ {t}}{\min  _ {t} q _ {t}}\right) \exp \left(\int_ {0} ^ {T} \frac {q _ {t}}{2} d t\right). \tag {I.126}
$$
Taking the square of both sides finishes the proof.
Fact I.13 (Gronwall's Inequality [13]). Let  $a < b$ , and let  $f: [a, b] \to \mathbb{R}_{\geq 0}$  and  $g: [a, b] \to \mathbb{R}$  be differentiable functions, such that
$$
f ^ {\prime} (t) \leq g (t) f (t) \tag {I.127}
$$
for all  $t \in [a, b]$ . Then, for all  $t \in [a, b]$ ,  $f(t) \leq f(a) \exp(\int_{a}^{s} g(s) ds)$ . Additionally, suppose  $f, g$  satisfy
$$
f ^ {\prime} (t) \geq g (t) f (t) \tag {I.128}
$$
for all  $t\in [a,b]$ . Then, for all  $t\in [a,b]$ ,  $f(t)\geq f(a)\exp (\int_a^s g(s)ds)$ .
Proposition I.14. For any  $\beta_{2},\beta_{4}$  such that  $0\leq \beta_2^2\leq \beta_4\leq \beta_2\leq 1$  , there exists a one-dimensional distribution  $\mu$  that satisfies
$$
\underset {w \sim \mu} {\mathbb {E}} \left[ w ^ {2} \right] = \beta_ {2}, \tag {I.129}
$$
$$
\underset {w \sim \mu} {\mathbb {E}} \left[ w ^ {4} \right] = \beta_ {4}, \tag {I.130}
$$
$$
\operatorname {s u p p} (\mu) \subseteq [ - 1, 1 ]. \tag {I.131}
$$
Proof. Let  $p = \beta_2^2 / \beta_4$ . By the assumptions we have  $0 \leq p \leq 1$ . Consider the distribution
$$
\mu (x) = \frac {\beta_ {2} ^ {2}}{\beta_ {4}} \delta_ {x = \beta_ {4} ^ {1 / 2} \beta_ {2} ^ {- 1 / 2}} + \left(1 - \frac {\beta_ {2} ^ {2}}{\beta_ {4}}\right) \delta_ {x = 0}, \tag {I.132}
$$
where  $\delta_{x = t}$  denotes the Dirac measure at  $x = t$ . First of all, since  $\beta_2^2\leq \beta_4$ , the distribution  $\mu (x)$  is well-defined. And we can verify that
$$
\underset {w \sim \mu} {\mathbb {E}} \left[ w ^ {2} \right] = \frac {\beta_ {2} ^ {2}}{\beta_ {4}} \left(\beta_ {4} ^ {1 / 2} \beta_ {2} ^ {- 1 / 2}\right) ^ {2} = \beta_ {2}, \tag {I.133}
$$
$$
\underset {w \sim \mu} {\mathbb {E}} \left[ w ^ {4} \right] = \frac {\beta_ {2} ^ {2}}{\beta_ {4}} \left(\beta_ {4} ^ {1 / 2} \beta_ {2} ^ {- 1 / 2}\right) ^ {4} = \beta_ {4}. \tag {I.134}
$$
Note that  $0 \leq \beta_{4} \leq \beta_{2}$  implies  $\beta_{4}^{1/2} \beta_{2}^{-1/2} \in [0, 1]$ . Consequently,
$$
\operatorname {s u p p} (\mu) \subseteq [ - 1, 1 ]. \tag {I.135}
$$
Lemma I.15. Let  $\rho$  be a distribution which is rotationally invariant as in Definition 4.1. Let  $\nabla_u L(\rho)$  be defined as in Eq. (3.4), and let  $\nabla_z L(\rho)$  denote the last  $(d - 1)$  coordinates of  $\nabla_u L(\rho)$ . Then, for any particle  $u = (w, z)$ , we have that  $\nabla_z L(\rho)$  is a scalar multiple of  $z$ . In particular, if  $\operatorname{grad}_u L(\rho)$  is defined as in Eq. (3.4) and  $\operatorname{grad}_z L(\rho)$  denotes the last  $(d - 1)$  coordinates of  $\operatorname{grad}_u L(\rho)$ , then  $\operatorname{grad}_z L(\rho)$  is a scalar multiple of  $z$ .
Proof. From Eq. (3.4), we have
$$
\nabla_ {u} L (\rho) = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho} (x) - y (x)\right) \sigma^ {\prime} \left(u ^ {\top} x\right) x \right]. \tag {I.136}
$$
Since  $\rho$  is rotationally invariant, if  $x,x^{\prime}\in \mathbb{S}^{d - 1}$  such that  $\langle e_1,x\rangle = \langle e_1,x'\rangle$ , then  $f_{\rho}(x) - y(x) = f_{\rho}(x') - y(x')$ . Thus,  $(f_{\rho}(x) - y(x))\sigma '(u^\top x)$  only depends on  $\langle e_1,x\rangle$  and  $\langle u,x\rangle$ .
Let  $P_{e_1,u} \in \mathbb{R}^{d \times d}$  denote the projection matrix into the subspace spanned by  $e_1$  and  $u$ . Then,
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho} (x) - y (x)\right) \sigma^ {\prime} \left(u ^ {\top} x\right) \left(x - P _ {e _ {1}, u} x\right) \right] = 0 \tag {I.137}
$$
because if  $v \in \mathbb{S}^{d - 1}$  is orthogonal to  $e_1$  and  $u$ , then the distribution of  $\langle v, x \rangle$  conditioned on  $\langle e_1, x \rangle$  and  $\langle u, x \rangle$  is symmetric around 0. Thus,
$$
\nabla_ {u} L (\rho) = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho} (x) - y (x)\right) \sigma^ {\prime} \left(u ^ {\top} x\right) P _ {e _ {1}, u} x \right] = P _ {e _ {1}, u} \left(\nabla_ {u} L (\rho)\right) \tag {I.138}
$$
and we have that  $\nabla_u L(\rho)$  is in the subspace spanned by  $e_1$  and  $u$ . In other words, there are scalars  $c_1, c_2 \in \mathbb{R}$  such that
$$
\nabla_ {u} L (\rho) = c _ {1} e _ {1} + c _ {2} u \tag {I.139}
$$
and taking the last  $(d - 1)$  coordinates of both sides of this equation gives  $\nabla_u L(\rho) = c_2 z$ , as desired. The last statement of the lemma holds because  $\mathrm{grad}_u L(\rho) = (I - u u^\top) \nabla_u L(\rho)$ , which is also a linear combination of  $e_1$  and  $u$ .
Lemma I.16. Let  $P$  be a polynomial of degree at most  $k > 0$ , where  $k$  is bounded by a universal constant, and let  $w_{1}, w_{2} \in [-1,1]$ . Suppose the coefficients of  $P$  are bounded by  $B$ . Then,  $|P(w_{1}) - P(w_{2})| \lesssim B|w_{1} - w_{2}|$ .
Proof. Let  $c_{i}$  be the coefficient of the  $i^{th}$  degree term in  $P$ . Then, the lemma follows from expanding  $P(w_{1}) - P(w_{2})$  as
$$
\begin{array}{l} \left| P \left(w _ {1}\right) - P \left(w _ {2}\right) \right| \leq \sum_ {i = 0} ^ {k} \left| c _ {i} \right| \left| w _ {1} ^ {i} - w _ {2} ^ {i} \right| (I.140) \\ \leq \sum_ {i = 0} ^ {k} \left| c _ {i} \right| \left| w _ {1} - w _ {2} \right| \cdot \sum_ {j = 0} ^ {i} \left| w _ {1} \right| ^ {j} \left| w _ {2} \right| ^ {i - j} (I.141) \\ \leq \sum_ {i = 0} ^ {k} (i + 1) \cdot | c _ {i} | | w _ {1} - w _ {2} | \quad (\mathrm {B . c .} | w _ {1} |, | w _ {2} | \leq 1) \\ \leq B (k + 1) \sum_ {i = 0} ^ {k} | w _ {1} - w _ {2} | (I.142) \\ = B (k + 1) ^ {2} \left| w _ {1} - w _ {2} \right| (I.143) \\ \lesssim B \left| w _ {1} - w _ {2} \right| \quad (\mathrm {B . c .} k \text {i s a t m o s t a n a b s o l u t e c o n s t a n t}) \\ \end{array}
$$
Lemma I.17 (Powers of Dot Products on  $\mathbb{S}^{d - 1}$ ). Let  $u_{1}, u_{2}, u_{3}, u_{4}, u_{5} \in \mathbb{S}^{d - 1}$ , and  $p, q, r, s, t \geq 1$ . Then,
$$
\mathop {\mathbb {E}} _ {x \sim \mathbb {S} ^ {d - 1}} [ | \langle u _ {1}, x \rangle | ^ {p} | \langle u _ {2}, x \rangle | ^ {q} | \langle u _ {3}, x \rangle | ^ {r} | \langle u _ {4}, x \rangle | ^ {s} | \langle u _ {5}, x \rangle | ^ {t} ] \leq C _ {p, q, r, s, t} \frac {1}{d ^ {(p + q + r + s + t) / 2}}. \tag {I.144}
$$
Here,  $C_{p,q,r,s,t}$  is a constant depending on  $p,q,r,s,t$ .
Proof of Lemma I.17. By repeated applications of the Cauchy-Schwarz inequality, we have
$$
\begin{array}{l} \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ | \langle u _ {1}, X \rangle | ^ {p} | \langle u _ {2}, X \rangle | ^ {q} | \langle u _ {3}, X \rangle | ^ {r} | \langle u _ {4}, X \rangle | ^ {s} | \langle u _ {5}, X \rangle | ^ {t} \right] (I.145) \\ \leq \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ \langle u _ {1}, X \rangle | ^ {2 p} ] ^ {1 / 2} \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ | \langle u _ {2}, X \rangle | ^ {2 q} | \langle u _ {3}, X \rangle | ^ {2 r} | \langle u _ {4}, X \rangle | ^ {2 s} | \langle u _ {5}, X \rangle | ^ {2 t} ] ^ {1 / 2} (I.146) \\ \leq \underset {X \sim \sqrt {d \mathbb {S} ^ {d - 1}}} {\mathbb {E}} [ \langle u _ {1}, X \rangle | ^ {2 p} ] ^ {1 / 2} \underset {X \sim \sqrt {d \mathbb {S} ^ {d - 1}}} {\mathbb {E}} [ | \langle u _ {2}, X \rangle | ^ {4 q} ] ^ {1 / 4} \underset {X \sim \sqrt {d \mathbb {S} ^ {d - 1}}} {\mathbb {E}} [ | \langle u _ {3}, X \rangle | ^ {4 r} | \langle u _ {4}, X \rangle | ^ {4 s} | \langle u _ {5}, X \rangle | ^ {4 t} ] ^ {1 / 4} (I.147) \\ \leq \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ \langle u _ {1}, X \rangle | ^ {2 p} ] ^ {1 / 2} \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ | \langle u _ {2}, X \rangle | ^ {4 q} ] ^ {1 / 4} \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ | \langle u _ {3}, X \rangle | ^ {8 r} ] ^ {1 / 8} (I.148) \\ \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ | \langle u _ {4}, X \rangle | ^ {8 s} | \langle u _ {5}, X \rangle | ^ {8 t} ] ^ {1 / 8} (I.149) \\ \leq \underset {X \sim \sqrt {d \mathbb {S} ^ {d - 1}}} {\mathbb {E}} [ \langle u _ {1}, X \rangle | ^ {2 p} ] ^ {1 / 2} \underset {X \sim \sqrt {d \mathbb {S} ^ {d - 1}}} {\mathbb {E}} [ | \langle u _ {2}, X \rangle | ^ {4 q} ] ^ {1 / 4} \underset {X \sim \sqrt {d \mathbb {S} ^ {d - 1}}} {\mathbb {E}} [ | \langle u _ {3}, X \rangle | ^ {8 r} ] ^ {1 / 8} (I.150) \\ \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ | \langle u _ {4}, X \rangle | ^ {1 6 s} ] ^ {1 / 1 6} \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ | \langle u _ {5}, X \rangle | ^ {1 6 t} ] ^ {1 / 1 6} (I.151) \\ \end{array}
$$
where each line results from a single application of the Cauchy-Schwarz inequality. Thus, by Lemma I.10, we have
$$
\underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ | \langle u _ {1}, X \rangle | ^ {p} | \langle u _ {2}, X \rangle | ^ {q} | \langle u _ {3}, X \rangle | ^ {r} | \langle u _ {4}, X \rangle | ^ {s} | \langle u _ {5}, X \rangle | ^ {t} ] \leq C _ {p, q, r, s, t}. \tag {I.152}
$$
Dividing both sides by  $d^{(p + q + r + s + t) / 2}$ , we obtain
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ | \langle u _ {1}, x \rangle | ^ {p} | \langle u _ {2}, x \rangle | ^ {q} | \langle u _ {3}, x \rangle | ^ {r} | \langle u _ {4}, x \rangle | ^ {s} | \langle u _ {5}, x \rangle | ^ {t} ] \leq C _ {p, q, r, s, t} \frac {1}{d ^ {(p + q + r + s + t) / 2}} \tag {I.153}
$$