SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- We let  $a = 2B^{p_1 + p_2 + p_3 + p_4}(B')^{q_1 + q_2}$ , which is a uniform bound on all the functions in  $\mathcal{F}$ .
- Observe that for all  $f \in  \mathcal{F},{\mathbb{E}}_{X \sim  \sqrt{d}{\mathbb{S}}^{d - 1}}f\left( X\right)  = 0$  .
- We let  $Z = \sup_{f\in \mathcal{F}}\sum_{i = 1}^{N}f(X_{i})$  and  $\sigma^2 = \sup_{f\in \mathcal{F}}\sum_{i = 1}^{N}\mathbb{E}f(X_i)^2$ . Observe that  $\sigma^2\leq a^2 N$ .
Thus, we apply Lemma 4.8 of Adamczak et al. [4] with
$$
t = a \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}}\right) \cdot \sqrt {\frac {d}{N}} \cdot (\log d) ^ {2} \geq a (\log d) ^ {2} \underset {X _ {i}} {\mathbb {E}} U ^ {\prime} \tag {I.49}
$$
to find that
$$
U ^ {\prime} \leq \underset {X _ {i}} {\mathbb {E}} U ^ {\prime} + a \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}}\right) \cdot \sqrt {\frac {d}{N}} \tag {I.50}
$$
with failure probability at most
$$
\exp \left(- \frac {t ^ {2} N ^ {2}}{2 \left(\sigma^ {2} + 2 a N \mathbb {E} U ^ {\prime}\right) + 3 a t N}\right) \leq \exp \left(- \frac {t ^ {2} N ^ {2}}{1 2 \max  \left(\sigma^ {2} + 2 a \cdot N \mathbb {E} U ^ {\prime} , a t N\right)}\right) \tag {I.51}
$$
$$
= \exp \left(- \frac {1}{1 2} \min  \left(\frac {t ^ {2} N ^ {2}}{\sigma^ {2} + 2 a N \mathbb {E} U ^ {\prime}}, \frac {t N}{a}\right)\right) \tag {I.52}
$$
$$
\leq \exp \left(- \frac {1}{1 2} \min  \left(\frac {t ^ {2} N ^ {2}}{a ^ {2} N + 2 a N \mathbb {E} U ^ {\prime}}, \frac {t N}{a}\right)\right) \tag {I.53}
$$
$$
\leq \exp \left(- \frac {1}{1 2} \min  \left(\frac {t ^ {2} N ^ {2}}{a ^ {2} N + 2 a N \mathbb {E} U ^ {\prime}}, \sqrt {d N}\right)\right) \tag {I.54}
$$
$$
\leq \exp \left(- \frac {1}{1 2} \min  \left(\frac {t ^ {2} N}{a ^ {2} + 2 a   \mathbb {E}   U ^ {\prime}}, \sqrt {d N}\right)\right) \tag {I.55}
$$
$$
\leq \exp \left(- \frac {1}{2 4} \min  \left(\frac {t ^ {2} N}{a ^ {2}}, \frac {t ^ {2} N}{2 a \mathbb {E} U ^ {\prime}}, \sqrt {d N}\right)\right) \tag {I.56}
$$
$$
\leq \exp \left(- \frac {1}{2 4} \min  \left(d, \frac {t ^ {2} N}{2 a \mathbb {E} U ^ {\prime}}, \sqrt {d N}\right)\right) \tag {I.57}
$$
$$
\leq \exp \left(- \frac {1}{2 4} \min  \left(d, t N, \sqrt {d N}\right)\right) \tag {I.58}
$$
$$
\leq \exp \left(- \frac {1}{2 4} \min  \left(d, \sqrt {d N}\right)\right) \tag {I.59}
$$
$$
\leq \exp (- \Omega (\sqrt {d})). \tag {I.60}
$$
Observe that Talagrand's concentration inequality only gives us a bound on  $\sup_{f\in \mathcal{F}}\sum_{i = 1}^{N}f(X_i)$ , while we wish to obtain a high-probability bound on  $\sup_{f\in \mathcal{F}}|\sum_{i = 1}^{N}f(X_i)|$ . However, using the same argument, we can obtain a similar high-probability bound on  $\sup_{f\in \mathcal{F}}\sum_{i = 1}^{N} - f(X_i) = -\inf_{f\in \mathcal{F}}\sum_{i = 1}^{N}f(X_i)$ . Thus, we obtain a lower bound on  $\inf_{f\in \mathcal{F}}\sum_{i = 1}^{N}f(X_i)$  which is the negative of the upper bound that we obtained for  $\sup_{f\in \mathcal{F}}\sum_{i = 1}^{N}f(X_i)$ , meaning that the upper bound we obtained for  $\sup_{f\in \mathcal{F}}\sum_{i = 1}^{N}f(X_i)$  also holds for  $\sup_{f\in \mathcal{F}}|\sum_{i = 1}^{N}f(X_i)|$ . In summary,
$$
U ^ {\prime} \leq \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}}\right) \cdot (\log d) ^ {2} \cdot \sqrt {\frac {d}{N}} \tag {I.61}
$$
with probability at least  $1 - \exp (-\Omega (\sqrt{d}))$
Now that we have obtained a high-probability bound on  $U'$ , let us analyze the difference between  $U'$  and  $U$ . Recall the definition of  $g_{B,B',u_1,u_2,u_3,u_4,w_1,w_2}$  from above. Additionally, for convenience, let  $g_{B',u_1,u_2,u_3,u_4,w_1,w_2}(x) = |\langle x,u_1\rangle|^{p_1}|\langle x,u_2\rangle|^{p_2}|\langle x,u_3\rangle|^{p_3}|\langle x,u_4\rangle|^{p_4}|\varphi_{B'}(\langle x,w_1\rangle)|^{q_1}|\varphi_{B'}(\langle x,w_2\rangle)|^{q_2}$ , i.e. we are simply removing the truncation by  $B$ . Then, following the proof of Proposition 4.4 in Adamczak et al. [4], we have
$$
\begin{array}{l} U \leq U ^ {\prime} (I.62) \\ + \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \frac {1}{N} \sum_ {i = 1} ^ {N} \left| g _ {B ^ {\prime}, u _ {1}, u _ {2}, u _ {3}, u _ {4}, w _ {1}, w _ {2}} \left(X _ {i}\right) - g _ {B, B ^ {\prime}, u _ {1}, u _ {2}, u _ {3}, u _ {4}, w _ {1}, w _ {2}} \left(X _ {i}\right) \right| 1 \left(\bigcup_ {j = 1} ^ {4} \left\{\left| \langle X _ {i}, u _ {j} \rangle \right| \geq B \right\}\right) (I.63) \\ + \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \frac {1}{N} \sum_ {i = 1} ^ {N} \underset {X _ {i}} {\mathbb {E}} | g _ {B ^ {\prime}, u _ {1}, u _ {2}, u _ {3}, u _ {4}, w _ {1}, w _ {2}} (X _ {i}) - g _ {B, B ^ {\prime}, u _ {1}, u _ {2}, u _ {3}, u _ {4}, w _ {1}, w _ {2}} (X _ {i}) | 1 \left(\bigcup_ {j = 1} ^ {4} \{\left| \left\langle X _ {i}, u _ {j} \right\rangle \right| \geq B \}\right), \\ \end{array}
$$
where  $\{C\}$  denotes the event that the condition  $C$  holds. Here the inequality holds because the difference  $g_{B',u_1,u_2,u_3,u_4,w_1,w_2}(X_i) - g_{B,B',u_1,u_2,u_3,u_4,w_1,w_2}(X_i)$  is nonzero if and only if  $1\left(\bigcup_{j=1}^{4}\{|\langle X_i, u_j\rangle| \geq B\}\right)$  is nonzero. We have obtained a high probability bound on  $U'$ . Let us now obtain a bound on the third term on the right-hand side of Eq. (I.62) (here,  $C_{p_1,p_2,p_3,p_4,q_1,q_2}$  is a constant that depends on  $p_1,p_2,p_3,p_4,q_1,q_2$ ):
$$
\underset {X _ {i}} {\mathbb {E}} \left| g _ {B ^ {\prime}, u _ {1}, u _ {2}, u _ {3}, u _ {4}, w _ {1}, w _ {2}} \left(X _ {i}\right) - g _ {B, B ^ {\prime}, u _ {1}, u _ {2}, u _ {3}, u _ {4}, w _ {1}, w _ {2}} \left(X _ {i}\right) \mid 1 \left(\bigcup_ {j = 1} ^ {4} \{\left| \langle X _ {i}, u _ {j} \rangle \right| \geq B \}\right) \right. \tag {I.65}
$$
$$
\leq \underset {X _ {i}} {\mathbb {E}} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} | \langle X _ {i}, w _ {1} \rangle | ^ {q _ {1}} | \langle X _ {i}, w _ {2} \rangle | ^ {q _ {2}} 1 \left(\bigcup_ {j = 1} ^ {4} \{| \langle X _ {i}, u _ {j} \rangle | \geq B \}\right) \tag {I.66}
$$
$$
\leq \underset {X _ {i}} {\mathbb {E}} \left[ | \langle X _ {i}, u _ {1} \rangle | ^ {2 p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {2 p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {2 p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {2 p _ {4}} | \langle X _ {i}, w _ {1} \rangle | ^ {2 q _ {1}} | \langle X _ {i}, w _ {2} \rangle | ^ {2 q _ {2}} \right] ^ {1 / 2} \tag {I.67}
$$
$$
\mathbb {P} \left(\bigcup_ {j = 1} ^ {4} \{\left| \langle X _ {i}, u _ {j} \rangle \right| \geq B \}\right) ^ {1 / 2}
$$
$$
\leq C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}, q _ {1}, q _ {2}} \mathbb {P} \left(\bigcup_ {j = 1} ^ {4} \left\{\left| \langle X _ {i}, u _ {j} \rangle \right| \geq B \right\}\right) ^ {1 / 2}. \tag {I.68}
$$
To obtain the last inequality, we apply the Cauchy-Schwarz inequality multiple times to write
$$
\underset {X _ {i}} {\mathbb {E}} \left[ | \langle X _ {i}, u _ {1} \rangle | ^ {2 p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {2 p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {2 p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {2 p _ {4}} | \langle X _ {i}, w _ {1} \rangle | ^ {2 q _ {1}} | \langle X _ {i}, w _ {2} \rangle | ^ {2 q _ {2}} \right] ^ {1 / 2} \tag {I.69}
$$
as a product of moments of  $|\langle X_i, u_j \rangle|$  and  $|\langle X_i, w_j \rangle|$ , and by Lemma I.10, each of these moments can be bounded above by a constant that depends on  $p_1, p_2, p_3, p_4, q_1, q_2$ . Additionally, by Lemma I.10 and Proposition 2.7.1, part (a) of Vershynin [74], we have  $\mathbb{P}(|\langle X_i, u_j \rangle| \geq B) \lesssim e^{-\Omega(B)}$ , since  $|\langle X_i, u_j \rangle|$  has constant sub-exponential norm. In summary,
$$
\begin{array}{l} \mathbb {E} _ {X _ {i}} \left| g _ {B ^ {\prime}, u _ {1}, u _ {2}, u _ {3}, u _ {4}, w _ {1}, w _ {2}} (X _ {i}) - g _ {B, B ^ {\prime}, u _ {1}, u _ {2}, u _ {3}, u _ {4}, w _ {1}, w _ {2}} (X _ {i}) \right| 1 \left(\bigcup_ {j = 1} ^ {4} \left\{\left| \langle X _ {i}, u _ {j} \rangle \right| \geq B \right\}\right) (I.70) \\ \leq C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}, q _ {1}, q _ {2}} e ^ {- \Omega (B)}. (I.71) \\ \end{array}
$$
To complete the proof, we bound the second term on the right-hand side of Eq. (I.62). Following the proof of Proposition 4.4 of Adamczak et al. [4], we apply Lemma I.11 (a variant of Theorem 3.6 of Adamczak et al. [4]) to find that, with failure probability at most  $e^{-\Omega(\sqrt{d})}$ , for all  $m$ -sparse vectors  $z \in \mathbb{S}^{N-1}$ , we have
$$
\left\| \sum_ {i = 1} ^ {N} z _ {i} X _ {i} \right\| _ {2} \lesssim \sqrt {d} + \sqrt {m} \log \left(\frac {2 N}{m}\right). \tag {I.72}
$$
Let  $E_{\mathrm{sparse}}$  be the event that Eq. (I.72) holds. For convenience, let  $A \in \mathbb{R}^{d \times N}$  be the matrix whose  $i^{th}$  column is  $X_i$ . Additionally, for any subset  $S \subset [N]$ , let  $A_S$  be the matrix whose columns have indices in  $S$ . If  $S \subset [N]$  with  $|S| = m$ , then assuming  $E_{\mathrm{sparse}}$  holds, the operator norm of  $A_S$  is at
most  $\sqrt{d} + \sqrt{m} \log \left( \frac{2N}{m} \right)$  up to a constant factor. Thus, for any set  $S \subset [N]$ , if  $E_{\text{sparse}}$  holds,
$$
\begin{array}{l} \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \left(\sum_ {i \in S} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} | \varphi_ {B ^ {\prime}} (\langle X _ {i}, w _ {1} \rangle) | ^ {q _ {1}} | \varphi_ {B ^ {\prime}} (\langle X _ {i}, w _ {2} \rangle) | ^ {q _ {2}}\right) ^ {\frac {1}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} (I.73) \\ \leq \left(B ^ {\prime}\right) ^ {\frac {q _ {1} + q _ {2}}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \left(\sum_ {i \in S} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}}\right) ^ {\frac {1}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} (I.74) \\ \leq \left(B ^ {\prime}\right) ^ {\frac {q _ {1} + q _ {2}}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \left(\sum_ {i \in S} | \langle X _ {i}, u _ {1} \rangle | ^ {\frac {2 p _ {1}}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} | \langle X _ {i}, u _ {2} \rangle | ^ {\frac {2 p _ {2}}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} \right. (I.75) \\ \end{array}
$$
$$
\left. \left| \left\langle X _ {i}, u _ {3} \right\rangle \right| ^ {\frac {2 p _ {3}}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} \left| \left\langle X _ {i}, u _ {4} \right\rangle \right| ^ {\frac {2 p _ {4}}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}}\right) ^ {\frac {1}{2}}
$$
(B.c.  $\| x\| _2\geq \| x\| _p$  for  $p\geq 2$  and  $x\in \mathbb{R}^d$  ,and  $p_1 + p_2 + p_3 + p_4\geq 2)$
$$
\begin{array}{l} \leq C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}} \left(B ^ {\prime}\right) ^ {\frac {q _ {1} + q _ {2}}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \left(\sum_ {i \in S} | \langle X _ {i}, u _ {1} \rangle | ^ {2} + | \langle X _ {i}, u _ {2} \rangle | ^ {2} \right. \tag {I.76} \\ \left. + | \langle X _ {i}, u _ {3} \rangle | ^ {2} + | \langle X _ {i}, u _ {4} \rangle | ^ {2}\right) ^ {\frac {1}{2}} \\ \end{array}
$$
(By Weighted AM-GM Inequality, with  $C_{p_1,p_2,p_3,p_4}$  a constant depending on  $p_1,p_2,p_3,p_4$ )
$$
\lesssim C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}} \left(B ^ {\prime}\right) ^ {\frac {q _ {1} + q _ {2}}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} \cdot \left(\sqrt {d} + \sqrt {| S |} \log \left(\frac {2 N}{| S |}\right)\right). \tag {I.77}
$$
where the last inequality is because  $E_{\mathrm{sparse}}$  holds.
Following the proof of Proposition 4.4 of Adamczak et al. [4], we now define  $S_B = \{i \in [N] \mid |\langle X_i, u_j \rangle| \geq B$  for some  $j \in [4]\}$ . Note that  $S_B$  depends on  $u_1, u_2, u_3, u_4$ , and we will now show that if we select  $B$  to be a certain value which is not too large, this will still ensure that  $S_B$  is small. Observe that
$$
\left(\sum_ {i \in S _ {B}} | \langle X _ {i}, u _ {1} \rangle | ^ {2} + | \langle X _ {i}, u _ {2} \rangle | ^ {2} + | \langle X _ {i}, u _ {3} \rangle | ^ {2} + | \langle X _ {i}, u _ {4} \rangle | ^ {2}\right) ^ {1 / 2} \geq \left(\sum_ {i \in S _ {B}} B ^ {2}\right) ^ {1 / 2} \geq B \sqrt {| S _ {B} |} \tag {I.78}
$$
$$
\Big (\sum_ {i \in S _ {B}} | \langle X _ {i}, u _ {1} \rangle | ^ {2} + | \langle X _ {i}, u _ {2} \rangle | ^ {2} + | \langle X _ {i}, u _ {3} \rangle | ^ {2} + | \langle X _ {i}, u _ {4} \rangle | ^ {2} \Big) ^ {1 / 2} \lesssim \sqrt {d} + \sqrt {| S _ {B} |} \log \left(\frac {N}{| S _ {B} |}\right).
$$
(B.c.  $E_{\mathrm{sparse}}$  holds)
Combining these we obtain
$$
B \sqrt {| S _ {B} |} \lesssim \sqrt {d} + \sqrt {| S _ {B} |} \log \left(\frac {N}{| S _ {B} |}\right). \tag {I.79}
$$
Therefore, if we select  $B \gtrsim \log\left(\frac{N}{|S_B|}\right)$ , we can eliminate the second term on the right-hand side of Eq. (I.79), finding that  $|S_B| \lesssim \frac{d}{B^2}$ . Combining this with Eq. (I.73), if  $B \gtrsim \log N$  and  $E_{\mathrm{sparse}}$  holds we obtain
$$
\begin{array}{l} \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \left(\sum_ {i \in S _ {B}} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} | \varphi_ {B ^ {\prime}} (\langle X _ {i}, w _ {1} \rangle) | ^ {q _ {1}} | \varphi_ {B ^ {\prime}} (\langle X _ {i}, w _ {2} \rangle) | ^ {q _ {2}}\right) ^ {\frac {1}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} (I.80) \\ \lesssim C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}} \left(B ^ {\prime}\right) ^ {\frac {q _ {1} + q _ {2}}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} \cdot \left(\sqrt {d} + \sqrt {| S _ {B} |} \log N\right) (I.81) \\ \lesssim C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}} \left(B ^ {\prime}\right) ^ {\frac {q _ {1} + q _ {2}}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} \cdot \left(\sqrt {d} + \frac {\sqrt {d}}{B} \log N\right) \quad (\mathrm {B . c .} | S _ {B} | \lesssim \frac {d}{B ^ {2}}) \\ \lesssim C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}} \left(B ^ {\prime}\right) ^ {\frac {q _ {1} + q _ {2}}{p _ {1} + p _ {2} + p _ {3} + p _ {4}}} \sqrt {d}. \quad (\mathrm {B . c .} B \gtrsim \log N) \\ \end{array}
$$
Raising both sides to the  $(p_1 + p_2 + p_3 + p_4)^{\mathrm{th}}$  power, we finally obtain
$$
\begin{array}{l} \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \sum_ {i \in S _ {B}} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} | \varphi_ {B ^ {\prime}} (\langle X _ {i}, w _ {1} \rangle) | ^ {q _ {1}} | \varphi_ {B ^ {\prime}} (\langle X _ {i}, w _ {2} \rangle) (I.82) \\ \lesssim C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}} ^ {\prime} \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}} \cdot d ^ {\frac {p _ {1} + p _ {2} + p _ {3} + p _ {4}}{2}} (I.83) \\ \end{array}
$$
with probability  $1 - e^{-\Omega (\sqrt{d})}$ , where  $C_{p_1,p_2,p_3,p_4}^{\prime}$  is a constant which depends on  $p_1,p_2,p_3,p_4$
In summary, assuming  $E_{\mathrm{sparse}}$  holds and the high-probability bound on  $U'$  holds (and these both hold simultaneously with probability at least  $1 - \exp(-\Omega(\sqrt{d}))$ ), we obtain
$$
\begin{array}{l} U \lesssim \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}}\right) \cdot \left(\log d\right) ^ {2} \cdot \sqrt {\frac {d}{N}} + C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}} ^ {\prime} \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}} \cdot \frac {d ^ {\frac {p _ {1} + p _ {2} + p _ {3} + p _ {4}}{2}}}{N} (I.84) \\ + C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}, q _ {1}, q _ {2}} e ^ {- \Omega (B)} (I.85) \\ \end{array}
$$
by Eq. (I.62). Finally, by Eq. (I.33), we have
$$
\begin{array}{l} \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} | \langle X _ {i}, w _ {1} \rangle | ^ {q _ {1}} | \langle X _ {i}, w _ {2} \rangle | ^ {q _ {2}} \right. (I.86) \\ - \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ | \langle X, u _ {1} \rangle | ^ {p _ {1}} | \langle X, u _ {2} \rangle | ^ {p _ {2}} | \langle X, u _ {3} \rangle | ^ {p _ {3}} | \langle X, u _ {4} \rangle | ^ {p _ {4}} | \langle X, w _ {1} \rangle | ^ {q _ {1}} | \langle X, w _ {2} \rangle | ^ {q _ {2}} \right] \Bigg | (I.87) \\ = U \pm C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}, q _ {1}, q _ {2}} e ^ {- \Omega \left(B ^ {\prime}\right)} (I.88) \\ \end{array}
$$
as long as the events  $E_{1}$  and  $E_{2}$  hold, and these events simultaneously hold with probability at least  $1 - \frac{1}{d^{\Omega(\log d)}}$ . Thus, by our above bound on  $U$  we have
$$
\begin{array}{l} \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} | \langle X _ {i}, w _ {1} \rangle | ^ {q _ {1}} | \langle X _ {i}, w _ {2} \rangle | ^ {q _ {2}} \right. (I.89) \\ - \underset {X \sim \sqrt {d \mathbb {S} ^ {d - 1}}} {\mathbb {E}} \left[ | \langle X, u _ {1} \rangle | ^ {p _ {1}} | \langle X, u _ {2} \rangle | ^ {p _ {2}} | \langle X, u _ {3} \rangle | ^ {p _ {3}} | \langle X, u _ {4} \rangle | ^ {p _ {4}} | \langle X, w _ {1} \rangle | ^ {q _ {1}} | \langle X, w _ {2} \rangle | ^ {q _ {2}} \right] \Bigg | (I.90) \\ \lesssim \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}}\right) \cdot \left(\log d\right) ^ {2} \cdot \sqrt {\frac {d}{N}} (I.91) \\ + C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}} ^ {\prime} \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}} \cdot \frac {d ^ {\frac {p _ {1} + p _ {2} + p _ {3} + p _ {4}}{2}}}{N} + C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}, q _ {1}, q _ {2}} e ^ {- \Omega (B)} (I.92) \\ + C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}, q _ {1}, q _ {2}} e ^ {- \Omega \left(B ^ {\prime}\right)}. (I.93) \\ \end{array}
$$
Setting  $B' = (\log d)^2$  and  $B \gtrsim \max(\log N, (\log d)^2) \asymp (\log d)^2$  (recall that  $N \leq d^C$  for a universal constant  $C$ ), we have
$$
\begin{array}{l} \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} | \langle X _ {i}, w _ {1} \rangle | ^ {q _ {1}} | \langle X _ {i}, w _ {2} \rangle | ^ {q _ {2}} \right. (I.94) \\ - \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ | \langle X, u _ {1} \rangle | ^ {p _ {1}} | \langle X, u _ {2} \rangle | ^ {p _ {2}} | \langle X, u _ {3} \rangle | ^ {p _ {3}} | \langle X, u _ {4} \rangle | ^ {p _ {4}} | \langle X, w _ {1} \rangle | ^ {q _ {1}} | \langle X, w _ {2} \rangle | ^ {q _ {2}} \right] \Bigg | (I.95) \\ \leq C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}, q _ {1}, q _ {2}} \left(\log d\right) ^ {O \left(p _ {1} + p _ {2} + p _ {3} + p _ {4} + q _ {1} + q _ {2}\right)} \cdot \sqrt {\frac {d}{N}} (I.96) \\ + C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}} \left(\log d\right) ^ {O \left(q _ {1} + q _ {2}\right)} \cdot \frac {d ^ {\frac {p _ {1} + p _ {2} + p _ {3} + p _ {4}}{2}}}{N} + \frac {C _ {p _ {1} , p _ {2} , p _ {3} , p _ {4} , q _ {1} , q _ {2}}}{d ^ {\Omega (\log d)}} (I.97) \\ \end{array}
$$
with probability at least  $1 - \frac{1}{d^{\Omega(\log d)}}$ , as desired.
Lemma 1.7 (Corollary for Uniform Samples from  $\mathbb{S}^{d - 1}$ ). Let  $x_{1},\ldots ,x_{N}$  be i.i.d. random vectors sampled uniformly from  $\mathbb{S}^{d - 1}$ , and let  $p_1,p_2,p_3,p_4,q_1,q_2$  be nonnegative integers. Suppose  $p_1 + p_2 + p_3 + p_4\geq 2$ . Then, for fixed  $w_{1},w_{2}\in \mathbb{S}^{d - 1}$ , as long as  $N\leq d^C$  for any universal constant  $C > 0$ , we have
$$
\begin{array}{l} \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} | \langle x _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle x _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle x _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle x _ {i}, u _ {4} \rangle | ^ {p _ {4}} | \langle x _ {i}, w _ {1} \rangle | ^ {q _ {1}} | \langle x _ {i}, w _ {2} \rangle | ^ {q _ {2}} \right. (I.98) \\ - \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ | \langle x, u _ {1} \rangle | ^ {p _ {1}} | \langle x, u _ {2} \rangle | ^ {p _ {2}} | \langle x, u _ {3} \rangle | ^ {p _ {3}} | \langle x, u _ {4} \rangle | ^ {p _ {4}} | \langle x, w _ {1} \rangle | ^ {q _ {1}} | \langle x, w _ {2} \rangle | ^ {q _ {2}} \right] (I.99) \\ \leq \frac {1}{d ^ {\frac {p _ {1} + p _ {2} + p _ {3} + p _ {4} + q _ {1} + q _ {2}}{2}}} \left(C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}, q _ {1}, q _ {2}} (\log d) ^ {O (p _ {1} + p _ {2} + p _ {3} + p _ {4} + q _ {1} + q _ {2})} \cdot \sqrt {\frac {d}{N}} \right.} (I.100) \\ + C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}} \left(\log d\right) ^ {O \left(q _ {1} + q _ {2}\right)} \cdot \frac {d ^ {\frac {p _ {1} + p _ {2} + p _ {3} + p _ {4}}{2}}}{N} + \frac {C _ {p _ {1} , p _ {2} , p _ {3} , p _ {4} , q _ {1} , q _ {2}}}}{d ^ {\Omega (\log d)}} (I.101) \\ \end{array}
$$
with probability at least  $1 - \frac{1}{d^{\Omega(\log d)}}$ . Here,  $C_{p_1,p_2,p_3,p_4,q_1,q_2}$  is a constant which depends on  $p_1,p_2,p_3,p_4,q_1,q_2$ , and  $C_{p_1,p_2,p_3,p_4}$  is a constant which depends on  $p_1,p_2,p_3,p_4$ .
Proof of Lemma I.7. This is a direct corollary of Lemma I.6, obtained by dividing both sides of Lemma I.6 by  $\frac{1}{d(p_1 + p_2 + p_3 + p_4 + q_1 + q_2) / 2}$ .
Lemma I.8 (Moments for Truncated Dot Products with Random Spherical Vectors). Let  $P$  be a positive integer, and for  $i = 1, \dots, P$ , let  $u_i \in \mathbb{S}^{d-1}$  and  $p_i$  be a nonnegative integer. Additionally, let  $B \geq 1$ . Finally, let  $X$  be a random vector drawn uniformly from  $\sqrt{d}\mathbb{S}^{d-1}$ . Then, for some absolute constant  $c > 0$  we have
$$
\left| \mathbb {E} _ {X} \left[ \prod_ {i = 1} ^ {P} | \langle X, u _ {i} \rangle | ^ {p _ {i}} \right] - \mathbb {E} _ {X} \left[ \prod_ {i = 1} ^ {P} | \varphi_ {B} (\langle X, u _ {i} \rangle) | ^ {p _ {i}} \right] \right| \leq C _ {p _ {1}, \dots , p _ {P}} e ^ {- c B} \tag {I.102}
$$
where  $\varphi_B$  is defined as
$$
\varphi_ {B} (t) = \left\{ \begin{array}{l l} t & t \in [ - B, B ] \\ B & t > B \\ - B & t <   B \end{array} \right. \tag {I.103}
$$
and  $C_{p_1,\dots,p_P}$  is a constant which depends only on  $p_1,\ldots ,p_P$
Proof of Lemma I.8. We use an argument similar to one used in part of the proof of Proposition 4.4 in Adamczak et al. [4]. First, for  $i \in [P]$ , we can write
$$
| \langle X, u _ {i} \rangle | ^ {p _ {i}} = \left| \varphi_ {B} \left(\langle X, u _ {i} \rangle\right) \right| ^ {p _ {i}} + \left(| \langle X, u _ {i} \rangle | ^ {p _ {i}} - B ^ {p _ {i}}\right) 1 _ {| \langle X, u _ {i} \rangle \geq B}. \tag {I.104}
$$
Thus, in the expression