SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \left[ \lambda_ {\min } (M) \right] \geq 1 - O (1) \sqrt {\frac {n (\ln n) (3 2 k \ln (d n)) ^ {k}}{N _ {k , d}}} \geq 1 - O (1) \sqrt {\frac {1}{\ln d}} \geq 7 / 8. \tag {H.31}
$$
Now we prove Eq. (H.17). Let  $t = 1 - \lambda_{\min}(M)$  be a random variable. Note that by Eq. (H.18) we have  $\mathbb{E}[t] \leq 1/8$ . By basic algebra, we also have
$$
\lambda_ {\min } (M) \leq M _ {1, 1} = P _ {k, d} (\langle x _ {1}, x _ {1} \rangle) = P _ {k, d} (1) = 1, \tag {H.32}
$$
which implies  $t \geq 0$ . Therefore, by Markov inequality we get
$$
\Pr \left(\lambda_ {\min } (M) \leq 1 / 2\right) = \Pr (t \geq 1 / 2) \leq 2 \mathbb {E} [ t ] \leq 1 / 4. \tag {H.33}
$$
I Toolbox
I.1 Rademacher Complexity and Generalization Bounds
Lemma I.1 (Special Case of Contraction Principle (Lemma 4.6 of Adamczak et al. [4])) Let  $\mathcal{F}$  be a family of functions from  $D$  to  $\mathbb{R}$ , and let  $\varphi$  be an  $L$ -Lipschitz function with  $\varphi(0) = 0$ . Then, for any  $S = \{x_1, \ldots, x_N\} \subset D$ ,
$$
\mathbb {E} _ {\epsilon_ {i}} \sup  _ {f \in \mathcal {F}} \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \varphi (f (x _ {i})) \leq L \cdot \mathbb {E} _ {\epsilon_ {i}} \sup  _ {f \in \mathcal {F}} \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} f (x _ {i}) \tag {I.1}
$$
where the expectations are taken over i.i.d. Rademacher random variables  $\epsilon_1, \ldots, \epsilon_N$ .
In the following, for a function family  $\mathcal{H}$ , we let  $R_{N}(\mathcal{H})$  denote its Rademacher complexity and  $R_{S}(\mathcal{H})$  denote its empirical Rademacher complexity (given a fixed dataset  $S$ ) — see Section 4.4.2 of Ma [53] for a definition of these concepts.
Lemma 1.2 (Empirical Rademacher Complexity of Products). Suppose  $\mathcal{F}$  and  $\mathcal{G}$  are two families of functions from  $D$  to  $\mathbb{R}$ , with  $D\subset \mathbb{R}^d$ , which are uniformly bounded by  $B > 0$ . Then, for any set  $S = \{x_{1},\ldots ,x_{N}\} \subset D$ ,
$$
R _ {S} (\mathcal {F} \cdot \mathcal {G}) \leq B \cdot \left(R _ {S} (\mathcal {F}) + R _ {S} (\mathcal {G})\right), \tag {I.2}
$$
where  $\mathcal{F}\cdot \mathcal{G} = \{fg\mid f\in \mathcal{F},g\in \mathcal{G}\}$  .Asacorollary,
$$
R _ {N} (\mathcal {F} \cdot \mathcal {G}) \leq B \cdot \left(R _ {N} (\mathcal {F}) + R _ {N} (\mathcal {G})\right). \tag {I.3}
$$
Proof of Lemma I.2. We can write
$$
\begin{array}{l} R _ {S} (\mathcal {F} \cdot \mathcal {G}) = \mathbb {E} _ {\epsilon_ {i}} \sup  _ {f \in \mathcal {F}, g \in \mathcal {G}} \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} f (x _ {i}) g (x _ {i}) (I.4) \\ = \mathbb {E} _ {\epsilon_ {i}} \sup  _ {f \in \mathcal {F}, g \in \mathcal {G}} \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \cdot \frac {(f (x _ {i}) + g (x _ {i})) ^ {2} - (f (x _ {i}) - g (x _ {i})) ^ {2}}{4} (I.5) \\ \leq \mathbb {E} _ {\epsilon_ {i}} \sup  _ {f \in \mathcal {F}, g \in \mathcal {G}} \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \cdot \frac {(f \left(x _ {i}\right) + g \left(x _ {i}\right)) ^ {2}}{4} + \mathbb {E} _ {\epsilon_ {i}} \sup  _ {f \in \mathcal {F}, g \in \mathcal {G}} \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \frac {(f \left(x _ {i}\right) - g \left(x _ {i}\right)) ^ {2}}{4}. (I.6) \\ \end{array}
$$
Using Lemma I.1 and the fact that the function  $x \mapsto x^2$  is  $2B$  Lipschitz on the interval  $[-B, B]$ , we have
$$
\begin{array}{l} R _ {S} (\mathcal {F} \cdot \mathcal {G}) \leq 2 B \cdot \underset {\epsilon_ {i}} {\mathbb {E}} \sup  _ {f \in \mathcal {F}, g \in \mathcal {G}} \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \cdot \frac {f (x) + g (x)}{4} + 2 B \cdot \underset {\epsilon_ {i}} {\mathbb {E}} \sup  _ {f \in \mathcal {F}, g \in \mathcal {G}} \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \frac {f (x _ {i}) - g (x _ {i})}{4} (I.7) \\ \leq \frac {B}{2} \cdot \left(R _ {S} (\mathcal {F}) + R _ {S} (\mathcal {G})\right) + \frac {B}{2} \cdot \left(R _ {S} (\mathcal {F}) + R _ {S} (\mathcal {G})\right) (I.8) \\ = B \cdot \left(R _ {S} (\mathcal {F}) + R _ {S} (\mathcal {G})\right), (I.9) \\ \end{array}
$$
Lemma I.3 (Symmetrization — Analogous to Corollary 4.7 of Adamczak et al. [4]). Let  $X_{1},\ldots ,X_{N}$  be independent random variables. Let  $\mathcal{F}$  be a family of functions from  $D\subset \mathbb{R}^d$  to  $\mathbb{R}$  that is uniformly bounded by  $B_{1}\geq 1$ , and let  $\mathcal{G}$  be a family of functions from  $D\subset \mathbb{R}^d$  to  $\mathbb{R}$ . Then, for nonnegative integers  $p_1,p_2,p_3,p_4$  and a fixed  $g\in \mathcal{G}$  that is bounded by  $B_{2}\geq 1$ , we have
$$
\begin{array}{l} \underset {X _ {i}} {\mathbb {E}} \sup  _ {f _ {1}, f _ {2}, f _ {3}, f _ {4} \in \mathcal {F}, g \in \mathcal {G}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \left(| f _ {1} (X _ {i}) | ^ {p _ {1}} | f _ {2} (X _ {i}) | ^ {p _ {2}} | f _ {3} (X _ {i}) | ^ {p _ {3}} | f _ {4} (X _ {i}) | ^ {p _ {4}} | g (X _ {i}) | \right. \right. (I.10) \\ \left. - \underset {X _ {i}} {\mathbb {E}} \left[ | f _ {1} (X _ {i}) | ^ {p _ {1}} | f _ {2} (X _ {i}) | ^ {p _ {2}} | f _ {3} (X _ {i}) | ^ {p _ {3}} | f _ {4} (X _ {i}) | ^ {p _ {4}} | g (X _ {i}) | \right]\right) \Bigg) (I.11) \\ \leq \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B _ {1} ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, B _ {2}\right) \cdot R _ {N} (\mathcal {F}) (I.12) \\ \end{array}
$$
Proof of Lemma I.3. By Theorem 4.13 of Ma [53],
$$
\begin{array}{l} \mathbb {E} _ {X _ {i}} \sup  _ {f _ {1}, f _ {2}, f _ {3}, f _ {4} \in \mathcal {F}, g \in \mathcal {G}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \left(| f _ {1} (X _ {i}) | ^ {p _ {1}} | f _ {2} (X _ {i}) | ^ {p _ {2}} | f _ {3} (X _ {i}) | ^ {p _ {3}} | f _ {4} (X _ {i}) | ^ {p _ {4}} | g (X _ {i}) |\right. \right. (I.13) \\ \left. - \underset {X _ {i}} {\mathbb {E}} \left[ | f _ {1} (X _ {i}) | ^ {p _ {1}} | f _ {2} (X _ {i}) | ^ {p _ {2}} | f _ {3} (X _ {i}) | ^ {p _ {3}} | f _ {4} (X _ {i}) | ^ {p _ {4}} | g (X _ {i}) | \right]\right) \Bigg) \Bigg | (I.14) \\ \lesssim R _ {N} \left(\mathcal {F} ^ {p _ {1}} \cdot \mathcal {F} ^ {p _ {2}} \cdot \mathcal {F} ^ {p _ {3}} \cdot \mathcal {F} ^ {p _ {4}} \cdot g\right) \quad (\text {B y}) \\ \lesssim \max  \left(B _ {1} ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, B _ {2}\right) \cdot \left(R _ {N} \left(\mathcal {F} ^ {p _ {1}} \cdot \mathcal {F} ^ {p _ {2}} \cdot \mathcal {F} ^ {p _ {3}} \cdot \mathcal {F} ^ {p _ {4}}\right) + R _ {N} (\{g \})\right) (By Lemma I.2) \\ \lesssim \max  \left(B _ {1} ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, B _ {2}\right) \cdot R _ {N} \left(\mathcal {F} ^ {p _ {1}} \cdot \mathcal {F} ^ {p _ {2}} \cdot \mathcal {F} ^ {p _ {3}} \cdot \mathcal {F} ^ {p _ {4}}\right) (I.15) \\ \lesssim \max  \left(B _ {1} ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, B _ {2}\right) \cdot \max  \left(B _ {1} ^ {p _ {1}}, B _ {1} ^ {p _ {2} + p _ {3} + p _ {4}}\right) \cdot \max  \left(B _ {1} ^ {p _ {2}}, B _ {1} ^ {p _ {3} + p _ {4}}\right) (I.16) \\ \cdot \max  \left(B _ {1} ^ {p _ {3}}, B _ {1} ^ {p _ {4}}\right) \cdot \left(R _ {N} \left(\mathcal {F} ^ {p _ {1}}\right) + R _ {N} \left(\mathcal {F} ^ {p _ {2}}\right) + R _ {N} \left(\mathcal {F} ^ {p _ {3}}\right) + R _ {N} \left(\mathcal {F} ^ {p _ {4}}\right)\right) \\ \text {(B y r e p a t e d a p p l i c a t i o n s o f L e m m a I . 2)} \\ \lesssim \max  \left(B _ {1} ^ {O \left(p _ {1} + p _ {2} + p _ {3} + p _ {4}\right)}, B _ {2}\right) \cdot \left(R _ {N} \left(\mathcal {F} ^ {p _ {1}}\right) + R _ {N} \left(\mathcal {F} ^ {p _ {2}}\right) + R _ {N} \left(\mathcal {F} ^ {p _ {3}}\right) + R _ {N} \left(\mathcal {F} ^ {p _ {4}}\right)\right) (I.17) \\ \end{array}
$$
Finally, we simplify  $R_{N}(\mathcal{F}^{p_i})$  for each  $i$ . For nonnegative integers  $p$ , since  $\mathcal{F}$  is uniformly bounded by  $B_{1}$ , the function  $x \mapsto x^p$  is  $pB_1^{p - 1}$ -Lipschitz on  $[-B_1,B_1]$  for  $p \geq 1$  and is 1-Lipschitz for  $p = 0$ . Thus, by Lemma I.1, we have  $R_{N}(\mathcal{F}^{p}) \leq \min (1,pB_{1}^{p - 1})R_{N}(\mathcal{F})$ , and
$$
\begin{array}{l} \mathbb {E} _ {X _ {i}} \sup  _ {f _ {1}, f _ {2}, f _ {3}, f _ {4} \in \mathcal {F}, g \in \mathcal {G}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \left(| f _ {1} (X _ {i}) | ^ {p _ {1}} | f _ {2} (X _ {i}) | ^ {p _ {2}} | f _ {3} (X _ {i}) | ^ {p _ {3}} | f _ {4} (X _ {i}) | ^ {p _ {4}} | g (X _ {i}) |\right. \right. (I.18) \\ \left. - \underset {X _ {i}} {\mathbb {E}} \left[ | f _ {1} (X _ {i}) | ^ {p _ {1}} | f _ {2} (X _ {i}) | ^ {p _ {2}} | f _ {3} (X _ {i}) | ^ {p _ {3}} | f _ {4} (X _ {i}) | ^ {p _ {4}} | g (X _ {i}) | \right]\right) \Bigg) \Bigg | (I.19) \\ \lesssim \max  \left(B _ {1} ^ {O \left(p _ {1} + p _ {2} + p _ {3} + p _ {4}\right)}, B _ {2}\right) \cdot \max  \left(p _ {1} B _ {1} ^ {p _ {1}}, p _ {2} B _ {1} ^ {p _ {2}}, p _ {3} B _ {1} ^ {p _ {3}}, p _ {4} B _ {1} ^ {p _ {4}}\right) \cdot R _ {N} (\mathcal {F}) (I.20) \\ \lesssim \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B _ {1} ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, B _ {2}\right) \cdot R _ {N} (\mathcal {F}) (I.21) \\ \end{array}
$$
I.2 Concentration Lemmas
Theorem I.4 (Theorem 5.62 of Vershynin [73]). Let  $A$  be an  $m \times n$  matrix with independent columns  $\{A_j\}_{j=1}^n$  and  $m \geq n$ . Suppose  $A_j$  is isotropic, i.e.,  $\mathbb{E}[A_j A_j^\top] = I$  and  $\| A_j \|_2 = \sqrt{m}$  for every  $j \in [n]$ . Let  $p$  be the incoherence parameter defined by
$$
p \triangleq \frac {1}{m} \mathbb {E} \left[ \max  _ {i \leq n} \sum_ {j \in [ n ], j \neq i} \left\langle A _ {i}, A _ {j} \right\rangle^ {2} \right]. \tag {I.22}
$$
Then there is a universal constant  $C_0$  such that
$$
\mathbb {E} \left[ \left\| \frac {1}{m} A ^ {\top} A - I \right\| \right] \leq C _ {0} \sqrt {\frac {p \ln n}{m}}. \tag {I.23}
$$
The following proposition gives the tail bound for inner products of random vectors in  $\mathbb{S}^{d - 1}$  (c.f. Theorem 3.4.6 of Vershynin [74]).
Proposition I.5. Let  $u$  be a vector drawn uniformly at random from the unit sphere  $\mathbb{S}^{d-1}$ . For any fixed vector  $v \in \mathbb{S}^{d-1}$ , we have
$$
\forall t > 0, \quad \Pr \left(\left| u ^ {\top} v \right| \geq t\right) \leq 2 \exp \left(- \frac {t ^ {2} d}{2}\right). \tag {I.24}
$$
Proof of Proposition I.5. By the symmetry of  $u$  and  $v$ , we can assume that  $v = (1,0,\dots ,0)$  without loss of generality. Let  $x = \frac{1}{2} (u^{\top}v + 1)$ . Then we have  $x\sim \mathrm{Beta}\left(\frac{d - 1}{2},\frac{d - 1}{2}\right)$ . Hence, the desired result follows directly from Skorski [63, Theorem 1].
Lemma I.6 (Modification of Proposition 4.4 of Adamczak et al. [4]). Let  $X_{1},\ldots ,X_{N}$  be i.i.d. random vectors drawn uniformly from  $\sqrt{d}\mathbb{S}^{d - 1}$ , and let  $p_1,p_2,p_3,p_4,q_1,q_2$  be nonnegative integers. Suppose  $p_1 + p_2 + p_3 + p_4\geq 2$ . Then, for fixed  $w_{1},w_{2}\in \mathbb{S}^{d - 1}$ , as long as  $N\leq d^C$  for any universal constant  $C > 0$ , we have
$$
\begin{array}{l} \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} | \langle X _ {i}, w _ {1} \rangle | ^ {q _ {1}} | \langle X _ {i}, w _ {2} \rangle | ^ {q _ {2}} \right. (I.25) \\ - \underset {X \sim \sqrt {d S ^ {d - 1}}} {\mathbb {E}} \left[ \right. | \langle X, u _ {1} \rangle | ^ {p _ {1}} | \langle X, u _ {2} \rangle | ^ {p _ {2}} | \langle X, u _ {3} \rangle | ^ {p _ {3}} | \langle X, u _ {4} \rangle | ^ {p _ {4}} | \langle X, w _ {1} \rangle | ^ {q _ {1}} | \langle X, u _ {2} \rangle | ^ {q _ {2}} | \langle X, w _ {3} \rangle | ^ {q _ {3}} | \langle X, w _ {4} \rangle | ^ {q _ {4}} | \langle X, w _ {5} \rangle | ^ {q _ {5}} | \langle X, w _ {6} \rangle | ^ {q _ {6}} | \langle X, w _ {7} \rangle | ^ {q _ {7}} | \langle X, w _ {8} \rangle | ^ {q _ {8}} | \langle X, w _ {9} \rangle | ^ {q _ {9}} | \langle X, w _ {1 0} \rangle | ^ {q _ {1 0}} | \langle X, w _ {1 1} \rangle | ^ {q _ {1 1}} | \langle X, w _ {1 2} \rangle | ^ {q _ {1 2}} | \langle X, w _ {1 3} \rangle | ^ {q _ {1 3}} | \langle X, w _ {1 4} \rangle | ^ {q _ {1 4}} | \langle X, w _ {1 5} \rangle | ^ {q _ {1 5}} | \langle X, w _ {1 6} \rangle | ^ {q _ {1 6}} | \langle X, w _ {1 7} \rangle | ^ {q _ {1 7}} | \langle X, w _ {1 8} \rangle | ^ {q _ {1 8}} | \langle X, w _ {1 9} \rangle | ^ {q _ {1 9}} | \langle X, w _ {2 0} \rangle | ^ {q _ {2 0}} | \langle X, w _ {2 1} \rangle | ^ {q _ {2 1}} | \langle X, w _ {2 2} \rangle | ^ {q _ {2 2}} | \langle X, w _ {2 3} \rangle | ^ {q _ {2 3}} | \langle X, w _ {2 4} \rangle | ^ {q _ {2 4}} | \langle X, w _ {2 5} \rangle | ^ {q _ {2 5}} | \langle X, w _ {2 6} \rangle | ^ {q _ {2 6}} | \langle X, w _ {2 7} \rangle | ^ {q _ {2 7}} | \langle X, w _ {2 8} \rangle | ^ {q _ {\text {(I)}}}, (I. 2 6) \\ \leq C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}, q _ {1}, q _ {2}} \left(\log d\right) ^ {O \left(p _ {1} + p _ {2} + p _ {3} + p _ {4} + q _ {1} + q _ {2}\right)} \cdot \sqrt {\frac {d}{N}} (I.27) \\ + C _ {p _ {1}, p _ {2}, p _ {3}, p _ {4}} \left(\log d\right) ^ {O \left(q _ {1} + q _ {2}\right)} \cdot \frac {d ^ {\frac {p _ {1} + p _ {2} + p _ {3} + p _ {4}}{2}}}{N} + \frac {C _ {p _ {1} , p _ {2} , p _ {3} , p _ {4} , q _ {1} , q _ {2}}}{d ^ {\Omega (\log d)}} (I.28) \\ \end{array}
$$
with probability at least  $1 - \frac{1}{d^{\Omega(\log d)}}$ . Here,  $C_{p_1,p_2,p_3,p_4,q_1,q_2}$  is a sufficiently large constant which depends on  $p_1,p_2,p_3,p_4,q_1,q_2$  and  $C_{p_1,p_2,p_3,p_4}$  is a constant which depends on  $p_1,p_2,p_3,p_4$ .
Proof of Lemma I.6. We largely follow the proof of Proposition 4.4 of Adamczak et al. [4], with some minor modifications. First, let  $E_{1}$  be the event that  $|\langle X_i, w_1 \rangle| \leq (\log d)^2$  for all  $i = 1, \dots, N$ , and define  $E_{2}$  analogously for  $w_{2}$ . By Lemma I.10, the sub-exponential norm of  $\langle X_i, w_1 \rangle$  is at most an absolute constant  $K > 0$ . Thus, by Proposition 2.7.1 of Vershynin [74], we have
$$
\begin{array}{l} \mathbb {P} \left(\left| \langle X _ {i}, w _ {1} \rangle \right| \geq (\log d) ^ {2}\right) \leq 2 \exp \left(- \Omega \left(\frac {(\log d) ^ {2}}{\| \langle X _ {i} , w _ {1} \rangle \| _ {\psi_ {1}}}\right)\right) (I.29) \\ \lesssim \exp \left(- \Omega (\log d) ^ {2}\right) (I.30) \\ \lesssim \frac {1}{d ^ {\Omega (\log d)}}. (I.31) \\ \end{array}
$$
Thus, by a union bound, with probability at least  $1 - \frac{N}{d^{\Omega(\log d)}}$ , for all  $i = 1,\dots ,N$ , we have  $|\langle X_i,w_1\rangle |\leq (\log d)^2$ , and  $|\langle X_i,w_2\rangle |\leq (\log d)^2$  (by performing a similar union bound using the sub-exponential norm of  $\langle X_i,w_2\rangle$ ). In summary,  $E_{1}$  and  $E_{2}$  simultaneously hold with probability at least  $1 - \frac{1}{d^{\Omega(\log d)}}$  since  $N\leq d^{C}$  for some universal constant  $C > 0$ .
Now, as in Adamczak et al. [4], define  $B > 1$ , which we will specify later — this is the amount by which we will truncate  $\langle X_i, u_1 \rangle$ ,  $\langle X_i, u_2 \rangle$ ,  $\langle X_i, u_3 \rangle$  and  $\langle X_i, u_4 \rangle$ . For convenience, define the function  $\varphi_B: \mathbb{R} \to \mathbb{R}$  given by
$$
\varphi_ {B} (t) = \left\{ \begin{array}{l l} t & t \in [ - B, B ] \\ B & t > B \\ - B & t <   B \end{array} \right. \tag {I.32}
$$
Define  $B' = (\log d)^2$ , and define  $\varphi_{B'}$  similarly to  $\varphi_B$ , but with  $B$  replaced by  $B'$ . If  $E_1$  and  $E_2$  hold, then for all  $u_1, u_2, u_3, u_4 \in \mathbb{S}^{d-1}$ , we have
$$
\begin{array}{l} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} \right| \varphi_ {B ^ {\prime}} (\langle X _ {i}, w _ {1} \rangle) | ^ {q _ {1}} \left| \varphi_ {B ^ {\prime}} (\langle X _ {i}, w _ {2} \rangle) \right| ^ {q _ {2}} (I.33) \\ - \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left| \langle X, u _ {1} \rangle | ^ {p _ {1}} | \langle X, u _ {2} \rangle | ^ {p _ {2}} | \langle X, u _ {3} \rangle | ^ {p _ {3}} | \langle X, u _ {4} \rangle | ^ {p _ {4}} \right| \varphi_ {B ^ {\prime}} (\langle X, w _ {1} \rangle) | ^ {q _ {1}} \mid \varphi_ {B ^ {\prime}} (\langle X, w _ {2} \rangle) | ^ {q _ {2}} \right] \Bigg | (I.34) \\ \end{array}
$$
$$
\begin{array}{l} = \left| \frac {1}{N} \sum_ {i = 1} ^ {N} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} | \langle X _ {i}, w _ {1} \rangle | ^ {q _ {1}} | \langle X _ {i}, w _ {2} \rangle | ^ {q _ {2}} \right. \tag {I.35} \\ - \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left| \langle X, u _ {1} \rangle | ^ {p _ {1}} | \langle X, u _ {2} \rangle | ^ {p _ {2}} | \langle X, u _ {3} \rangle | ^ {p _ {3}} | \langle X, u _ {4} \rangle | ^ {p _ {4}} | \varphi_ {B ^ {\prime}} (\langle X, w _ {1} \rangle) | ^ {q _ {1}} | \varphi_ {B ^ {\prime}} (\langle X, w _ {2} \rangle) | ^ {q _ {2}} \right] \right| (B. c. E _ {1} a n d E _ {2} h o l d) \\ \end{array}
$$
$$
\begin{array}{l} = \left| \frac {1}{N} \sum_ {i = 1} ^ {N} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} | \langle X _ {i}, w _ {1} \rangle | ^ {q _ {1}} | \langle X _ {i}, w _ {2} \rangle | ^ {q _ {2}} \right. (I.36) \\ - \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ | \langle X, u _ {1} \rangle | ^ {p _ {1}} | \langle X, u _ {2} \rangle | ^ {p _ {2}} | \langle X, u _ {3} \rangle | ^ {p _ {3}} | \langle X, u _ {4} \rangle | ^ {p _ {4}} | \langle X, w _ {1} \rangle | ^ {q _ {1}} | \langle X, w _ {2} \rangle | ^ {q _ {2}} \right] \Bigg | (I.37) \\ \pm C _ {p _ {1}, \dots , p _ {4}, q _ {1}, q _ {2}} e ^ {- \Omega \left(B ^ {\prime}\right)}. (By Lemma I.8) \\ \end{array}
$$
Thus, the rest of the proof will focus on obtaining an upper bound on
$$
\begin{array}{l} U := \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} | \langle X _ {i}, u _ {1} \rangle | ^ {p _ {1}} | \langle X _ {i}, u _ {2} \rangle | ^ {p _ {2}} | \langle X _ {i}, u _ {3} \rangle | ^ {p _ {3}} | \langle X _ {i}, u _ {4} \rangle | ^ {p _ {4}} \right| \varphi_ {B ^ {\prime}} (\langle X _ {i}, w _ {1} \rangle) | ^ {q _ {1}} \left| \varphi_ {B ^ {\prime}} (\langle X _ {i}, w _ {2} \rangle) \right| ^ {q _ {2}} (I.38) \\ - \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left| \left\langle X, u _ {1} \right\rangle \right| ^ {p _ {1}} \left| \left\langle X, u _ {2} \right\rangle \right| ^ {p _ {2}} \left| \left\langle X, u _ {3} \right\rangle \right| ^ {p _ {3}} \left| \left\langle X, u _ {4} \right\rangle \right| ^ {p _ {4}} \left| \varphi_ {B ^ {\prime}} \left(\left\langle X, w _ {1} \right\rangle\right) \right| ^ {q _ {1}} \left| \varphi_ {B ^ {\prime}} \left(\left\langle X, w _ {2} \right\rangle\right) \right| ^ {q _ {2}} \right] \Bigg | (I.39) \\ \end{array}
$$
First define
$$
U ^ {\prime} := \sup  _ {u _ {1}, u _ {2}, u _ {3}, u _ {4}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \left| \varphi_ {B} \left(\left\langle X _ {i}, u _ {1} \right\rangle\right) \right| ^ {p _ {1}} \left| \varphi_ {B} \left(\left\langle X _ {i}, u _ {2} \right\rangle\right) \right| ^ {p _ {2}} \left| \varphi_ {B} \left(\left\langle X _ {i}, u _ {3} \right\rangle\right) \right| ^ {p _ {3}} \left| \varphi_ {B} \left(\left\langle X _ {i}, u _ {4} \right\rangle\right) \right| ^ {p _ {4}} \right. \tag {I.40}
$$
$$
\left| \varphi_ {B ^ {\prime}} \left(\left\langle X _ {i}, w _ {1} \right\rangle\right) \right| ^ {q _ {1}} \left| \varphi_ {B ^ {\prime}} \left(\left\langle X _ {i}, w _ {2} \right\rangle\right) \right| ^ {q _ {2}} \tag {I.41}
$$
$$
- \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ | \varphi_ {B} (\langle X, u _ {1} \rangle) | ^ {p _ {1}} | \varphi_ {B} (\langle X, u _ {2} \rangle) | ^ {p _ {2}} | \varphi_ {B} (\langle X, u _ {3} \rangle) | ^ {p _ {3}} | \varphi_ {B} (\langle X, u _ {4} \rangle) | ^ {p _ {4}} \right. \tag {I.42}
$$
$$
\left. \left| \varphi_ {B ^ {\prime}} \left(\langle X, w _ {1} \rangle\right) \right| ^ {q _ {1}} \left| \varphi_ {B ^ {\prime}} \left(\langle X, w _ {2} \rangle\right) \right| ^ {q _ {2}} \right] \Bigg | \tag {I.43}
$$
We will first obtain an upper bound on  $U'$ , then show that  $U'$  and  $U$  are close. We apply Lemma I.3 with  $\mathcal{F} = \{x \mapsto \varphi_B(\langle x, u \rangle) \mid u \in \mathbb{S}^{d-1}\}$ , and with  $f_i$  corresponding to  $\varphi_B(\langle x, u_i \rangle)$ , and finally with  $g(x) = |\varphi_{B'}(\langle x, w_1 \rangle)|^{q_1} |\varphi_{B'}(\langle x, w_2 \rangle)|^{q_2}$ , to find that
$$
\underset {X _ {i}} {\mathbb {E}} U ^ {\prime} \leq \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}}\right) \cdot R _ {N} (\mathcal {F}) \quad \text {(B y L e m m a I . 3)}
$$
$$
\leq \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}}\right) \cdot \underset {X _ {i}, \epsilon_ {i}} {\mathbb {E}} \sup  _ {u \in \mathbb {S} ^ {d - 1}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} \varphi_ {B} \left(\langle X _ {i}, u \rangle\right) \right| \tag {I.44}
$$
$$
\leq \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}}\right) \cdot \underset {X _ {i}, \epsilon_ {i}} {\mathbb {E}} \sup  _ {u \in \mathbb {S} ^ {d - 1}} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon \langle X _ {i}, u \rangle \right|
$$
(By Lemma I.1 since  $\varphi_B$  is 1-Lipschitz)
$$
\leq \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}}\right) \cdot \underset {X _ {i}, \epsilon_ {i}} {\mathbb {E}} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \epsilon_ {i} X _ {i} \right\| _ {2} \tag {I.45}
$$
$$
\leq \operatorname {p o l y} \left(p _ {1}, p _ {2}, p _ {3}, p _ {4}, B ^ {p _ {1} + p _ {2} + p _ {3} + p _ {4}}, \left(B ^ {\prime}\right) ^ {q _ {1} + q _ {2}}\right) \cdot \sqrt {\frac {d}{N}}. \tag {By Lemma I.9}
$$
Now, to show a high-probability upper bound on  $U'$ , we apply Lemma 4.8 of Adamczak et al. [4] (Talagrand's concentration inequality), and we ensure that all of the conditions are satisfied:
- We set  $X_{1},\ldots ,X_{N}$  in Lemma 4.8 of Adamczak et al. [4] to be as defined in the statement of Lemma I.6.
- For convenience, define
$$
\begin{array}{l} g _ {B, B ^ {\prime}, u _ {1}, u _ {2}, u _ {3}, u _ {4}, w _ {1}, w _ {2}} (x) = | \varphi_ {B} (\langle x, u _ {1} \rangle) | ^ {p _ {1}} | \varphi_ {B} (\langle x, u _ {2} \rangle) | ^ {p _ {2}} | \varphi_ {B} (\langle x, u _ {3} \rangle) | ^ {p _ {3}} | \varphi_ {B} (\langle x, u _ {4} \rangle) | ^ {p _ {4}} (I.46) \\ \left| \varphi_ {B ^ {\prime}} \left(\langle x, w _ {1} \rangle\right) \right| ^ {q _ {1}} \left| \varphi_ {B ^ {\prime}} \left(\langle x, w _ {2} \rangle\right) \right| ^ {q _ {2}}. (I.47) \\ \end{array}
$$
Then, we let
$$
\mathcal {F} = \left\{x \mapsto g _ {B, B ^ {\prime}, u _ {1}, u _ {2}, u _ {3}, u _ {4}, w _ {1}, w _ {2}} (x) - \underset {X \sim \sqrt {d} \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ g _ {B, B ^ {\prime}, u _ {1}, u _ {2}, u _ {3}, u _ {4}, w _ {1}, w _ {2}} (X) \right] \mid u _ {1}, u _ {2}, u _ {3}, u _ {4} \in \mathbb {S} ^ {d - 1} \right\} \tag {I.48}
$$
where  $w_{1}$  and  $w_{2}$  are fixed (i.e. as defined in the statement of Lemma I.6).