SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {P} \Big (\frac {d (\mu^ {2} + \sigma^ {2})}{b} C _ {2} H \preceq \frac {1}{b n} \sum_ {i \in [ n ]} \| x _ {i} \| _ {2} ^ {2} x _ {i} x _ {i} ^ {\top} \preceq \frac {d (\mu^ {2} + \sigma^ {2})}{b} C _ {3} H \Big) \geqslant 1 - 2 n e ^ {- d / 1 6}.
$$
We now show that the second term is of smaller order. Writing  $x_{i} = \mu \mathbf{1} + \sigma z_{i}$  where  $z_{i}\sim \mathcal{N}(0,I_{d})$ , we have:
$$
\frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \langle x _ {i}, x _ {j} \rangle x _ {i} x _ {j} ^ {\top} = \frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \langle x _ {i}, x _ {j} \rangle x _ {i} x _ {j} ^ {\top}
$$
For  $i \neq j$ ,  $\langle x_i, x_j \rangle = \sum_{k=1}^d x_{ik} x_{jk} = \sum_{k=1}^d a_k$  where  $a_k = x_{ik} x_{jk}$  satisfies  $\mathbb{E}a_k = 0$ ,  $\mathbb{E}a_k^2 = 1$  and  $\mathbb{P}(a_k \geqslant t) \leqslant 2\mathbb{P}(|x_{ik}| \geqslant \sqrt{t}) \leqslant 4e^{-t/2}$ . Hence,  $a_k$  is a centered subexponential random variables. Using concentration of subexponential random variables [58], for  $t \leqslant 1$ ,
$$
\mathbb {P} \left(\frac {1}{d} | \langle x _ {i}, x _ {j} \rangle | \geqslant t\right) \leqslant 2 e ^ {- c d t ^ {2}}.
$$
$$
\mathbb {P} \left(\forall i \neq j, \frac {1}{d} | \langle x _ {i}, x _ {j} \rangle | \leqslant t\right) \geqslant 1 - n (n - 1) e ^ {- c d t ^ {2}}.
$$
Then, taking  $t = d^{-1 / 2}4\ln (n) / c$  we have:
$$
\mathbb {P} \left(\forall i \neq j, \frac {1}{d} | \langle x _ {i}, x _ {j} \rangle | \leqslant \frac {4 \ln (n)}{c \sqrt {d}}\right) \geqslant 1 - \frac {1}{n ^ {2}}.
$$
Going back to our second term,
$$
\begin{array}{l} \frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \langle x _ {i}, x _ {j} \rangle x _ {i} x _ {j} ^ {\top} = \frac {(b - 1)}{b n (n - 1)} \sum_ {i <   j} \langle x _ {i}, x _ {j} \rangle \left(x _ {i} x _ {j} ^ {\top} + x _ {j} x _ {i} ^ {\top}\right) \\ \preceq \frac {(b - 1)}{b n (n - 1)} \sum_ {i <   j} | \langle x _ {i}, x _ {j} \rangle | \left(x _ {i} x _ {i} ^ {\top} + x _ {j} x _ {j} ^ {\top}\right), \\ \end{array}
$$
where we used  $x_{i}x_{j}^{\top} + x_{j}x_{i}^{\top}\preceq x_{i}x_{i}^{\top} + x_{j}x_{j}^{\top}$  . Thus,
$$
\begin{array}{l} \frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \langle x _ {i}, x _ {j} \rangle x _ {i} x _ {j} ^ {\top} \preceq \sup  _ {i \neq j} | \langle x _ {i}, x _ {j} \rangle | \times \frac {(b - 1)}{b n (n - 1)} \sum_ {i <   j} \left(x _ {i} x _ {i} ^ {\top} + x _ {j} x _ {j} ^ {\top}\right) \\ = \sup  _ {i \neq j} | \langle x _ {i}, x _ {j} \rangle | \times \frac {b - 1}{b} \frac {1}{n - 1} \sum_ {i = 1} ^ {n} x _ {i} x _ {i} ^ {\top} \\ = \sup  _ {i \neq j} | \langle x _ {i}, x _ {j} \rangle | \times \frac {b - 1}{b} \frac {n}{n - 1} H. \\ \end{array}
$$
$$
\frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \langle x _ {i}, x _ {j} \rangle x _ {i} x _ {j} ^ {\top} \succeq - \sup _ {i \neq j} | \langle x _ {i}, x _ {j} \rangle | \times \frac {b - 1}{b} \frac {n}{n - 1} H.
$$
Hence, with probability  $1 - 1 / n^2$
$$
- \frac {4 \ln (n)}{c \sqrt {d}} \times \frac {b - 1}{b} \frac {n}{n - 1} H \preceq \frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \langle x _ {i}, x _ {j} \rangle x _ {i} x _ {j} ^ {\top} \preceq \frac {4 \ln (n)}{c \sqrt {d}} \times \frac {b - 1}{b} \frac {n}{n - 1} H.
$$
Wrapping things up, with probability  $1 - 1 / n^{2} - 2ne^{-d / 16}$
$$
\left(- \frac {4 \ln (n)}{c \sqrt {d}} \frac {b - 1}{b} \frac {n}{n - 1} + C _ {2} \frac {d}{b}\right) \times H \preceq \tilde {H} _ {b} \preceq \left(\frac {4 \ln (n)}{c \sqrt {d}} \frac {b - 1}{b} \frac {n}{n - 1} + C _ {3} \frac {d}{b}\right) \times H.
$$
Thus, provided that  $\frac{4\ln(n)}{c\sqrt{d}} \leqslant \frac{C_2d}{2b}$  and  $d \geqslant 48\ln (n)$ , we have with probability  $1 - 3 / n^2$ :
$$
C _ {2} ^ {\prime} \frac {d}{b} \times H \preceq \tilde {H} _ {b} \preceq C _ {3} ^ {\prime} \frac {d}{b} \times H.
$$
$$
\begin{array}{l} \frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \langle x _ {i}, x _ {j} \rangle x _ {i} x _ {j} ^ {\top} = \frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \left(\sigma^ {2} \langle z _ {i}, z _ {j} \rangle + \sigma \mu \langle \mathbf {1}, z _ {i} + z _ {j} \rangle + \mu^ {2} d\right) x _ {i} x _ {j} ^ {\top} \\ = \frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \left(\sigma^ {2} \langle z _ {i}, z _ {j} \rangle + \sigma \mu \langle \mathbf {1}, z _ {i} + z _ {j} \rangle\right) x _ {i} x _ {j} ^ {\top} + \frac {(b - 1)}{b n (n - 1)} \mu^ {2} d \sum_ {i \neq j} x _ {i} x _ {j} ^ {\top} \\ \end{array}
$$
For  $i \neq j$ ,  $\langle z_i, z_j \rangle = \sum_{k=1}^d z_{ik}z_{jk} = \sum_{k=1}^d a_k$  where  $a_k = z_{ik}z_{jk}$  satisfies  $\mathbb{E}a_k = 0$ ,  $\mathbb{E}a_k^2 = 1$  and  $\mathbb{P}(a_k \geqslant t) \leqslant 2\mathbb{P}(|z_{ik}| \geqslant \sqrt{t}) \leqslant 4e^{-t/2}$ . Hence,  $a_k$  is a centered subexponential random variables. Using concentration of subexponential random variables [58], for  $t \leqslant 1$ ,
For  $i\in [n]$ $\langle \mathbf{1},z_i\rangle \sim \mathcal{N}(0,d)$  so that  $\mathbb{P}(|\langle \mathbf{1},z_i\rangle |\geqslant t)\leqslant 2e^{-t^2 /(2d)}$  , and
$$
\mathbb {P} (\forall i \in [ n ], | \langle \mathbf {1}, z _ {i} \rangle | \leqslant t) \geqslant 1 - 2 n e ^ {- t ^ {2} / (2 d)} = 1 - \frac {2}{n ^ {2}},
$$
for  $t = 3\sqrt{d}\ln (n)$ . Hence, with probability  $1 - 3 / n^2$ , for all  $i \neq j$  we have  $|\sigma^2\langle z_i,z_j\rangle +\sigma \mu \langle \mathbf{1},z_i + z_j\rangle |\leqslant (\sigma^2 +\sigma \mu)C\ln (n) / \sqrt{d}$ .
$$
\begin{array}{l} \frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \left(\sigma^ {2} \langle z _ {i}, z _ {j} \rangle + \sigma \mu \langle {\bf 1}, z _ {i} + z _ {j} \rangle\right) x _ {i} x _ {j} ^ {\top} = \frac {(b - 1)}{b n (n - 1)} \sum_ {i <   j} \left(\sigma^ {2} \langle z _ {i}, z _ {j} \rangle + \sigma \mu \langle {\bf 1}, z _ {i} + z _ {j} \rangle\right) (x _ {i} x _ {j} ^ {\top} + x _ {j} x _ {i} ^ {\top}) \\ \preceq \frac {(b - 1)}{b n (n - 1)} \sum_ {i <   j} | \sigma^ {2} \langle z _ {i}, z _ {j} \rangle + \sigma \mu \langle \mathbf {1}, z _ {i} + z _ {j} \rangle) | \left(x _ {i} x _ {i} ^ {\top} + x _ {j} x _ {j} ^ {\top}\right), \\ \end{array}
$$
where we used  $x_{i}x_{j}^{\top} + x_{j}x_{i}^{\top} \preceq x_{i}x_{i}^{\top} + x_{j}x_{j}^{\top}$ . Thus,
$$
\begin{array}{l} \frac{(b - 1)}{bn(n - 1)}\sum_{i\neq j}\bigl(\sigma^{2}\langle z_{i},z_{j}\rangle +\sigma \mu \langle \mathbf{1},z_{i} + z_{j}\rangle \bigr)\\ x_{i}x_{j}^{\top}\preceq \sup_{i\neq j}\bigl|\sigma^{2}\langle z_{i},z_{j}\rangle +\sigma \mu \langle \mathbf{1},z_{i} + z_{j}\rangle \bigr)|\\ \times \frac{(b - 1)}{bn(n - 1)}\sum_{i <   j}\bigl(x_{i}x_{i}^{\top} + x_{j}x_{j}^{\top}\bigr) \\ = \sup  _ {i \neq j} \left| \sigma^ {2} \langle z _ {i}, z _ {j} \rangle + \sigma \mu \langle \mathbf {1}, z _ {i} + z _ {j} \rangle \right| \times \frac {b - 1}{b} \frac {1}{n - 1} \sum_ {i = 1} ^ {n} x _ {i} x _ {i} ^ {\top} \\ = \sup  _ {i \neq j} \left| \sigma^ {2} \langle z _ {i}, z _ {j} \rangle + \sigma \mu \langle \mathbf {1}, z _ {i} + z _ {j} \rangle \right| \times \frac {b - 1}{b} \frac {n}{n - 1} H. \\ \end{array}
$$
$$
\frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \left(\sigma^ {2} \langle z _ {i}, z _ {j} \rangle + \sigma \mu \langle {\bf 1}, z _ {i} + z _ {j} \rangle\right) x _ {i} x _ {j} ^ {\top} \succeq - \sup _ {i \neq j} \left| \sigma^ {2} \langle z _ {i}, z _ {j} \rangle + \sigma \mu \langle {\bf 1}, z _ {i} + z _ {j} \rangle\right) | \times \frac {b - 1}{b} \frac {n}{n - 1} H.
$$
Hence, with probability  $1 - 3 / n^2$
$$
\begin{array}{l} - \frac {(\sigma^ {2} + \sigma \mu) C \ln (n)}{\sqrt {d}} \times \frac {b - 1}{b} \frac {n}{n - 1} H \preceq \frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \big (\sigma^ {2} \langle z _ {i}, z _ {j} \rangle + \sigma \mu \langle {\bf 1}, z _ {i} + z _ {j} \rangle \big) x _ {i} x _ {j} ^ {\top} \\ \succeq \frac {(\sigma^ {2} + \sigma \mu) C \ln (n)}{\sqrt {d}} \times \frac {b - 1}{b} \frac {n}{n - 1} H. \\ \end{array}
$$
We thus have shown that this term (the one in the middle of the above inequality) is of smaller order.
We are hence left with  $\frac{(b - 1)}{bn(n - 1)}\mu^2 d\sum_{i\neq j}x_ix_j^\top$ . Denoting  $\bar{x} = \frac{1}{n}\sum_{i}x_{i}$ , we have  $\frac{1}{n^2}\sum_{i\neq j}x_ix_j^\top = \frac{1}{n^2}\sum_{i,j}x_ix_j^\top -\frac{1}{n^2}\sum_i x_ix_i^\top$ , so that:
$$
\frac {(b - 1)}{b n (n - 1)} \mu^ {2} d \sum_ {i \neq j} x _ {i} x _ {j} ^ {\top} = \frac {(b - 1) n}{b (n - 1)} \mu^ {2} d \left(\bar {x} \bar {x} ^ {\top} - \frac {1}{n} H\right).
$$
We note that we have  $H = \frac{1}{n}\sum_{i}x_{i}x_{i}^{\top} = \frac{1}{n^{2}}\sum_{i < j}x_{i}x_{i}^{\top} + x_{j}x_{j}^{\top} \succeq \frac{1}{n^{2}}\sum_{i < j}x_{i}x_{j}^{\top} + x_{j}x_{i}^{\top} = \bar{x}\bar{x}^{\top}$  using  $x_{i}x_{i}^{\top} + x_{j}x_{j}^{\top} \succeq x_{i}x_{j}^{\top} + x_{j}x_{i}^{\top}$ . Thus,  $H \succeq \bar{x}\bar{x}^{\top} \succeq 0$ , and:
$$
- \frac {(b - 1) n}{b (n - 1)} \mu^ {2} d \frac {1}{n} H \preceq \frac {(b - 1)}{b n (n - 1)} \mu^ {2} d \sum_ {i \neq j} x _ {i} x _ {j} ^ {\top} \preceq \frac {(b - 1) n}{b (n - 1)} \mu^ {2} d (1 - 1 / n) H.
$$
We are now able to wrap everything together. With probability  $1 - 2ne^{-d / 16} - 3 / n^2$ , we have, for some numerical constants  $c_{1},c_{2},c_{3},C > 0$ :