SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \forall i, j \in [ d ]: (1 - \delta) ^ {2} (u _ {i} v _ {j}) ^ {2} \leq W _ {1} ^ {2} [ i, j ] \leq (1 + \delta) ^ {2} (u _ {i} v _ {j}) ^ {2} \\ \Rightarrow \quad (1 - \delta) ^ {2} u _ {i} ^ {2} \| \boldsymbol {v} \| _ {2} ^ {2} \leq \| W _ {1} [ i, : ] \| _ {2} ^ {2} \leq (1 + \delta) ^ {2} u _ {i} ^ {2} \| \boldsymbol {v} \| _ {2} ^ {2} \\ \Rightarrow \quad (1 - \delta) ^ {2} \max  _ {i} u _ {i} ^ {2} \| \boldsymbol {v} \| _ {2} ^ {2} \leq \max  _ {i} \| W _ {1} [ i,: ] \| _ {2} ^ {2} \leq (1 + \delta) ^ {2} \max  _ {i} u _ {i} ^ {2} \| \boldsymbol {v} \| _ {2} ^ {2} \\ \end{array}
$$
$$
(1 - \delta) ^ {2} \text {m e d i a n} u _ {i} ^ {2} \| \boldsymbol {v} \| _ {2} ^ {2} \leq \text {m e d i a n} \| W _ {1} [ i,: ] \| _ {2} ^ {2} \leq (1 + \delta) ^ {2} \text {m e d i a n} u _ {i} ^ {2} \| \boldsymbol {v} \| _ {2} ^ {2},
$$
$$
\frac {(1 - \delta) ^ {2}}{(1 + \delta) ^ {2}} \cdot \frac {\max _ {i} u _ {i} ^ {2} \| \boldsymbol {v} \| _ {2} ^ {2}}{\operatorname* {m e d i a n} u _ {i} ^ {2} \| \boldsymbol {v} \| _ {2} ^ {2}} \leq R _ {\text {m e d}, 2} = \frac {\max _ {i} \| W _ {1} [ i , : ] \| _ {2} ^ {2}}{\operatorname* {m e d i a n} \| W _ {1} [ i , : ] \| _ {2} ^ {2}} \leq \frac {(1 + \delta) ^ {2}}{(1 - \delta) ^ {2}} \cdot \frac {\max _ {i} u _ {i} ^ {2} \| \boldsymbol {v} \| _ {2} ^ {2}}{\operatorname* {m e d i a n} u _ {i} ^ {2} \| \boldsymbol {v} \| _ {2} ^ {2}}.
$$
That means
$$
\frac {(1 - \delta) ^ {2}}{(1 + \delta) ^ {2}} \cdot \frac {\max  _ {i} u _ {i} ^ {2}}{\operatorname* {m e d i a n} u _ {i} ^ {2}} \leq R _ {\text {m e d}, 2} \leq \frac {(1 + \delta) ^ {2}}{(1 - \delta) ^ {2}} \cdot \frac {\max  _ {i} u _ {i} ^ {2}}{\operatorname* {m e d i a n} u _ {i} ^ {2}}.
$$
H Auxiliary lemmas
Lemma 32. Let  $A = \frac{1}{m} YX^T$ ,  $\Lambda_{xx} \coloneqq \frac{1}{m} XX^T$ ,  $g_k^{(t)} = \nabla_{W_k}L(W^{(t)})$ ,  $k = 1,2$ . Denote  $\tilde{A}^{(t)}, \tilde{\Lambda}_{xx}^{(t)}$  and  $\tilde{g}_k^{(t)}, k = 1,2$  as the corresponding batch versions at time  $t$ . Let  $M_1^{(t)} = \max_{i,j}\left|W_1^{(t)}[i,j]\right|$  and  $M_2^{(t)} = \max_i\left|w_{2i}^{(t)}\right|$ . Under Assumption 3, we have with probability at least  $1 - \frac{1}{d}$ , for  $\forall t \leq T$  and  $\forall i,j \in [d]$ ,
$$
\begin{array}{l} \left| \tilde {g} _ {1} ^ {(t)} [ i, j ] - g _ {1} ^ {(t)} [ i, j ] \right| \leq d ^ {3} M _ {1} ^ {(t)} \left(M _ {2} ^ {(t)}\right) ^ {2} \sigma \sqrt {d T} + M _ {2} ^ {(t)} \sigma \sqrt {d ^ {2} T}, \\ \left| g _ {2 i} ^ {(t)} - g _ {2 i} ^ {(t)} \right| \leq d ^ {4} \left(M _ {1} ^ {(t)}\right) ^ {2} M _ {2} ^ {(t)} \sigma \sqrt {d T} + d M _ {1} ^ {(t)} \sigma \sqrt {d ^ {2} T}. \\ \end{array}
$$
Proof. By Assumption 3 and Chebyshev's inequality, we have for fixed  $i,j\in [d]$  and  $t\leq T$
$$
\mathbb {P} \left(\left| \tilde {A} _ {i} ^ {(t)} - A _ {i} \right| > \lambda\right) \leq \frac {\sigma^ {2}}{\lambda^ {2}}, \quad \mathbb {P} \left(\left| \tilde {\Lambda} _ {x x} ^ {(t)} [ i, j ] - \Lambda_ {x x} [ i, j ] \right| > \lambda\right) \leq \frac {\sigma^ {2}}{\lambda^ {2}}.
$$
Applying the union bound gives us
$$
\begin{array}{l} \mathbb {P} \left(\exists i \in [ d ], \exists t \leq T: \quad \left| \tilde {A} _ {i} ^ {(t)} - A _ {i} \right| > \lambda\right) \leq \frac {T d \sigma^ {2}}{\lambda^ {2}}, \\ \mathbb {P} \left(\exists i, j \in [ d ], \exists t \leq T: \quad \left| \tilde {\Lambda} _ {x x} ^ {(t)} [ i, j ] - \Lambda_ {x x} [ i, j ] \right| > \lambda\right) \leq \frac {T d ^ {2} \sigma^ {2}}{\lambda^ {2}}, \\ \end{array}
$$
which gives us with probability at least  $1 - \frac{1}{d}$ , for  $\forall t \leq T, \forall i, j \in [d]$
$$
\left| \tilde {A} _ {i} ^ {(t)} - A _ {i} \right| \leq \sigma \sqrt {d ^ {2} T}, \quad \left| \tilde {\Lambda} _ {x x} ^ {(t)} [ i, j ] - \Lambda_ {x x} [ i, j ] \right| \leq \sigma d \sqrt {d T}.
$$
Now we are ready to bound  $\tilde{g}_k^{(t)} - g_k^{(t)}$  for  $k = 1,2$  and  $t\leq T$
Note that for all  $t \leq T$  and  $\forall i \in [d]$ ,
$$
\left| \left(W _ {2} ^ {(t)} W _ {1} ^ {(t)}\right) _ {i} \right| = \left| \sum_ {j = 1} ^ {d} w _ {2 j} ^ {(t)} W _ {1} ^ {(t)} [ j, i ] \right| \leq \sum_ {j = 1} ^ {d} \left| w _ {2 j} ^ {(t)} \right| \left| W _ {1} ^ {(t)} [ j, i ] \right| \leq d M _ {1} ^ {(t)} M _ {2} ^ {(t)}.
$$
Then we have with probability at least  $1 - \frac{1}{d}$ , for all  $t \leq T$  and  $\forall i \in [d]$
$$
\begin{array}{l} \left| \left(W _ {2} ^ {(t)} W _ {1} ^ {(t)} \left(\tilde {\Lambda} _ {x x} ^ {(t)} - \Lambda_ {x x}\right)\right) _ {i} \right| \leq \sum_ {j = 1} ^ {d} \left| \left(W _ {2} ^ {(t)} W _ {1} ^ {(t)}\right) _ {j} \right| \left| \tilde {\Lambda} _ {x x} ^ {(t)} [ j, i ] - \Lambda_ {x x} [ j, i ] \right| \\ \leq d ^ {3} M _ {1} ^ {(t)} M _ {2} ^ {(t)} \sigma \sqrt {d T}. \\ \end{array}
$$
Combining with  $\tilde{g}_1^{(t)} - g_1^{(t)} = W_2^{(t)T}\left(W_2^{(t)}W_1^{(t)}\left(\tilde{\Lambda}_{xx}^{(t)} - \Lambda_{xx}\right) - \left(\tilde{A}^{(t)} - A\right)\right)$ , we get that with probability at least  $1 - \frac{1}{d}$ , for all  $t \leq T$  and  $\forall i, j \in [d]$ ,
$$
\begin{array}{l} \left| \tilde {g} _ {1} ^ {(t)} [ i, j ] - g _ {1} ^ {(t)} [ i, j ] \right| \leq \left| w _ {2 i} ^ {(t)} \right| \left| \left(W _ {2} ^ {(t)} W _ {1} ^ {(t)} \left(\tilde {\Lambda} _ {x x} ^ {(t)} - \Lambda_ {x x}\right)\right) _ {j} \right| + \left| w _ {2 i} ^ {(t)} \right| \left| \tilde {A} _ {j} ^ {(t)} - A _ {j} \right| \\ \leq d ^ {3} M _ {1} ^ {(t)} \left(M _ {2} ^ {(t)}\right) ^ {2} \sigma \sqrt {d T} + M _ {2} ^ {(t)} \sigma \sqrt {d ^ {2} T}. \\ \end{array}
$$
Similarly, note that  $\tilde{g}_{2i}^{(t)} - g_{2i}^{(t)} = \left(W_2^{(t)}W_1^{(t)}\left(\tilde{\Lambda}_{xx}^{(t)} - \Lambda_{xx}\right) - \left(\tilde{A}^{(t)} - A\right)\right)W_1^{(t)T}$ , we then have that with probability at least  $1 - \frac{1}{d}$ , for all  $t \leq T$  and  $\forall i, j \in [d]$ ,
$$
\begin{array}{l} \left| \tilde {g} _ {2 i} ^ {(t)} - g _ {2 i} ^ {(t)} \right| \leq \sum_ {j = 1} ^ {d} \left| \left(W _ {2} ^ {(t)} W _ {1} ^ {(t)} \left(\tilde {\Lambda} _ {x x} ^ {(t)} - \Lambda_ {x x}\right)\right) _ {j} \right| \left| W _ {1} ^ {(t)} [ i, j ] \right| + \sum_ {j = 1} ^ {d} \left| \tilde {A} _ {j} ^ {(t)} - A _ {j} \right| \left| W _ {1} ^ {(t)} [ i, j ] \right| \\ \leq d ^ {4} \left(M _ {1} ^ {(t)}\right) ^ {2} M _ {2} ^ {(t)} \sigma \sqrt {d T} + d M _ {1} ^ {(t)} \sigma \sqrt {d ^ {2} T}. \\ \end{array}
$$
Lemma 33. Consider two sequences  $\{a^{(t)}\}_{t\geq 0},\{b^{(t)}\}_{t\geq 0}$ , which satisfy
$$
a ^ {(t)} = (1 - \beta) \sum_ {\tau = 0} ^ {t} \beta^ {\tau} b ^ {(t - \tau)}, \beta \in (0, 1).
$$
Suppose  $\forall \tau \leq t:|b^{(t)}|\leq B$ , then for any  $\epsilon >0$ , the following truncated version
$$
\tilde {a} ^ {(t)} = (1 - \beta) \sum_ {\tau = 0} ^ {H} \beta^ {\tau} b ^ {(t - \tau)}
$$
with  $H \geq \frac{1}{1 - \beta} \log \frac{B}{\epsilon} = \tilde{\Omega}\left(\frac{1}{1 - \beta}\right)$  satisfies
$$
\left| a ^ {(t)} - \tilde {a} ^ {(t)} \right| \leq \epsilon .
$$
$$
\left| a ^ {(t)} - \tilde {a} ^ {(t)} \right| \leq \left| (1 - \beta) \sum_ {\tau = H + 1} ^ {t} \beta^ {\tau} b ^ {(t - \tau)} \right| \leq (1 - \beta) \sum_ {\tau = H + 1} ^ {t} \beta^ {\tau} B \leq B \beta^ {H + 1}.
$$
To make it less than  $\epsilon$ , it suffices to choose  $H \geq \log \left(\frac{\epsilon}{B}\right) / \log \beta$ .
Since  $\beta \in (0,1)$ , we know that  $\log \beta \leq \beta - 1 < 0$ . We also have  $\log \frac{\epsilon}{B} < 0$ . Then it suffices to choose
$$
H \geq \frac {\log (\epsilon / B)}{\beta - 1} \geq \frac {\log (\epsilon / B)}{\log \beta} \Rightarrow H \geq \frac {1}{1 - \beta} \log \frac {B}{\epsilon} = \tilde {\Omega} \left(\frac {1}{1 - \beta}\right).
$$
Lemma 34. Suppose  $a, b, c, e_a, e_b, e_c \in \mathbb{R}, b > 0, c > 0$  satisfy  $b + e_b + e_c > 0$ ,  $|e_a| \leq \delta |a|$ ,  $|e_b| \leq \delta b$ ,  $|e_c| \leq \delta^2 c^2$  with  $0 < \delta \ll 1$ , then we have
$$
\frac {a + e _ {a}}{\sqrt {b + e _ {b} + e _ {c}} + c} = \frac {a}{\sqrt {b} + c} (1 + R), \quad w h e r e | R | = \mathcal {O} (\delta).
$$
$$
\begin{array}{l} \frac {a + e _ {a}}{\sqrt {b + e _ {b} + e _ {c}} + c} = \frac {a}{\sqrt {b} + c} + \frac {a}{\sqrt {b + e _ {b} + e _ {c}} + c} - \frac {a}{\sqrt {b} + c} + \frac {e _ {a}}{\sqrt {b + e _ {b} + e _ {c}} + c} \\ = \frac {a}{\sqrt {b} + c} \left(1 + \underbrace {\frac {\sqrt {b} + c}{\sqrt {b + e _ {b} + e _ {c}} + c} - 1} _ {q _ {1}} + \underbrace {\frac {e _ {a}}{a} \cdot \frac {\sqrt {b} + c}{\sqrt {b + e _ {b} + e _ {c}} + c}} _ {q _ {2}}\right). \\ \end{array}
$$
Define  $R \coloneqq q_{1} + q_{2}$ . The term  $|q_{1}|$  can be bounded by
$$
\begin{array}{l} | q _ {1} | = \frac {\left| \sqrt {b} - \sqrt {b + e _ {b} + e _ {c}} \right|}{\sqrt {b + e _ {b} + e _ {c}} + c} \\ = \frac {\left| e _ {b} + e _ {c} \right|}{\left(\sqrt {b + e _ {b} + e _ {c}} + c\right) \left(\sqrt {b} + \sqrt {b + e _ {b} + e _ {c}}\right)} \\ \leq \frac {\left| e _ {b} \right|}{(\sqrt {b + e _ {b} + e _ {c}} + c) (\sqrt {b} + \sqrt {b + e _ {b} + e _ {c}})} + \frac {\left| e _ {c} \right|}{(\sqrt {b + e _ {b} + e _ {c}} + c) (\sqrt {b} + \sqrt {b + e _ {b} + e _ {c}})} \\ \leq \frac {\left| e _ {b} \right|}{(\sqrt {b + e _ {b} + e _ {c}} + c) \sqrt {b}} + \frac {\sqrt {\left| e _ {c} \right|}}{c} \cdot \frac {\sqrt {\left| e _ {c} \right|}}{\sqrt {b} + \sqrt {b + e _ {b} + e _ {c}}} \\ \leq \underbrace {\frac {| e _ {b} |}{(\sqrt {b + e _ {b} + e _ {c}} + c) \sqrt {b}}} _ {q _ {3}} + \delta \underbrace {\frac {\sqrt {| e _ {c} |}}{\sqrt {b} + \sqrt {b + e _ {b} + e _ {c}}}} _ {q _ {4}}, \\ \end{array}
$$
where  $|q_3|$  can be bounded by
$$
\left| q _ {3} \right| \stackrel {(i)} {\leq} \frac {\delta b}{(\sqrt {b + e _ {b}} - \sqrt {| e _ {c} |} + c) \sqrt {b}} \leq \frac {\delta \sqrt {b}}{\sqrt {b (1 - \delta)} + c (1 - \delta)} \leq \frac {\delta \sqrt {b}}{\sqrt {b (1 - \delta)}} = \mathcal {O} (\delta).
$$
Here the denominator of (i) uses  $b + e_b \geq b(1 - \delta) > 0$  and  $\sqrt{x + y} \geq \sqrt{x} - \sqrt{|y|}$  when  $x \geq 0, x + y \geq 0$ .
Now let's bound  $|q_4|$ . If  $e_c > 0$ , we have  $e_c = |e_c|$  and  $|q_4| \leq \frac{\sqrt{e_c}}{\sqrt{e_c}} = 1$  since  $b + e_b \geq b(1 - \delta) > 0$ .
If  $e_c \leq 0$ , note that  $b + e_b + e_c > 0$ , we have  $|e_c| < b + b_e \leq b(1 + \delta)$ , which yields  $|q_4| \leq \frac{\sqrt{|e_c|}}{\sqrt{b}} = \mathcal{O}(1)$ . Combining the above bounds give us  $|q_1| \leq |q_3| + \delta |q_4| = \mathcal{O}(\delta)$ .
On the other hand,  $|q_2|$  can be bounded by
$$
\left| q _ {2} \right| \leq \delta \frac {\sqrt {b} + c}{\sqrt {b + e _ {b}} - \sqrt {\left| e _ {c} \right|} + c} \leq \delta \frac {\sqrt {b} + c}{\sqrt {b (1 - \delta)} + c (1 - \delta)} = \mathcal {O} (\delta).
$$
Then  $|R| \leq |q_1| + |q_2| = \mathcal{O}(\delta)$
Lemma 35. Suppose  $X_{1}, X_{2}, \ldots, X_{d}$  are i.i.d Gaussian with mean 0 and variance  $\sigma^2$ , then for  $0 < \delta < \frac{1}{e}$ , we have with probability at least  $1 - \delta$ ,
$$
\max  _ {1 \leq i \leq d} X _ {i} ^ {2} \geq \sigma^ {2} \left(C _ {1} \log d - C _ {2} \log \log \frac {1}{\delta}\right)
$$
for some  $C_1, C_2 > 0$ .
Proof. It suffices to assume that  $\sigma^2 = 1$  and prove that w.p. at least  $1 - \delta$ ,  $\max_{1 \leq i \leq d} X_i^2 \geq C_1 \log d - C_2 \log \log \frac{1}{\delta}$ .
First, by the lower bound of Gaussian tail, there exists  $\alpha, \beta > 0$  such that  $\mathbb{P}(|X_i| > x) = 2\mathbb{P}(X_i > x) \geq \alpha e^{-\beta x^2}$  for  $x \geq 0$ . Then by i.i.d., we have
$$
\begin{array}{l} \mathbb {P} \left(\max  _ {i} | X _ {i} | \leq x\right) = \mathbb {P} \left(\bigcap_ {i = 1} ^ {d} \{| X _ {i} | \leq x \}\right) \\ = \prod_ {i = 1} ^ {d} \mathbb {P} \left(\left| X _ {i} \right| \leq x\right) = \left(1 - \mathbb {P} \left(\left| X _ {i} \right| > x\right)\right) ^ {d} \\ \leq \left(1 - \alpha e ^ {- \beta x ^ {2}}\right) ^ {d} \\ \leq \exp (- d \alpha e ^ {- \beta x ^ {2}}), \\ \end{array}
$$
where the last inequality uses  $1 - x \leq e^{-x}$  for  $x \in [0, 1]$ . Let  $\exp(-d\alpha e^{-\beta x^2}) = \delta$ , we get that w.p. at least  $1 - \delta$ ,
$$
\max  _ {1 \leq i \leq d} | X _ {i} | \geq \sqrt {\frac {1}{\beta} \left(\log (\alpha d) - \log \log \frac {1}{\delta}\right)}.
$$
Then we have w.p. at least  $1 - \delta$
$$
\max  _ {1 \leq i \leq d} X _ {i} ^ {2} = \left(\max  _ {1 \leq i \leq d} | X _ {i} |\right) ^ {2} \geq \frac {1}{\beta} \left(\log (\alpha d) - \log \log \frac {1}{\delta}\right).
$$
Lemma 36. Suppose  $X_{1}, X_{2}, \ldots, X_{d}$  are i.i.d Gaussian with mean 0 and variance  $\sigma^2$ , then we have with constant probability,
$$
\frac {1}{d} \sum_ {i = 1} ^ {d} \frac {1}{| X _ {i} |} \leq \mathcal {O} \left(\frac {1}{\sigma} \log d\right).
$$
Proof. It suffices to assume that  $\sigma^2 = 1$  and prove that with constant probability,  $\frac{1}{d}\sum_{i=1}^{d}\frac{1}{|X_i|} \leq \mathcal{O}(\log d)$ .
Consider  $X_{i}$  for some fixed  $i$ . Since  $X_{i} \sim \mathcal{N}(0,1)$ , we have  $\mathbb{P}(|X_i| \leq t) \leq \frac{2t}{\sqrt{2\pi}}$ . Then we know that with probability at least  $1 - \Theta(d^{-1})$ ,  $|X_{i}| \geq \frac{C}{d}$  for some  $C > 0$ . Then by union bound, with constant probability,  $\forall i \in [d] : |X_{i}| \geq \frac{C}{d}$ .
Now we split the interval  $\left[\frac{C}{d}, 1\right]$  into several subintervals  $\mathcal{I}_k = \{i : |X_i| \in [2^{-k-1}, 2^{-k}]\}$  for  $k = 0, 1, \ldots, \lceil \log_2 \frac{d}{C} \rceil - 1$ . Let  $p_k = \mathbb{P}(|X_i| \in [2^{-k-1}, 2^{-k}])$ , we know that  $|\mathcal{I}_k| \sim \mathrm{Binomial}(d, p_k)$  and  $p_k \leq C_1 \cdot 2^{-k-1}$ . Then by the concentration of binomial variables, we have w.p. at least  $1 - d^{-p}$  for  $p > 0$ ,  $|\mathcal{I}_k| = \mathcal{O}(dp_k + \sqrt{dp_k \log d} + \log d) = \mathcal{O}(d \cdot 2^{-k-1} + \sqrt{d \cdot 2^{-k-1} \log d} + \log d)$ . Then we have
$$
\sum_ {i \in \mathcal {I} _ {k}} \frac {1}{| X _ {i} |} \leq | \mathcal {I} _ {k} | 2 ^ {k + 1} = \mathcal {O} \left(d + \sqrt {d \cdot 2 ^ {k + 1} \log d} + 2 ^ {k + 1} \log d\right), \quad k = 0, 1, \dots , \lceil \log_ {2} \frac {d}{C} \rceil - 1.
$$
Therefore, with constant probability,
$$
\begin{array}{l} \sum_ {i = 1} ^ {d} \frac {1}{| X _ {i} |} = \sum_ {k = 0} ^ {\lceil \log_ {2} \frac {d}{C} \rceil - 1} \sum_ {i \in \mathcal {I} _ {k}} \frac {1}{| X _ {i} |} + \sum_ {| X _ {i} | > 1} \frac {1}{| X _ {i} |} \\ \leq \sum_ {k = 0} ^ {\lceil \log_ {2} \frac {d}{C} \rceil - 1} \mathcal {O} \left(d + \sqrt {d \cdot 2 ^ {k + 1} \log d} + 2 ^ {k + 1} \log d\right) + d \\ = \mathcal {O} \left(d \log_ {2} \frac {d}{C}\right) + \mathcal {O} \left(\sqrt {d \log d} \cdot (\sqrt {2}) ^ {\lceil \log_ {2} \frac {d}{C} \rceil + C _ {2}}\right) + 2 ^ {\lceil \log_ {2} \frac {d}{C} \rceil + 1} \log d + d \\ = \mathcal {O} (d \log d), \\ \end{array}
$$
which means with constant probability,  $\frac{1}{d}\sum_{i = 1}^{d}\frac{1}{|X_i|} = \mathcal{O}(\log d)$