SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma H.3 (Weyl's inequality). Given  $A, B \in \mathbb{R}^{m \times n}$ , let  $\sigma_{i}$  and  $\widehat{\sigma}_{i}$  be the  $i$ -th singular values of  $A$  and  $B$ , respectively, in decreasing order and repeated by multiplicities. Then for all  $i \in [m \wedge n]$ ,
$$
\left| \sigma_ {i} - \widehat {\sigma} _ {i} \right| \leq \left\| \boldsymbol {A} - \boldsymbol {B} \right\| _ {2}.
$$
Lemma H.4. Let  $X = [X_{1},\ldots ,X_{n}]$  where each  $X_{i}$ ,  $i\in [n]$  is a sub-gaussian (not necessarily independent) random variable with variance proxy  $\sigma^2$  and  $\mathbb{E}\left[X_i^2\right] = \gamma$ . Then with probability at least  $1 - c\exp \left(-\frac{t^2}{16n\sigma^2}\right)$ ,
$$
\left\| X \right\| _ {2} \leq t \tag {139}
$$
Proof. From Lemma H.1, we can see that  $\| X\| _2^2 -n\gamma = \sum_{i = 1}^n x_i^2 -n\gamma$  is sub-exponential with parameter  $16n\sigma^{2}$ . Let  $d\coloneqq \sum_{i = 1}^{n}x_{i}^{2} - n\gamma$ , then
$$
\mathbb {P} \left(d > t ^ {\prime}\right) \leq \frac {\mathbb {E} [ \exp (s d) ]}{\exp \left(s t ^ {\prime}\right)} \leq \exp \left(\frac {\nu^ {2} s ^ {2}}{2} - s t ^ {\prime}\right) \quad \forall | s | \leq \frac {1}{\nu}. \tag {140}
$$
Choosing  $t' = t^2 - n\gamma$ ,  $\nu = 16n\sigma^2$ , and  $s = \frac{1}{16n\sigma^2}$ , we get,
$$
\begin{array}{l} \mathbb {P} \left(d > t ^ {2} - n \gamma\right) = \mathbb {P} \left(\| X \| _ {2} ^ {2} > t ^ {2}\right) \leq \exp \left(\frac {1}{2}\right) \exp \left(- \frac {t ^ {2} - n \gamma}{1 6 n \sigma^ {2}}\right) (141) \\ \leq \exp \left(\frac {1}{2}\right) \exp \left(\frac {\gamma}{1 6 \sigma^ {2}}\right) \exp \left(- \frac {t ^ {2}}{1 6 n \sigma^ {2}}\right) (142) \\ \leq c \exp \left(- \frac {t ^ {2}}{1 6 n \sigma^ {2}}\right) (143) \\ \end{array}
$$
where the last inequality follows from the fact the second moment of  $x_{1}$  is bounded by its variance proxy up to a multiplicative constant.
Lemma H.5. Let  $X_{1},\ldots ,X_{n}$  be i.i.d. sub-exponential random variables with parameter  $\nu$ . Let  $S = \sum_{i = 1}^{n}X_{i}$ . Then
$$
\mathbb {P} (| S | \leq n t) \geq 1 - c \exp \left(- \frac {t \sqrt {n}}{\nu}\right). \tag {144}
$$
Proof. Recall that the sum of  $n$  i.i.d. sub-exponential random variables is also a  $x$  sub-exponential random variable with parameter  $\nu \sqrt{n}$  (see Lemma H.1). Thus, using the definition of sub-exponential random variable, we have
$$
\mathbb {P} (S > n t) \leq \frac {\mathbb {E} [ \exp (\lambda S) ]}{\exp (\lambda n t)} \leq \exp \left(\frac {n \nu^ {2} \lambda^ {2}}{2} - \lambda n t\right) \quad \forall | s | \leq \frac {1}{\sqrt {n} \nu}. \tag {145}
$$
Choosing  $\lambda = \frac{1}{\sqrt{n}\nu}$ , we get,
$$
\begin{array}{l} \mathbb {P} (S > n t) \leq \exp \left(\frac {1}{2}\right) \exp \left(- \frac {n t}{\sqrt {n} \nu}\right) (146) \\ \leq 2 \exp \left(\frac {- t \sqrt {n}}{\nu}\right). (147) \\ \end{array}
$$
Applying the same inequality to  $-S$  completes the proof.
Lemma H.6. [[33]] Let  $\mathbf{A} \in \mathbb{R}^{m \times n}$  and  $\mathbf{B} = \mathbf{A} + \mathbf{E}$ . Then
$$
\left\| \boldsymbol {B} ^ {\dagger} - \boldsymbol {A} ^ {\dagger} \right\| _ {2} \leqslant 2 \max  \left\{\left\| \boldsymbol {A} ^ {\dagger} \right\| _ {2} ^ {2}, \left\| \boldsymbol {B} ^ {\dagger} \right\| _ {2} ^ {2} \right\} \| E \| _ {2},
$$
where  $M^{\dagger} \coloneqq (M^{\top}M)^{-1}M^{\top}$  denote the Moore-Penrose inverse of  $M$ .
Lemma H.7. [Lemma 4.2 of [27]] Let  $\{\mathcal{F}_t\}_{t\geq 0}$  be a filtration, and  $\{Z_t\}_{t\geq 1}$  and  $\{W_t\}_{t\geq 1}$  be real-valued processes adapted to  $\mathcal{F}_t$  and  $\mathcal{F}_{t + 1}$  respectively. Moreover, assume  $\overline{W}_t|\mathcal{F}_t$  is mean zero and  $\sigma^2$ -sub-gaussian. Then, for any positive real numbers  $\alpha, \beta$  we have
$$
\mathbb {P} \left(\left\{\sum_ {t = 1} ^ {T} Z _ {t} W _ {t} \geq \alpha \right\} \cap \left\{\sum_ {t = 1} ^ {T} Z _ {t} ^ {2} \leq \beta \right\}\right) \leq \exp \left(- \frac {\alpha^ {2}}{2 \sigma^ {2} \beta}\right). \tag {148}
$$