SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} x _ {t + 1} - x _ {t} \geq C \cdot e ^ {- x _ {t}} \\ \Rightarrow \quad e ^ {x _ {t}} \left(x _ {t + 1} - x _ {t}\right) \geq C \quad (\text {m u l t i p l y i n g b o t h s i d e s b y} e ^ {x _ {t}}). \\ \end{array}
$$
$$
\sum_ {t = 0} ^ {T - 1} e ^ {x _ {t}} \left(x _ {t + 1} - x _ {t}\right) \geq C \cdot T.
$$
$$
\int_ {x _ {0}} ^ {x _ {T}} e ^ {x} d x \geq C \cdot T.
$$
$$
e ^ {x _ {T}} - e ^ {x _ {0}} \geq C \cdot T.
$$
$$
e ^ {x _ {T}} \geq e ^ {x _ {0}} + C \cdot T.
$$
$$
x _ {T} \geq \log \left(e ^ {x _ {0}} + C \cdot T\right).
$$
Therefore, we have shown that  $x_{T} \geq \log (e^{x_{0}} + C \cdot T)$ , as required.
Lemma H.3 (Theorem 4.4.5 in Vershynin (2018)). Let  $\mathbf{A}$  be an  $m\times n$  random matrix whose entries  $a_{ij}$  are independent, mean zero, sub-gaussian random variables. Then for any  $t > 0$  we have
$$
\| \mathbf {A} \| _ {2} \leq C K (\sqrt {m} + \sqrt {n} + t)
$$
with probability at least  $1 - 2\exp (-t^2)$ . Here  $K = \max_{i,j}\| a_{ij}\|_{\phi_2}$  where  $\| \cdot \|_{\phi_2}$  is the sub-gaussian norm.
Lemma H.4. For  $t \geq s > 0$ , we have
$$
\frac {\log (1 + a t)}{\log (1 + b t)} \geq \frac {\log (1 + a s)}{\log (1 + b s)},
$$
if  $b > a > 0$
Proof of Lemma H.4. Let  $f(t) = \log (1 + at) / \log (1 + bt)$ , and we want to prove that  $f^{\prime}(t) > 0$  for all  $t > 0$ . To find the derivative of  $f(t)$ , we use the quotient rule:
$$
\begin{array}{l} f ^ {\prime} (t) = \frac {(\log (1 + b t)) \frac {d}{d t} (\log (1 + a t)) - (\log (1 + a t)) \frac {d}{d t} (\log (1 + b t))}{(\log (1 + b t)) ^ {2}} \\ = \frac {(\log (1 + b t)) \frac {a}{1 + a t} - (\log (1 + a t)) \frac {b}{1 + b t}}{(\log (1 + b t)) ^ {2}} \\ = \frac {a (1 + b t) \log (1 + b t) - b (1 + a t) \log (1 + a t)}{(1 + a t) (1 + b t) (\log (1 + b t)) ^ {2}}. \\ \end{array}
$$
Next, we define the function  $g(t) = \left(\frac{1}{b} + t\right)\log(1 + bt) - \left(\frac{1}{a} + t\right)\log(1 + at)$ , and we aim to show that  $g'(t) > 0$  for all  $t > 0$ . We start by computing the derivative of  $g(t)$ :
$$
g ^ {\prime} (t) = \log (1 + b t) - \log (1 + a t).
$$
Since  $b > a$  and  $t > 0$ , we have  $1 + bt > 1 + at$ , which implies that  $\log(1 + bt) > \log(1 + at)$ . Therefore, we have  $g'(t) > 0$  for all  $t > 0$ . Note that  $g(0) = 0$ , we then have  $g(t) > 0$  for all  $t > 0$ . Therefore, we have  $a(1 + bt)\log(1 + bt) - b(1 + at)\log(1 + at) > 0$  for all  $t > 0$ , which in turn implies that  $f'(t) > 0$  for all  $t > 0$ . Thus, we have shown that  $f(t)$  is increasing for  $t > 0$  and hence  $f(t) > f(s)$ , which completes the proof.
Lemma H.5. Let  $g(z) = \ell'(z) = -1 / (1 + \exp(z))$ , then we have that
$$
\frac {g (z _ {2})}{g (z _ {1})} \leq 2 \big (1 + \exp (z _ {1} - z _ {2}) \big), \forall z _ {1}, z _ {2} \in \mathbb {R},
$$
$$
\frac {g (z _ {2})}{g (z _ {1})} \geq \frac {1}{4} \exp (z _ {1} - z _ {2}), \forall z _ {1} \in \mathbb {R}, z _ {2} \geq - 1.
$$
Proof of Lemma H.5. We first prove the first inequality. For if  $z_{1} \leq 0$ , we have
$$
\frac {g (z _ {2})}{g (z _ {1})} = \frac {1 + \exp (z _ {1})}{1 + \exp (z _ {2})} \leq \frac {2}{1 + \exp (z _ {2})} \leq 2.
$$
For if  $z_{1} > 0$ , we have
$$
\frac {g (z _ {2})}{g (z _ {1})} = \frac {1 + \exp (z _ {1})}{1 + \exp (z _ {2})} \leq \frac {2 \exp (z _ {1})}{\exp (z _ {2})} = 2 \exp (z _ {1} - z _ {2}).
$$
Thus, for any  $z_{1}\in \mathbb{R}$  , we have
$$
\frac {g (z _ {2})}{g (z _ {1})} \leq 2 + 2 \exp (z _ {1} - z _ {2}).
$$
Now we prove the second inequality. We have
$$
\frac {g (z _ {2})}{g (z _ {1})} = \frac {1 + \exp (z _ {1})}{1 + \exp (z _ {2})} = \frac {\exp (- z _ {2}) + \exp (z _ {1} - z _ {2})}{\exp (- z _ {2}) + 1} \geq \frac {\exp (z _ {1} - z _ {2})}{\exp (1) + 1} \geq \frac {1}{4} \exp (z _ {1} - z _ {2}),
$$