SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \nabla_ {\Theta} \mathcal {F} (\boldsymbol {x}; \Theta) \right\| _ {2} \leq \mathcal {O} \left(\sqrt {m _ {\mathcal {F}} L _ {\mathcal {F}}}\right), \\ \left\| \nabla_ {\Theta} \mathcal {L} (\boldsymbol {x}; \Theta) \right\| _ {2} \leq \mathcal {O} \left(\sqrt {m _ {\mathcal {F}} L _ {\mathcal {F}}}\right). \\ \end{array}
$$
Proof. This lemma is a direct application of Lemma 9 of [47] and Lemma B.2, B.3 of [11].
Lemma D.16. Suppose  $m_{\mathcal{F}}, \eta_1, \eta_2$  satisfy the conditions in Theorem 5.2. With probability at least  $1 - \mathcal{O}(KL_{\mathcal{F}}) \cdot \exp(-\Omega(m_{\mathcal{F}}\omega^{2/3}L_{\mathcal{F}}))$ , for all  $t \in [T]$ ,  $i \in [k]$ ,  $\Theta, \Theta'$  (or  $\Theta, \Theta'$ ) satisfying  $\| \Theta - \Theta^{(0)} \|_2$ ,  $\| \Theta' - \Theta^{(0)} \|_2 \leq \omega$  with  $\omega \leq \mathcal{O}(L_{\mathcal{F}}^{-9/2}[\log m_{\mathcal{F}}]^{-3})$ , it holds uniformly that
$$
| \mathcal {F} (\pmb {x}; \Theta) - \mathcal {F} (\pmb {x}; \Theta^ {\prime}) - \langle \nabla_ {\Theta^ {\prime}} \mathcal {F} (\pmb {x}; \Theta^ {\prime}), \Theta - \Theta^ {\prime} \rangle | \leq \mathcal {O} (w ^ {1 / 3} L _ {\mathcal {F}} ^ {2} \sqrt {m _ {\mathcal {F}} \log (m _ {\mathcal {F}})}) \| \Theta - \Theta^ {\prime} \| _ {2}.
$$
Proof. The proof for this lemma directly follows the proof of Lemma 4.1 in [11] and Lemma 7 in [47].  $\square$