SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \frac {n _ {m}}{n _ {i}} \sum_ {j = 1} ^ {n _ {i} / n _ {m}} \left[ \nabla \ell (\mathbf {w}; \mathbf {z} ^ {(i, j)}) - \nabla R _ {i} (\mathbf {w}) \right] \right\| _ {w, *} \geq 2 G (\sqrt {2 \kappa} + \sqrt {2} \gamma) \sqrt {\frac {n _ {m}}{n _ {i}}} \leq \exp (- \gamma^ {2} / 2) \tag {74}
$$
which is a directly consequence of the concentration inequality of vector norms [Juditsky and Nemirovski, 2008, Theorem 2.1.(iii)] and (62). Then, we introduce the following lemma to simplify (74).
Lemma 6 Suppose we have
$$
\Pr \left[ X \geq \alpha + \gamma \right] \leq \exp \left(- \gamma^ {2} / 2\right), \forall \gamma > 0
$$
where  $X$  is nonnegative. Then, we have
$$
\operatorname * {P r} \left[ X \geq \gamma \right] \leq 2 \exp \left(- \gamma^ {2} / \max (6 \alpha^ {2}, 8)\right), \forall \gamma > 0.
$$
From (74) and Lemma 6, we have
$$
\begin{array}{l} \Pr \left[ \frac {1}{2 \sqrt {2} G} \sqrt {\frac {n _ {i}}{n _ {m}}} \left\| \frac {n _ {m}}{n _ {i}} \sum_ {j = 1} ^ {n _ {i} / n _ {m}} \left[ \nabla \ell (\mathbf {w}; \mathbf {z} ^ {(i, j)}) - \nabla R _ {i} (\mathbf {w}) \right] \right\| _ {w, *} \geq \gamma \right] \\ \leq 2 \exp \left(- \gamma^ {2} / \max  (6 \kappa , 8)\right) \leq 2 \exp \left(- \gamma^ {2} / (8 \kappa)\right), \forall \gamma > 0 \\ \end{array}
$$
which satisfies the Proposition 1.(i). From the equivalence between Proposition 1.(i) and Proposition 1.(iv), we have
$$
\mathrm {E} \left[ \exp \left(\left\| \frac {n _ {m}}{n _ {i}} \sum_ {j = 1} ^ {n _ {i} / n _ {m}} \left[ \nabla \ell (\mathbf {w}; \mathbf {z} ^ {(i, j)}) - \nabla R _ {i} (\mathbf {w}) \right] \right\| _ {w, *} ^ {2} \Bigg / \frac {c \kappa G ^ {2} n _ {m}}{n _ {i}}\right) \right] \leq 2.
$$
Inserting the scaling factor  $p_i$ , we have
$$
\mathrm {E} \left[ \right. \exp \left( \right.\left\| p _ {i} \frac {n _ {m}}{n _ {i}} \sum_ {j = 1} ^ {n _ {i} / n _ {m}} \left[ \nabla \ell (\mathbf {w}; \mathbf {z} ^ {(i, j)}) - \nabla R _ {i} (\mathbf {w}) \right]\right\| _ {w, *} ^ {2} / \left. \frac {c \kappa G ^ {2} p _ {i} ^ {2} n _ {m}}{n _ {i}}\right)\left. \right] \leq 2. \tag {75}
$$
To simplify the notation, we define
$$
\mathbf {u} _ {i} = p _ {i} \frac {n _ {m}}{n _ {i}} \sum_ {j = 1} ^ {n _ {i} / n _ {m}} \left[ \nabla \ell (\mathbf {w}; \mathbf {z} ^ {(i, j)}) - \nabla R _ {i} (\mathbf {w}) \right], \mathrm {a n d} \omega_ {\max} = \max _ {i \in [ m ]} \frac {p _ {i} ^ {2} n _ {m}}{n _ {i}}.
$$
$$
\begin{array}{l} \left. \right. \operatorname {E} \left[ \exp \left(\frac {1}{c \kappa G ^ {2} \omega_ {\max}} \| \nabla_ {\mathbf {w}} \varphi (\mathbf {w}, \mathbf {q}) - \mathbf {g} _ {w} (\mathbf {w}, \mathbf {q}) \| _ {w, *} ^ {2}\right)\right] \\ = \mathrm {E} \left[ \exp \left(\left\| \sum_ {i = 1} ^ {m} q _ {i} \mathbf {u} _ {i} \right\| _ {w, *} ^ {2} / \left[ c \kappa G ^ {2} \omega_ {\max } \right]\right) \right] \\ \leq \sum_ {i = 1} ^ {m} q _ {i} \mathrm {E} \left[ \exp \left(\| \mathbf {u} _ {i} \| _ {w, *} ^ {2} / \left[ c \kappa G ^ {2} \omega_ {\max } \right]\right) \right] \stackrel {(7 5)} {\leq} \sum_ {i = 1} ^ {m} q _ {i} 2 = 2. \\ \end{array}
$$
where we use the fact that  $\| \cdot \|_{w, *}, (\cdot)^2$  and  $\exp(\cdot)$  are convex, and the last two functions are increasing in  $\mathbb{R}_+$ .
To analyze  $\| \nabla_{\mathbf{q}}\varphi (\mathbf{w},\mathbf{q}) - \mathbf{g}_q(\mathbf{w},\mathbf{q})\|_\infty^2$  , we consider the approximation error related to  $\mathcal{P}_i$  ..
$$
\left| \frac {n _ {m}}{n _ {i}} \sum_ {j = 1} ^ {n _ {i} / n _ {m}} \ell (\mathbf {w}; \mathbf {z} ^ {(i, j)}) - R _ {i} (\mathbf {w}) \right| = \left| \frac {n _ {m}}{n _ {i}} \sum_ {j = 1} ^ {n _ {i} / n _ {m}} \left[ \ell (\mathbf {w}; \mathbf {z} ^ {(i, j)}) - R _ {i} (\mathbf {w}) \right] \right|.
$$
Note that the absolute value  $|\cdot|$  is 1-regular [Juditsky and Nemirovski, 2008]. Following (59) and the derivation of (75), we have
$$
\left[ \exp \left(\left| p _ {i} \frac {n _ {m}}{n _ {i}} \sum_ {j = 1} ^ {n _ {i} / n _ {m}} \left[ \ell (\mathbf {w}; \mathbf {z} ^ {(i, j)}) - R _ {i} (\mathbf {w}) \right] \right| ^ {2} / \frac {c p _ {i} ^ {2} n _ {m}}{n _ {i}}\right) \right] \leq 2. \tag {76}
$$
To prove that  $\| \nabla_{\mathbf{q}}\varphi (\mathbf{w},\mathbf{q}) - \mathbf{g}_q(\mathbf{w},\mathbf{q})\|_\infty^2$  is also sub-Gaussian, we need to analyze the effect of the infinity norm. To this end, we develop the following lemma.
Lemma 7 Suppose
$$
\operatorname {E} \left[ \exp \left(| X _ {j} | ^ {2} / K _ {j} ^ {2}\right) \right] \leq 2, \forall j \in [ m ]. \tag {77}
$$
$$
\left. \right. \operatorname {E} \left[ \exp \left(\max  _ {j \in [ m ]} | X _ {j} | ^ {2} / \left[ c K _ {\max } ^ {2} \ln m \right]\right)\right] \leq 2.
$$
where  $c > 0$  is an absolute constant, and  $K_{\mathrm{max}} = \max_{j\in [m]}K_j$
From (76) and Lemma 7, we have
$$
\mathrm {E} \left[ \exp \left(\frac {1}{c \omega_ {\mathrm {m a x}} \ln m} \| \nabla_ {\mathbf {q}} \varphi (\mathbf {w}, \mathbf {q}) - \mathbf {g} _ {q} (\mathbf {w}, \mathbf {q}) \| _ {\infty} ^ {2}\right) \right] \leq 2.
$$
When  $\gamma \in [0,2\alpha ]$  , we have
$$
\operatorname * {P r} \left[ X \geq \gamma \right] \leq 1 \leq 2 \exp (- 2 / 3) \leq 2 \exp (- \gamma^ {2} / 6 \alpha^ {2}).
$$
When  $\gamma \geq 2\alpha$  , we have
$$
\Pr [ X \geq \gamma ] = \Pr [ X \geq \alpha + \gamma - \alpha ] \leq \exp (- (\gamma - \alpha) ^ {2} / 2) \leq \exp (- \gamma^ {2} / 8)
$$
where we use the fact
$$
\gamma - \alpha \geq \frac {\gamma}{2}.
$$
Thus, we always have
From (77), and the equivalence between Proposition 1.(i) and Proposition 1.(iv), we have
$$
\Pr \left[ | X _ {j} | \geq t \right] \leq 2 \exp \left(- t ^ {2} / c K _ {j} ^ {2}\right), \forall t \geq 0, \forall j \in [ m ].
$$
$$
\begin{array}{l} \Pr \left[ \max  _ {j \in [ m ]} | X _ {j} | \geq t \right] = \Pr \left[ \exists j, | X _ {j} | \geq t \right] \leq \sum_ {j = 1} ^ {m} \Pr \left[ | X _ {j} | \geq t \right] \leq 2 \sum_ {j = 1} ^ {m} \exp \left(- t ^ {2} / c K _ {j} ^ {2}\right) \\ \leq 2 m \exp \left(- t ^ {2} / c K _ {\mathrm {m a x}} ^ {2}\right) = \exp \left(- t ^ {2} / c K _ {\mathrm {m a x}} ^ {2} + \ln [ 2 m ]\right). \\ \end{array}
$$
Choosing  $t = \sqrt{cK_{\max}^2(\ln[2m] + \gamma^2 / 2)}$ , we have
$$
\operatorname * {P r} \left[ \max  _ {j \in [ m ]} | X _ {j} | \geq \sqrt {c K _ {\max } ^ {2} (\ln [ 2 m ] + \gamma^ {2} / 2)} \right] \leq \exp \left(- \gamma^ {2} / 2\right).
$$
$$
\begin{array}{l} \Pr \left[ \max  _ {j \in [ m ]} | X _ {j} | \geq \sqrt {c K _ {\max } ^ {2}} \left(\sqrt {\ln [ 2 m ]} + \gamma / \sqrt {2}\right) \right] \leq \exp \left(- \gamma^ {2} / 2\right) \\ \Leftrightarrow \Pr \left[ \sqrt {\frac {2}{c K _ {\operatorname* {m a x}} ^ {2}}} \max  _ {j \in [ m ]} | X _ {j} | \geq \sqrt {2 \ln [ 2 m ]} + \gamma \right] \leq \exp \left(- \gamma^ {2} / 2\right). \\ \end{array}
$$
By Lemma 6, we have
$$
\operatorname * {P r} \left[ \sqrt {\frac {2}{c K _ {\max} ^ {2}}} \max  _ {j \in [ m ]} | X _ {j} | \geq \gamma \right] \leq 2 \exp \left(- \gamma^ {2} / \max  \left(1 2 \cdot \ln [ 2 m ], 8\right)\right), \forall \gamma > 0.
$$
From the equivalence between Proposition 1.(i) and Proposition 1.(iv), we have