SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname * {P r} \left[ \epsilon_ {\varphi} (\bar {\mathbf {w}}, \bar {\mathbf {q}}) \geq \frac {7 \widetilde {L}}{n _ {m}} + 1 4 \sqrt {\frac {2 \sigma^ {2}}{3 n _ {m}}} + 7 \Lambda \sqrt {\frac {\sigma^ {2}}{n _ {m}}} \right] \leq \exp \left(- \frac {\Lambda^ {2}}{3}\right) + \exp \left(- \frac {\Lambda n _ {m}}{2}\right)
$$
for all  $\Lambda > 0$ . Choosing  $\Lambda$  such that  $\exp(-\Lambda^2/3) \leq \delta/2$  and  $\exp(-\Lambda n_m/2) \leq \delta/2$ , we have with probability at least  $1 - \delta$
$$
\epsilon_ {\varphi} (\bar {\mathbf {w}}, \bar {\mathbf {q}}) \leq \frac {7 \widetilde {L}}{n _ {m}} + 1 4 \sqrt {\frac {2 \sigma^ {2}}{3 n _ {m}}} + 7 \left(\sqrt {3 \log \frac {2}{\delta}} + \frac {2}{n _ {m}} \log \frac {2}{\delta}\right) \sqrt {\frac {\sigma^ {2}}{n _ {m}}}.
$$
Following the derivation of (58), we have
$$
R _ {i} (\bar {\mathbf {w}}) - \frac {1}{p _ {i}} \min  _ {\mathbf {w} \in \mathcal {W}} \max  _ {i \in [ m ]} p _ {i} R _ {i} (\mathbf {w})
$$
$$
\leq \frac {1}{p _ {i}} \left(\frac {7 \widetilde {L}}{n _ {m}} + \sqrt {\frac {\sigma^ {2}}{n _ {m}}} \left(1 4 \sqrt {\frac {2}{3}} + 7 \sqrt {3 \log \frac {2}{\delta}} + \frac {1 4}{n _ {m}} \log \frac {2}{\delta}\right)\right). \tag {68}
$$
Inspired by Juditsky et al. [2011, §4.3.1], we use the value of  $p_i$  in (32) to simplify (68). It is easy to verify that
$$
\begin{array}{l} \frac {p _ {\mathrm {m a x}}}{p _ {i}} = \frac {1 / \sqrt {n _ {m}} + \sqrt {n _ {m} / n _ {i}}}{1 / \sqrt {n _ {m}} + \sqrt {n _ {m} / n _ {1}}} \leq \left(1 + \frac {n _ {m}}{\sqrt {n _ {i}}}\right), \\ \frac {1}{p _ {i}} \frac {\widetilde {L}}{n _ {m}} = O \left(\frac {p _ {\operatorname* {m a x}}}{p _ {i}} \frac {\sqrt {\ln m}}{n _ {m}}\right) = O \left(\left(\frac {1}{n _ {m}} + \frac {1}{\sqrt {n _ {i}}}\right) \sqrt {\ln m}\right), \tag {69} \\ \end{array}
$$
$$
p _ {i} \leq \left(\frac {1}{\sqrt {n _ {m}}} + 1\right) \sqrt {\frac {n _ {i}}{n _ {m}}}, \quad \omega_ {\max} = \max _ {i \in [ m ]} \frac {p _ {i} ^ {2} n _ {m}}{n _ {i}} \leq \left(\frac {1}{\sqrt {n _ {m}}} + 1\right) ^ {2},
$$
$$
\frac {1}{p _ {i}} \sqrt {\omega_ {\mathrm {m a x}}} = \frac {1 / \sqrt {n _ {m}} + \sqrt {n _ {m} / n _ {i}}}{1 / \sqrt {n _ {m}} + 1} \sqrt {\omega_ {\mathrm {m a x}}} \leq \frac {1}{\sqrt {n _ {m}}} + \sqrt {\frac {n _ {m}}{n _ {i}}},
$$
$$
\frac {1}{p _ {i}} \sqrt {\frac {\sigma^ {2}}{n _ {m}}} = O \left(\frac {1}{p _ {i}} \sqrt {\frac {\omega_ {\operatorname* {m a x}} (\kappa + \ln^ {2} m)}{n _ {m}}}\right) = O \left(\left(\frac {1}{n _ {m}} + \frac {1}{\sqrt {n _ {i}}}\right) \sqrt {\kappa + \ln^ {2} m}\right). \tag {70}
$$
(a) The synthetic data
(b) The Adult dataset
(a) The synthetic data
Figure 2: Balanced settings: max risk of different methods versus the number of samples
Figure 1: Balanced settings: max risk of different methods versus the number of iterations
(b) The Adult dataset
Substituting (69) and (70) into (68), we have
$$
R _ {i} (\bar {\mathbf {w}}) - \frac {1}{p _ {i}} \min _ {\mathbf {w} \in \mathcal {W}} \max _ {i \in [ m ]} p _ {i} R _ {i} (\mathbf {w}) = O \left(\left(\frac {1}{n _ {m}} + \frac {1}{\sqrt {n _ {i}}}\right) \sqrt {\kappa + \ln^ {2} m}\right).
$$
In this section, we conduct empirical studies to evaluate our proposed algorithms.
Following the setup in previous work [Namkoong and Duchi, 2016, Soma et al., 2022], we use both synthetic and real-world datasets. First, we construct a synthetic data with group number  $m = 20$ . For each group  $i \in [m]$ , we draw a model  $\mathbf{w}_i^* \in \mathbb{R}^{1000}$  from the uniform distribution over the unit sphere. For distribution  $\mathcal{P}_i$ , the sample  $(\mathbf{x}, y)$  is generated with  $\mathbf{x} \sim \mathcal{N}(0, I)$  and  $y = \mathrm{sign}(\mathbf{x}^\top \mathbf{w}_i^*)$  with probability 0.9 and  $y = -\mathrm{sign}(\mathbf{x}^\top \mathbf{w}_i^*)$  with probability 0.1.
We also use the Adult dataset [Becker and Kohavi, 1996], which includes attributes such as age, gender, race, and educational background of 48842 individuals. The objective is to determine whether an individual's income exceeds 50000 USD. We set up  $m = 6$  groups based on the race and gender attributes, where each group represents a combination of {black, white, others} with {female, male}.
(a) Risk on  $\mathcal{P}_1$
(b) Risk on  $\mathcal{P}_4$
(c) Risk on  $\mathcal{P}_6$
(d) Risk on  $\mathcal{P}_8$
(e) Risk on  $\mathcal{P}_{10}$
(f) Risk on  $\mathcal{P}_{12}$
(g) Risk on  $\mathcal{P}_{16}$
(h) Risk on  $\mathcal{P}_{18}$
(i) Risk on  $\mathcal{P}_{20}$
Figure 3: Imbalanced settings with the synthetic dataset: individual risk versus the number of iterations
C.2 GDRO with Balanced Data
For experiments on the synthetic dataset, we will generate the random sample on the fly, according to the protocol above. For those on the Adult dataset, we will randomly select samples from ech group. In other words,  $\mathcal{P}_i$  is defined as the empirical distribution over the data in the  $i$ -th group.
We refer to the method of Sagawa et al. [2020] and our Algorithm 1 by SMD(1) and SMD(m) to underscore that they are instances of SMD with 1 sample and  $m$  samples in each iteration, respectively. We denote our Algorithm 2 as Online(1) to emphasize that it is based on techniques from online learning and uses 1 sample per iteration. We set  $\ell(\cdot; \cdot)$  to be the logistic loss and utilize different methods to train a linear model. In the balanced setting, we use the max risk, i.e.,  $\max_{i \in [m]} R_i(\mathbf{w})$ , as the performance measure. To estimate the risk value, we will draw a substantial number of samples, and use the empirical average to approximate the expectation.
We first report the max risk with respect to the number of iterations in Fig. 1. We observe that  $\mathrm{SMD}(m)$  is faster than Online(1), which in turn outperforms SMD(1). This observation is consistent with our theories, since their convergence rates are  $O(\sqrt{(\log m) / T})$ ,  $O(\sqrt{m(\log m) / T})$ , and  $O(m\sqrt{(\log m) / T})$ , respectively. Next, we plot the max risk against the number of samples consumed by each algorithm in Fig. 2. As can be seen, the curves of  $\mathrm{SMD}(m)$  and Online(1) are very close, indicating that they share the same sample complexity, i.e.,  $O(m(\log m) / \epsilon^2)$ . By contrast, SMD(1) has a much higher sample complexity, i.e.,  $O(m^2 (\log m) / \epsilon^2)$ .
(b) Risk on  $\mathcal{P}_2$
(c) Risk on  $\mathcal{P}_3$
(d) Risk on  $\mathcal{P}_4$
(e) Risk on  $\mathcal{P}_5$
(f) Risk on  $\mathcal{P}_6$
Figure 4: Imbalanced settings with the Adult dataset: individual risk versus the number of iterations
C.3 GDRO with Imbalanced Data
For experiments on the synthetic dataset, we set the number of samples as  $n_i = 800 \times (21 - i)$ , and generate each sample as before. For those on the Adult dataset, we first select 364 samples randomly from each group, reserving them for later use in estimating the risk of each group. Then, we visit the remaining samples in each group once to simulate the imbalanced setting, where the numbers of samples in 6 groups are 26656, 11518, 1780, 1720, 998, and 364.
Similarly, we label the Baseline mentioned in the first paragraph of Section 3 as  $\mathrm{SMD}(m)$ . We designate our Algorithms 3 and 4 as  $\mathrm{SMD_r}$  and  $\mathrm{SMPA_m}$  to highlight that the former combines SMD with random sampling and the latter one integrates SMPA and mini-batches. In the imbalanced setting, we will examine how the risk on each individual distribution decreases with respect to the number of iterations. Recall that the total number of iterations of  $\mathrm{SMD}(m)$ ,  $\mathrm{SMD_r}$  and  $\mathrm{SMPA_m}$  are  $n_m$ ,  $n_1$  and  $n_m / 2$ , respectively.
We present the risk on the individual distribution in Fig. 3 and Fig. 4. First, we observe that our  $\mathrm{SMPA}_{\mathrm{m}}$  is faster than both  $\mathrm{SMD}(m)$  and  $\mathrm{SMD}_{\mathrm{r}}$  across all distributions, and finally attains the lower risk is most cases. This behavior aligns with our Theorem 4, which reveals that  $\mathrm{SMPA}_{\mathrm{m}}$  achieves a nearly optimal rate of  $O((\log m) / \sqrt{n_i})$  for all distributions  $\mathcal{P}_i$ , after  $n_m / 2$  iterations. We also note that on distribution  $\mathcal{P}_1$ , although  $\mathrm{SMD}_{\mathrm{r}}$  converges slowly, its final risk is the lowest, as illustrated in Fig. 3(a) and Fig. 4(a). This phenomenon is again in accordance with our Theorem 3, which shows that the risk of  $\mathrm{SMD}_{\mathrm{r}}$  on  $\mathcal{P}_1$  reduces at a nearly optimal  $O(\sqrt{(\log m) / n_1})$  rate, after  $n_1$  iterations. From Fig. 3(i) and Fig. 4(f), we can see that the final risk of  $\mathrm{SMD}(m)$  on the last distribution  $\mathcal{P}_m$  matches that of  $\mathrm{SMPA}_{\mathrm{m}}$ . This outcome is anticipated, as they exhibit similar convergence rates of  $O(\sqrt{(\log m) / n_m})$  and  $O((\log m) / \sqrt{n_m})$ , respectively.
The proof follows the argument of Neu [2015, Proof of Lemma 1], and we generalize it to the setting with stochastic rewards. First, observe that for any  $i \in [m]$  and  $t \in [T]$ ,
$$
\begin{array}{l} \tilde {\xi} _ {t, i} = \frac {\hat {\xi} _ {t , i}}{p _ {t , i} + \gamma_ {t}} \cdot \mathbb {I} [ i _ {t} = i ] \\ \leq \frac {\hat {\xi} _ {t , i}}{p _ {t , i} + \gamma_ {t} \hat {\xi} _ {t , i}} \cdot \mathbb {I} [ i _ {t} = i ] \quad (\hat {\xi} _ {t, i} \in [ 0, 1 ]) \\ = \frac {1}{2 \gamma_ {t}} \frac {2 \gamma_ {t} \cdot \hat {\xi} _ {t , i} / p _ {t , i}}{1 + \gamma_ {t} \cdot \hat {\xi} _ {t , i} / p _ {t , i}} \cdot \mathbb {I} [ i _ {t} = i ] \\ \leq \frac {1}{\beta_ {t}} \log \left(1 + \beta_ {t} \bar {\xi} _ {t, i}\right) \tag {71} \\ \end{array}
$$
where the last step is due to the inequality  $\frac{z}{1 + z / 2} \leq \log(1 + z)$  for  $z \geq 0$  and we introduce the notations  $\beta_{t} = 2\gamma_{t}$  and  $\bar{\xi}_{t,i} = (\hat{\xi}_{t,i} / p_{t,i}) \cdot \mathbb{I}[i_{t} = i]$  to simplify the presentation.
Define the notation  $\tilde{\lambda}_t = \sum_{i=1}^m \alpha_{t,i} \tilde{\xi}_{t,i}$  and  $\lambda_t = \sum_{i=1}^m \alpha_{t,i} \xi_{t,i}$ . Then, we have
$$
\begin{array}{l} \mathrm {E} _ {t - 1} \left[ \exp (\tilde {\lambda} _ {t}) \right] = \mathrm {E} _ {t - 1} \left[ \exp \left(\sum_ {i = 1} ^ {m} \alpha_ {t, i} \tilde {\xi} _ {t, i}\right) \right] \\ \stackrel {(7 1)} {\leq} \mathrm {E} _ {t - 1} \left[ \exp \left(\sum_ {i = 1} ^ {m} \frac {\alpha_ {t , i}}{\beta_ {t}} \log \left(1 + \beta_ {t} \bar {\xi} _ {t, i}\right)\right) \right] \\ \leq \mathrm {E} _ {t - 1} \left[ \exp \left(\sum_ {i = 1} ^ {m} \log \left(1 + \alpha_ {t, i} \bar {\xi} _ {t, i}\right)\right) \right] \quad \left(\frac {\alpha_ {t , i}}{\beta_ {t}} \leq 1 \text {b y a s s u m p t i o n}\right) \\ = \mathrm {E} _ {t - 1} \left[ \prod_ {i = 1} ^ {m} \left(1 + \alpha_ {t, i} \bar {\xi} _ {t, i}\right) \right] \\ = \mathrm {E} _ {t - 1} \left[ 1 + \sum_ {i = 1} ^ {m} \alpha_ {t, i} \bar {\xi} _ {t, i} \right] \\ = 1 + \sum_ {i = 1} ^ {m} \alpha_ {t, i} \xi_ {t, i} \leq \exp \left(\sum_ {i = 1} ^ {m} \alpha_ {t, i} \xi_ {t, i}\right) = \exp (\lambda_ {t}) \tag {72} \\ \end{array}
$$
where the second inequality is by the inequality  $x\log (1 + y)\leq \log (1 + xy)$  that holds for all  $y\geq -1$  and  $x\in [0,1]$ , and the equality  $\mathrm{E}_{t - 1}\left[\Pi_{i = 1}^{m}\bigl (1 + \alpha_{t,i}\bar{\xi}_{t,i}\bigr)\right] = \mathrm{E}_{t - 1}\left[1 + \sum_{i = 1}^{m}\alpha_{t,i}\bar{\xi}_{t,i}\right]$  follows from the fact that  $\bar{\xi}_{t,i}\cdot \bar{\xi}_{t,j} = 0$  holds whenever  $i\neq j$ . The last line is due to  $\mathrm{E}_{t - 1}[\bar{\xi}_{t,i}] = \mathrm{E}_{t - 1}[(\hat{\xi}_{t,i} / p_{t,i})\cdot \mathbb{I}[i_t = i]] = \xi_{t,i}$  and the inequality  $1 + z\leq e^{z}$  for all  $z\in \mathbb{R}$ .
As a result, from (72) we conclude that the process  $Z_{t} = \exp \left(\sum_{s=1}^{t} (\tilde{\lambda}_{s} - \lambda_{s})\right)$  is a supermartingale. Indeed,  $\mathrm{E}_{t-1}[Z_{t}] = \mathrm{E}_{t-1}\left[\exp \left(\sum_{s=1}^{t-1} (\tilde{\lambda}_{s} - \lambda_{s})\right) \cdot \exp (\tilde{\lambda}_{t} - \lambda_{t})\right] \leq Z_{t-1}$ . Thus, we have  $\mathrm{E}[Z_T] \leq \mathrm{E}[Z_{T-1} \leq \ldots \leq \mathrm{E}[Z_0] = 1]$ . By Markov's inequality,
$$
\Pr \left[ \sum_ {t = 1} ^ {T} \left(\tilde {\lambda} _ {t} - \lambda_ {t}\right) > \epsilon \right] \leq \mathrm {E} \left[ \exp \left(\sum_ {t = 1} ^ {T} \left(\tilde {\lambda} _ {t} - \lambda_ {t}\right)\right) \right] \cdot \exp (- \epsilon) \leq \exp (- \epsilon)
$$
holds for any  $\epsilon > 0$ . By setting  $\exp(-\epsilon) = \delta$  and solving the value, we complete the proof for (54). And the inequality (55) for the scenario  $\gamma_t = \gamma$  can be immediately obtained by setting  $\alpha_{t,i} = 2\gamma$  and taking the union bound over all  $i \in [m]$ .
From the definition of norms in (34), we have
$$
\begin{array}{l} \left\| F \left(\left[ \mathbf {w}; \mathbf {q} \right]\right) - F \left(\left[ \mathbf {w} ^ {\prime}; \mathbf {q} ^ {\prime} \right]\right) \right\| _ {*} ^ {2} \\ = \left\| \left[ \sum_ {i = 1} ^ {m} q _ {i} p _ {i} \nabla R _ {i} (\mathbf {w}) - \sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \nabla R _ {i} (\mathbf {w} ^ {\prime}); \left[ p _ {1} R _ {1} (\mathbf {w} ^ {\prime}) - p _ {1} R _ {1} (\mathbf {w}), \dots , p _ {m} R _ {m} (\mathbf {w} ^ {\prime}) - p _ {m} R _ {m} (\mathbf {w}) \right] ^ {\top} \right] \right\| _ {*} ^ {2} \\ = 2 D ^ {2} \left\| \sum_ {i = 1} ^ {m} q _ {i} p _ {i} \nabla R _ {i} (\mathbf {w}) - \sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \nabla R _ {i} (\mathbf {w} ^ {\prime}) \right\| _ {w, *} ^ {2} \\ + 2 \left\| \left[ p _ {1} R _ {1} (\mathbf {w} ^ {\prime}) - p _ {1} R _ {1} (\mathbf {w}), \dots , p _ {m} R _ {m} (\mathbf {w} ^ {\prime}) - p _ {m} R _ {m} (\mathbf {w}) \right] ^ {\top} \right\| _ {\infty} ^ {2} \ln m \\ = 2 D ^ {2} \left\| \sum_ {i = 1} ^ {m} q _ {i} p _ {i} \nabla R _ {i} (\mathbf {w}) - \sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \nabla R _ {i} (\mathbf {w}) + \sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \nabla R _ {i} (\mathbf {w}) - \sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \nabla R _ {i} (\mathbf {w} ^ {\prime}) \right\| _ {w, *} ^ {2} \\ + 2 \left\| \left[ p _ {1} R _ {1} (\mathbf {w} ^ {\prime}) - p _ {1} R _ {1} (\mathbf {w}), \dots , p _ {m} R _ {m} (\mathbf {w} ^ {\prime}) - p _ {m} R _ {m} (\mathbf {w}) \right] ^ {\top} \right\| _ {\infty} ^ {2} \ln m \\ \leq \underbrace {4 D ^ {2} \left\| \sum_ {i = 1} ^ {m} q _ {i} p _ {i} \nabla R _ {i} (\mathbf {w}) - \sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \nabla R _ {i} (\mathbf {w}) \right\| _ {w , *} ^ {2}} _ {:= A} + \underbrace {4 D ^ {2} \left\| \sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \nabla R _ {i} (\mathbf {w}) - \sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \nabla R _ {i} (\mathbf {w} ^ {\prime}) \right\| _ {w , *} ^ {2}} _ {:= B} \\ + \underbrace {2 \max  _ {i \in [ m ]} \left| p _ {i} \left[ R _ {i} (\mathbf {w}) - R _ {i} \left(\mathbf {w} ^ {\prime}\right) \right] \right| ^ {2} \ln m} _ {:= C}. \\ \end{array}
$$
To bound term  $A$ , we have
$$
\begin{array}{l} 4 D ^ {2} \left\| \sum_ {i = 1} ^ {m} q _ {i} p _ {i} \nabla R _ {i} (\mathbf {w}) - \sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \nabla R _ {i} (\mathbf {w}) \right\| _ {w, *} ^ {2} \\ \leq 4 D ^ {2} \left(\sum_ {i = 1} ^ {m} | q _ {i} - q _ {i} ^ {\prime} | \| p _ {i} \nabla R _ {i} (\mathbf {w}) \| _ {w, *}\right) ^ {2} \stackrel {(6 0)} {\leq} 4 D ^ {2} \left(\sum_ {i = 1} ^ {m} | q _ {i} - q _ {i} ^ {\prime} | p _ {i} G\right) ^ {2} \leq 4 D ^ {2} G ^ {2} p _ {\max } ^ {2} \| \mathbf {q} - \mathbf {q} ^ {\prime} \| _ {1} ^ {2}. \\ \end{array}
$$
where  $p_{\mathrm{max}}$  is defined in (31). To bound  $B$ , we have
$$
\begin{array}{l} 4 D ^ {2} \left\| \sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \nabla R _ {i} (\mathbf {w}) - \sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \nabla R _ {i} (\mathbf {w} ^ {\prime}) \right\| _ {w, *} ^ {2} \\ \leq 4 D ^ {2} \left(\sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} \| \nabla R _ {i} (\mathbf {w}) - \nabla R _ {i} \left(\mathbf {w} ^ {\prime}\right) \| _ {w, *}\right) ^ {2} \stackrel {(3 0)} {\leq} 4 D ^ {2} \left(\sum_ {i = 1} ^ {m} q _ {i} ^ {\prime} p _ {i} L \| \mathbf {w} - \mathbf {w} ^ {\prime} \| _ {w}\right) ^ {2} \\ \leq 4 D ^ {2} L ^ {2} p _ {\max } ^ {2} \| \mathbf {w} - \mathbf {w} ^ {\prime} \| _ {w} ^ {2} \left(\sum_ {i = 1} ^ {m} q _ {i} ^ {\prime}\right) ^ {2} = 4 D ^ {2} L ^ {2} p _ {\max } ^ {2} \| \mathbf {w} - \mathbf {w} ^ {\prime} \| _ {w} ^ {2}. \\ \end{array}
$$
To bound  $C$ , we have
$$
\begin{array}{l} 2 \max _ {i \in [ m ]} \left| p _ {i} \left[ R _ {i} (\mathbf {w}) - R _ {i} (\mathbf {w} ^ {\prime}) \right] \right| ^ {2} \ln m \\ \stackrel {(6 1)} {\leq} 2 \max _ {i \in [ m ]} \left| p _ {i} G \| \mathbf {w} - \mathbf {w} ^ {\prime} \| _ {w} \right| ^ {2} \ln m \leq 2 G ^ {2} p _ {\max } ^ {2} \| \mathbf {w} - \mathbf {w} ^ {\prime} \| _ {w} ^ {2} \ln m. \\ \end{array}
$$
$$
\begin{array}{l} \left\| F ([ \mathbf {w}; \mathbf {q} ]) - F ([ \mathbf {w} ^ {\prime}; \mathbf {q} ^ {\prime} ]) \right\| _ {*} ^ {2} \leq (4 D ^ {2} L ^ {2} + 2 G ^ {2} \ln m) p _ {\max } ^ {2} \| \mathbf {w} - \mathbf {w} ^ {\prime} \| _ {w} ^ {2} + 4 D ^ {2} G ^ {2} p _ {\max } ^ {2} \| \mathbf {q} - \mathbf {q} ^ {\prime} \| _ {1} ^ {2} \\ \leq p _ {\max } ^ {2} \left(8 D ^ {4} L ^ {2} + 8 D ^ {2} G ^ {2} \ln m\right) \left(\frac {1}{2 D ^ {2}} \| \mathbf {w} - \mathbf {w} ^ {\prime} \| _ {w} ^ {2} + \frac {1}{2 \ln m} \| \mathbf {q} - \mathbf {q} ^ {\prime} \| _ {1} ^ {2}\right) \\ = p _ {\max } ^ {2} \left(8 D ^ {4} L ^ {2} + 8 D ^ {2} G ^ {2} \ln m\right) \left\| \left[ \mathbf {w} - \mathbf {w} ^ {\prime}; \mathbf {q} - \mathbf {q} ^ {\prime} \right] \right\| ^ {2} \\ \end{array}
$$
$$
\| F ([ \mathbf {w}; \mathbf {q} ]) - F ([ \mathbf {w} ^ {\prime}; \mathbf {q} ^ {\prime} ]) \| _ {*} \leq p _ {\max} \sqrt {8 D ^ {4} L ^ {2} + 8 D ^ {2} G ^ {2} \ln m} \big \| [ \mathbf {w} - \mathbf {w} ^ {\prime}; \mathbf {q} - \mathbf {q} ^ {\prime} ] \big \| \leq \widetilde {L} \big \| [ \mathbf {w} - \mathbf {w} ^ {\prime}; \mathbf {q} - \mathbf {q} ^ {\prime} ] \big \|
$$
The light tail condition, required by Juditsky et al. [2011], is essentially the sub-Gaussian condition. To this end, we introduce the following sub-gaussian properties [Vershynin, 2018, Proposition 2.5.2].
Proposition 1 (Sub-gaussian properties) Let  $X$  be a random variable. Then the following properties are equivalent; the parameters  $K_{i} > 0$  appearing in these properties differ from each other by at most an absolute constant factor.
(i) The tails of  $X$  satisfy
$$
\Pr [ | X | \geq t ] \leq 2 \exp (- t ^ {2} / K _ {1} ^ {2}), \forall t \geq 0.
$$
(ii) The moments of  $X$  satisfy
$$
\left\| X \right\| _ {L _ {p}} = \left(\operatorname {E} | X | ^ {p}\right) ^ {1 / p} \leq K _ {2} \sqrt {p}, \forall p \geq 1.
$$
(iii) The moment generating function (MGF) of  $X^2$  satisfies
$$
\mathrm {E} \left[ \exp \left(\lambda^ {2} X ^ {2}\right) \right] \leq \exp \left(K _ {3} ^ {2} \lambda^ {2}\right), \forall \lambda \text {s u c h t h a t} | \lambda | \leq 1 / K _ {3}.
$$
(iv) The MGF of  $X^2$  is bounded at some point, namely
$$
\mathrm {E} \left[ \exp \left(X ^ {2} / K _ {4} ^ {2}\right) \right] \leq 2.
$$
Moreover, if  $\operatorname{E}[X] = 0$  then properties (i)-(iv) are also equivalent to the following property. (v) The MGF of  $X$  satisfies
$$
\operatorname {E} \bigl [ \exp (\lambda X) \bigr ] \leq \exp (K _ {5} ^ {2} \lambda^ {2}), \forall \lambda \in \mathbb {R}.
$$
From the above proposition, we observe that the exact value of those constant  $K_{1}, \ldots, K_{5}$  is not important, and it is very tedious to calculate them. So, in the following, we only focus on the order of those constants. To simplify presentations, we use  $c$  to denote an absolute constant that is independent of all the essential parameters, and its value may change from line to line.
$$
\begin{array}{l} \left\| F \left(\left[ \mathbf {w}; \mathbf {q} \right]\right) - \mathbf {g} \left(\left[ \mathbf {w}; \mathbf {q} \right]\right) \right\| _ {*} ^ {2} \\ = 2 D ^ {2} \| \nabla_ {\mathbf {w}} \varphi (\mathbf {w}, \mathbf {q}) - \mathbf {g} _ {w} (\mathbf {w}, \mathbf {q}) \| _ {w, *} ^ {2} + 2 \| \nabla_ {\mathbf {q}} \varphi (\mathbf {w}, \mathbf {q}) - \mathbf {g} _ {q} (\mathbf {w}, \mathbf {q}) \| _ {\infty} ^ {2} \ln m, \\ \end{array}
$$
we proceed to analyze the behavior of  $\| \nabla_{\mathbf{w}}\varphi (\mathbf{w},\mathbf{q}) - \mathbf{g}_w(\mathbf{w},\mathbf{q})\|_{w,*}^2$  and  $\| \nabla_{\mathbf{q}}\varphi (\mathbf{w},\mathbf{q}) - \mathbf{g}_q(\mathbf{w},\mathbf{q})\|_{\infty}^{2}$ . To this end, we have the following lemma.
Lemma 5 We have
$$
\begin{array}{l} \left. \operatorname {E} \left[ \exp \left(\frac {1}{c \kappa G ^ {2} \omega_ {\max }} \| \nabla_ {\mathbf {w}} \varphi (\mathbf {w}, \mathbf {q}) - \mathbf {g} _ {w} (\mathbf {w}, \mathbf {q}) \| _ {w, *} ^ {2}\right) \right] \leq 2, \right. \tag {73} \\ \left. \operatorname {E} \left[ \exp \left(\frac {1}{c \omega_ {\max} \ln m} \| \nabla_ {\mathbf {q}} \varphi (\mathbf {w}, \mathbf {q}) - \mathbf {g} _ {q} (\mathbf {w}, \mathbf {q}) \| _ {\infty} ^ {2}\right) \right] \leq 2 \right. \\ \end{array}
$$
where  $\omega_{\mathrm{max}}$  is defined in (31) and  $c > 0$  is an absolute constant.
From Lemma 5, we have
$$
\begin{array}{l} \mathrm {E} \left[ \exp \left(\frac {1}{2 c \kappa D ^ {2} G ^ {2} \omega_ {\operatorname* {m a x}} + 2 c \omega_ {\operatorname* {m a x}} \ln^ {2} m} \| F ([ \mathbf {w}; \mathbf {q} ]) - \mathbf {g} ([ \mathbf {w}, \mathbf {q} ]) \| _ {*} ^ {2}\right) \right] \\ = \mathrm {E} \left[ \exp \left(\frac {2 D ^ {2}}{2 c \kappa D ^ {2} G ^ {2} \omega_ {\max } + 2 c \omega_ {\max } \ln^ {2} m} \| \nabla_ {\mathbf {w}} \varphi (\mathbf {w}, \mathbf {q}) - \mathbf {g} _ {w} (\mathbf {w}, \mathbf {q}) \| _ {w, *} ^ {2} \right. \right. \\ \left. \left. + \frac {2 \ln m}{2 c \kappa D ^ {2} G ^ {2} \omega_ {\mathrm {m a x}} + 2 c \omega_ {\mathrm {m a x}} \ln^ {2} m} \| \nabla_ {\mathbf {q}} \varphi (\mathbf {w}, \mathbf {q}) - \mathbf {g} _ {q} (\mathbf {w}, \mathbf {q}) \| _ {\infty} ^ {2}\right) \right] \\ = \mathrm {E} \left[ \exp \left(\frac {\kappa D ^ {2} G ^ {2}}{\kappa D ^ {2} G ^ {2} + \ln^ {2} m} \frac {\| \nabla_ {\mathbf {w}} \varphi (\mathbf {w} , \mathbf {q}) - \mathbf {g} _ {w} (\mathbf {w} , \mathbf {q}) \| _ {w , *} ^ {2}}{c \kappa G ^ {2} \omega_ {\max }}\right) \right. \\ \left. \left. + \frac {\ln^ {2} m}{\kappa D ^ {2} G ^ {2} + \ln^ {2} m} \frac {\left\| \nabla_ {\mathbf {q}} \varphi (\mathbf {w} , \mathbf {q}) - \mathbf {g} _ {q} (\mathbf {w} , \mathbf {q}) \right\| _ {\infty} ^ {2}}{c \omega_ {\operatorname* {m a x}} \ln m}\right) \right] \\ \leq \frac {\kappa D ^ {2} G ^ {2}}{\kappa D ^ {2} G ^ {2} + \ln^ {2} m} \mathrm {E} \left[ \exp \left(\frac {\| \nabla_ {\mathbf {w}} \varphi (\mathbf {w} , \mathbf {q}) - \mathbf {g} _ {w} (\mathbf {w} , \mathbf {q}) \| _ {w , *} ^ {2}}{c \kappa G ^ {2} \omega_ {\max}}\right) \right] \\ + \frac {\ln^ {2} m}{\kappa D ^ {2} G ^ {2} + \ln^ {2} m} \mathrm {E} \left[ \exp \left(\frac {\| \nabla_ {\mathbf {q}} \varphi (\mathbf {w} , \mathbf {q}) - \mathbf {g} _ {q} (\mathbf {w} , \mathbf {q}) \| _ {\infty} ^ {2}}{c \omega_ {\max } \ln m}\right) \right] \\ \stackrel {(7 3)} {\leq} \frac {\kappa D ^ {2} G ^ {2}}{\kappa D ^ {2} G ^ {2} + \ln^ {2} m} 2 + \frac {\ln^ {2} m}{\kappa D ^ {2} G ^ {2} + \ln^ {2} m} 2 = 2 \\ \end{array}
$$
where the first inequality follows from Jensen's inequality.
To analyze  $\| \nabla_{\mathbf{w}}\varphi (\mathbf{w},\mathbf{q}) - \mathbf{g}_{w}(\mathbf{w},\mathbf{q})\|_{w,*}^{2}$ , we first consider the approximation error caused by samples from  $\mathcal{P}_i$ :
$$
\left\| \frac {n _ {m}}{n _ {i}} \sum_ {j = 1} ^ {n _ {i} / n _ {m}} \nabla \ell (\mathbf {w}; \mathbf {z} ^ {(i, j)}) - \nabla R _ {i} (\mathbf {w}) \right\| _ {w, *} = \left\| \frac {n _ {m}}{n _ {i}} \sum_ {j = 1} ^ {n _ {i} / n _ {m}} \left[ \nabla \ell (\mathbf {w}; \mathbf {z} ^ {(i, j)}) - \nabla R _ {i} (\mathbf {w}) \right] \right\| _ {w, *}.
$$
Under the regularity condition of  $\| \cdot \|_{w,*}$  in Assumption 6, we have, for any  $\gamma \geq 0$