SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \nabla_ {\mathbf {w}} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \right\| _ {w, *} \leq \left\| \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \right\| _ {w, *} + \left\| \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \right\| _ {w, *} \\ \left. \right. _ {(6), (1 0)} \tag {44} \\ \leq \mathrm {E} _ {t - 1} \left[ \| \tilde {\mathbf {g}} _ {w} \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right) \| _ {w, *} \right] + \| \tilde {\mathbf {g}} _ {w} \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right) \| _ {w, *} \leq 2 G. \\ \end{array}
$$
$$
\begin{array}{l} \max  _ {\mathbf {w} \in \mathcal {W}} \left\{\sum_ {t = 1} ^ {T} \langle \nabla_ {\mathbf {w}} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}), \mathbf {w} _ {t} - \mathbf {w} \rangle \right\} \\ = \max  _ {\mathbf {w} \in \mathcal {W}} \left\{\sum_ {t = 1} ^ {T} \langle \nabla_ {\mathbf {w}} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}), \mathbf {v} _ {t} - \mathbf {w} \rangle \right\} + \sum_ {t = 1} ^ {T} \langle \nabla_ {\mathbf {w}} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}), \mathbf {w} _ {t} - \mathbf {v} _ {t} \rangle \\ \stackrel {(4 3)} {\leq} \frac {D ^ {2}}{\eta_ {w}} + 2 \eta_ {w} T G ^ {2} + \sum_ {t = 1} ^ {T} \underbrace {\left\langle \nabla_ {\mathbf {w}} \phi \left(\mathbf {w} _ {t} , \mathbf {q} _ {t}\right) - \tilde {\mathbf {g}} _ {w} \left(\mathbf {w} _ {t} , \mathbf {q} _ {t}\right) , \mathbf {w} _ {t} - \mathbf {v} _ {t} \right\rangle}. \tag {45} \\ \end{array}
$$
From the updating rule of  $\mathbf{v}_t$  in (42), we know that  $\mathbf{v}_t$  is independent from  $\nabla_{\mathbf{w}}\phi (\mathbf{w}_t,\mathbf{q}_t) - \tilde{\mathbf{g}}_w(\mathbf{w}_t,\mathbf{q}_t)$  and thus  $V_{1},\ldots ,V_{T}$  is a martingale difference sequence.
Substituting (45) into (41), we have
$$
\sum_ {t = 1} ^ {T} \phi \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right) - \min  _ {\mathbf {w} \in \mathcal {W}} \sum_ {t = 1} ^ {T} \phi \left(\mathbf {w}, \mathbf {q} _ {t}\right) \leq \frac {2 D ^ {2}}{\eta_ {w}} + \frac {5 \eta_ {w} T G ^ {2}}{2} + \sum_ {t = 1} ^ {T} V _ {t}. \tag {46}
$$
Taking expectation over both sides, we have
$$
\mathrm {E} \left[ \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \min _ {\mathbf {w} \in \mathcal {W}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w}, \mathbf {q} _ {t}) \right] \leq \frac {2 D ^ {2}}{\eta_ {w}} + \frac {5 \eta_ {w} T G ^ {2}}{2} = 2 D G \sqrt {5 T}
$$
where we set  $\eta_w = \frac{2D}{G\sqrt{5T}}$
To establish a high probability bound, we make use of the Hoeffding-Azuma inequality for martingales stated below [Cesa-Bianchi and Lugosi, 2006].
Lemma 1 Let  $V_{1}, V_{2}, \ldots$  be a martingale difference sequence with respect to some sequence  $X_{1}, X_{2}, \ldots$  such that  $V_{i} \in [A_{i}, A_{i} + c_{i}]$  for some random variable  $A_{i}$ , measurable with respect to  $X_{1}, \ldots, X_{i-1}$  and a positive constant  $c_{i}$ . If  $S_{n} = \sum_{i=1}^{n} V_{i}$ , then for any  $t > 0$ ,
$$
\operatorname * {P r} [ S _ {n} > t ] \leq \exp \left(- \frac {2 t ^ {2}}{\sum_ {i = 1} ^ {n} c _ {i} ^ {2}}\right).
$$
To apply the above lemma, we need to show that  $V_{t}$  is bounded. Indeed, we have
$$
\left| \left\langle \nabla_ {\mathbf {w}} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}), \mathbf {w} _ {t} - \mathbf {v} _ {t} \right\rangle \right| \leq \| \nabla_ {\mathbf {w}} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \| _ {w, *} \| \mathbf {w} _ {t} - \mathbf {v} _ {t} \| _ {w}
$$
$$
\begin{array}{l} \stackrel {(4 4)} {\leq} 2 G \left\| \mathbf {w} _ {t} - \mathbf {v} _ {t} \right\| _ {w} \leq 2 G \left(\left\| \mathbf {w} _ {t} - \mathbf {w} _ {1} \right\| _ {w} + \left\| \mathbf {v} _ {t} - \mathbf {w} _ {1} \right\| _ {w}\right) \\ \leq 2 G \left(\sqrt {2 B _ {w} (\mathbf {w} _ {t} , \mathbf {w} _ {1})} + \sqrt {2 B _ {w} (\mathbf {v} _ {t} , \mathbf {w} _ {1})}\right) \stackrel {(4 0)} {\leq} 4 \sqrt {2} D G. \\ \end{array}
$$
From Lemma 1, with probability at least  $1 - \delta$ , we have
$$
\sum_ {t = 1} ^ {T} V _ {t} \leq 8 D G \sqrt {T \ln \frac {1}{\delta}}. \tag {47}
$$
We complete the proof by substituting (47) into (46).
B.4 Proof of Theorem 6
Because we can only observe  $\ell(\mathbf{w}_t, \mathbf{z}_t^{(i_t)})$  instead of  $R_{i_t}(\mathbf{w}_t)$ , the theoretical guarantee of Exp3-IX [Neu, 2015] cannot be directly applied to Algorithm 2. To address this challenge, we generalize the regret analysis of Exp3-IX to stochastic rewards.
By the definition of  $\phi (\mathbf{w},\mathbf{q})$  in (3) and the property of linear optimization over the simplex, we have
$$
\begin{array}{l} \max _ {\mathbf {q} \in \Delta_ {m}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q}) - \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) = \max _ {\mathbf {q} \in \Delta_ {m}} \sum_ {i = 1} ^ {m} q _ {i} \left(\sum_ {t = 1} ^ {T} R _ {i} (\mathbf {w} _ {t})\right) - \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} q _ {t, i} R _ {i} (\mathbf {w} _ {t}) \\ = \sum_ {t = 1} ^ {T} R _ {j ^ {*}} \left(\mathbf {w} _ {t}\right) - \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} q _ {t, i} R _ {i} \left(\mathbf {w} _ {t}\right) = \sum_ {t = 1} ^ {T} \mathrm {E} _ {\mathbf {z} \sim \mathcal {P} _ {j ^ {*}}} [ \ell \left(\mathbf {w} _ {t}; \mathbf {z}\right) ] - \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} q _ {t, i} \mathrm {E} _ {\mathbf {z} \sim \mathcal {P} _ {i}} [ \ell \left(\mathbf {w} _ {t}; \mathbf {z}\right) ] \tag {48} \\ = \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} q _ {t, i} s _ {t, i} - \sum_ {t = 1} ^ {T} s _ {t, j ^ {*}} = \sum_ {t = 1} ^ {T} \left\langle \mathbf {q} _ {t}, \mathbf {s} _ {t} \right\rangle - \sum_ {t = 1} ^ {T} s _ {t, j ^ {*}} \\ \end{array}
$$
where  $j^{*} \in \operatorname{argmax}_{j \in [m]} \sum_{t=1}^{T} R_{j}(\mathbf{w}_{t})$  and the vector  $\mathbf{s}_t \in \mathbb{R}^m$  is defined as
$$
s _ {t, i} \triangleq 1 - \operatorname {E} _ {\mathbf {z} \sim \mathcal {P} _ {i}} [ \ell (\mathbf {w} _ {t}; \mathbf {z}) ] \stackrel {(5)} {\in} [ 0, 1 ], \forall i \in [ m ].
$$
To facilitate the analysis, we introduce a vector  $\hat{\mathbf{s}}_t\in \mathbb{R}^m$  with
$$
\hat {s} _ {t, i} \triangleq 1 - \ell \left(\mathbf {w} _ {t}; \mathbf {z} _ {t} ^ {(i)}\right) \in [ 0, 1 ], \forall i \in [ m ] \tag {49}
$$
where  $\mathbf{z}_t^{(i)}$  denotes a random sample drawn from the  $i$ -th distribution. Note that  $\hat{\mathbf{s}}_t$  is only used for analysis, with the purpose of handling the stochastic rewards. In the algorithm, only  $\hat{s}_{t,i_t} = 1 - \ell (\mathbf{w}_t;\mathbf{z}_t^{(i_t)})$  is observed in the  $t$ -th iteration.
Following the proof of Theorem 1 of Neu [2015], we have
$$
\sum_ {t = 1} ^ {T} \left\langle \mathbf {q} _ {t}, \tilde {\mathbf {s}} _ {t} \right\rangle - \sum_ {t = 1} ^ {T} \tilde {s} _ {t, j ^ {*}} \leq \frac {\ln m}{\eta_ {q}} + \frac {\eta_ {q}}{2} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \tilde {s} _ {t, i} \tag {50}
$$
which makes use of the property of online mirror descent with local norms [Bubeck and Cesa-Bianchi, 2012, Theorem 5.5]. From (5) of Neu [2015], we have
$$
\left\langle \mathbf {q} _ {t}, \tilde {\mathbf {s}} _ {t} \right\rangle = \sum_ {i = 1} ^ {m} q _ {t, i} \tilde {s} _ {t, i} = \hat {s} _ {t, i _ {t}} - \gamma \sum_ {i = 1} ^ {m} \tilde {s} _ {t, i}. \tag {51}
$$
Combining (50) and (51), we have
$$
\sum_ {t = 1} ^ {T} \hat {s} _ {t, i _ {t}} \leq \sum_ {t = 1} ^ {T} \tilde {s} _ {t, j ^ {*}} + \left(\frac {\eta_ {q}}{2} + \gamma\right) \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \tilde {s} _ {t, i} + \frac {\ln m}{\eta_ {q}}. \tag {52}
$$
From (48), we have
$$
\begin{array}{l} \max _ {\mathbf {q} \in \Delta_ {m}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q}) - \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \\ = \sum_ {t = 1} ^ {T} \hat {s} _ {t, i _ {t}} - \sum_ {t = 1} ^ {T} s _ {t, j ^ {*}} + \sum_ {t = 1} ^ {T} \left\langle \mathbf {q} _ {t}, \mathbf {s} _ {t} \right\rangle - \sum_ {t = 1} ^ {T} \hat {s} _ {t, i _ {t}} \tag {53} \\ \stackrel {(5 2)} {\leq} \underbrace {\sum_ {t = 1} ^ {T} \left(\tilde {s} _ {t , j ^ {*}} - s _ {t , j ^ {*}}\right)} _ {:= A} + \underbrace {\left(\frac {\eta_ {q}}{2} + \gamma\right) \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \tilde {s} _ {t , i}} _ {:= B} + \underbrace {\sum_ {t = 1} ^ {T} \left(\langle \mathbf {q} _ {t} , \mathbf {s} _ {t} \rangle - \hat {s} _ {t , i _ {t}}\right)} _ {:= C} + \frac {\ln m}{\eta_ {q}}. \\ \end{array}
$$
We proceed to bound the above three terms  $A$ ,  $B$  and  $C$  respectively.
To bound term  $A$ , we need the following concentration result concerning the IX loss estimates [Neu, 2015, Lemma 1], which we further generalize to the setting with stochastic rewards.
Lemma 2 Let  $\xi_{t,i} \in [0,1]$  for all  $t \in [T]$  and  $i \in [m]$ , and  $\tilde{\xi}_{t,i}$  be its IX-estimator defined as  $\tilde{\xi}_{t,i} = \frac{\hat{\xi}_{t,i}}{p_{t,i} + \gamma_t}\mathbb{I}[i_t = i]$ , where  $\hat{\xi}_{t,i}$  is such that  $\operatorname{E}[\hat{\xi}_{t,i}] = \xi_{t,i}$  with  $\hat{\xi}_{t,i} \in [0,1]$  and the index  $i_t$  is sampled from  $[m]$  according to the distribution  $\mathbf{p}_t \in \Delta_m$ . Let  $\{\gamma_t\}_{t=1}^T$  be a fixed non-increasing sequence with  $\gamma_t \geq 0$  and let  $\alpha_{t,i}$  be non-negative  $\mathcal{F}_{t-1}$ -measurable random variables satisfying  $\alpha_{t,i} \leq 2\gamma_t$  for all  $t \in [T]$  and  $i \in [m]$ . Then, for any  $\delta > 0$ , with probability at least  $1 - \delta$ ,
$$
\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \alpha_ {t, i} \left(\tilde {\xi} _ {t, i} - \xi_ {t, i}\right) \leq \ln \frac {1}{\delta}. \tag {54}
$$
Furthermore, when  $\gamma_t = \gamma \geq 0$  for all  $t \in [T]$ , the following holds with probability at least  $1 - \delta$ ,
$$
\sum_ {t = 1} ^ {T} \left(\tilde {\xi} _ {t, i} - \xi_ {t, i}\right) \leq \frac {1}{2 \gamma} \ln \frac {m}{\delta} \tag {55}
$$
simultaneously for all  $i\in [m]$
Notice that our construction of  $\tilde{\mathbf{s}}_t$  in (17) satisfies that  $\tilde{s}_{t,i} = \frac{\hat{s}_{t,i}}{q_{t,i} + \gamma}\mathbb{I}[i_t = i]$  and  $i_t$  is drawn from  $[m]$  according to  $\mathbf{q}_t\in \Delta_m$  as well as  $\operatorname {E}[\hat{s}_{t,i}] = s_{t,i}$ , which meets the conditions required by Lemma 2. As a result, according to (55), we have
$$
\sum_ {t = 1} ^ {T} (\tilde {s} _ {t, j} - s _ {t, j}) \leq \frac {1}{2 \gamma} \ln \frac {m}{\delta}
$$
for all  $j \in [m]$  (including  $j^*$ ) with probability at least  $1 - \delta$ .
To bound term  $B$ , we can directly use Lemma 1 of Neu [2015], because our setting  $\frac{\eta_q}{2} = \gamma$  satisfies its requirement. Thus, with probability at least  $1 - \delta$ , we have
$$
\left(\frac {\eta_ {q}}{2} + \gamma\right) \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \tilde {s} _ {t, j} \leq \left(\frac {\eta_ {q}}{2} + \gamma\right) \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \hat {s} _ {t, j} + \ln \frac {1}{\delta} \stackrel {(4 9)} {\leq} \left(\frac {\eta_ {q}}{2} + \gamma\right) m T + \ln \frac {1}{\delta}. \tag {56}
$$
We now consider term  $C$  in (53). Let  $V_{t} = \langle \mathbf{q}_{t},\mathbf{s}_{t}\rangle -\hat{s}_{t,i_{t}}$ . Then, it is easy to verify that  $\mathrm{E}_{t - 1}[V_t] = 0$ . So, the process  $\{V_t\}_{t = 1}^T$  forms a martingale difference sequence and it also satisfies  $|V_{t}|\leq 1$  for all  $t$ . Hence, we can apply Lemma 1 and have
$$
\sum_ {t = 1} ^ {T} \left(\langle \mathbf {q} _ {t}, \mathbf {s} _ {t} \rangle - \hat {s} _ {t, i _ {t}}\right) \leq \sqrt {2 T \ln \frac {1}{\delta}} \leq \sqrt {\frac {T}{2}} \left(1 + \ln \frac {1}{\delta}\right).
$$
Combining the three upper bounds for the terms  $A$ ,  $B$  and  $C$ , and further taking the union bound, we have, with probability at least  $1 - \delta$
$$
\begin{array}{l} \max  _ {q \in \Delta_ {m}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q}) - \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \\ \leq \frac {1}{2 \gamma} \ln \frac {3 m}{\delta} + \left(\frac {\eta_ {q}}{2} + \gamma\right) m T + \ln \frac {3}{\delta} + \sqrt {\frac {T}{2}} \left(1 + \ln \frac {3}{\delta}\right) + \frac {\ln m}{\eta_ {q}} \\ = 2 \sqrt {m T \ln m} + \sqrt {\frac {m T}{\ln m}} \cdot \ln \frac {3 m}{\delta} + \sqrt {\frac {T}{2}} + \left(\sqrt {\frac {T}{2}} + 1\right) \ln \frac {3}{\delta} \\ = 3 \sqrt {m T \ln m} + \sqrt {\frac {T}{2}} + \left(\sqrt {\frac {m T}{\ln m}} + \sqrt {\frac {T}{2}} + 1\right) \ln \frac {3}{\delta}, \\ \end{array}
$$
where the third line holds because of our parameter settings  $\gamma = \frac{\eta_q}{2}$  and  $\eta_{q} = \sqrt{\frac{\ln m}{mT}}$ .
Based on this high probability guarantee, we can then obtain the expected regret upper bound using the formula that
$$
\operatorname {E} [ X ] \leq \int_ {0} ^ {1} \frac {1}{\delta} \Pr \left[ X > \ln \frac {1}{\delta} \right] \mathrm {d} \delta
$$
holds for any real-valued random variable  $X$  [Bubeck and Cesa-Bianchi, 2012, § 3.2]. In particular, taking
$$
X = \left(\sqrt {\frac {m T}{\ln m}} + \sqrt {\frac {T}{2}} + 1\right) ^ {- 1} \cdot \left(\max _ {q \in \Delta_ {m}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q}) - \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - 3 \sqrt {m T \ln m} - \sqrt {\frac {T}{2}}\right)
$$
yields  $\operatorname {E}[X]\leq 3$  , which implies that
$$
\operatorname {E} \left[ \max _ {q \in \Delta_ {m}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q}) - \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \right] \leq 3 \sqrt {m T \ln m} + \sqrt {\frac {T}{2}} + 3 \left(\sqrt {\frac {m T}{\ln m}} + \sqrt {\frac {T}{2}} + 1\right).
$$
For the stochastic gradients in (22), their norm can be upper bounded in the same way as (9). That is,
$$
\left\| \mathbf {g} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \right\| _ {w, *} = \left\| \sum_ {i \in C _ {t}} q _ {t, i} \nabla \ell (\mathbf {w} _ {t}; \mathbf {z} _ {t} ^ {(i)}) \right\| _ {w, *} \leq \sum_ {i \in C _ {t}} q _ {t, i} \left\| \nabla \ell (\mathbf {w} _ {t}; \mathbf {z} _ {t} ^ {(i)}) \right\| _ {w, *} \stackrel {(6)} {\leq} \sum_ {i \in C _ {t}} q _ {t, i} G = G,
$$
$$
\left\| \mathbf {g} _ {q} (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \right\| _ {\infty} = \max  _ {i \in C _ {t}} | \ell (\mathbf {w} _ {t}; \mathbf {z} _ {t} ^ {(i)}) | \stackrel {(5)} {\leq} 1.
$$
So, with exactly the same analysis as Theorem 1, we have
$$
\operatorname {E} \left[ \epsilon_ {\varphi} (\bar {\mathbf {w}}, \bar {\mathbf {q}}) \right] \leq 2 \sqrt {\frac {1 0 \left(D ^ {2} G ^ {2} + \ln m\right)}{n _ {1}}}
$$
$$
\epsilon_ {\varphi} (\bar {\mathbf {w}}, \bar {\mathbf {q}}) \leq \left(8 + 2 \ln \frac {2}{\delta}\right) \sqrt {\frac {1 0 \left(D ^ {2} G ^ {2} + \ln m\right)}{n _ {1}}}. \tag {57}
$$
Next, we discuss how to bound the risk of  $\bar{\mathbf{w}}$  on every distribution  $\mathcal{P}_i$ , i.e.,  $R_{i}(\bar{\mathbf{w}})$ . Following the derivation in (8), we know
$$
\max  _ {i \in [ m ]} p _ {i} R _ {i} (\bar {\mathbf {w}}) - \min  _ {\mathbf {w} \in \mathcal {W}} \max  _ {i \in [ m ]} p _ {i} R _ {i} (\mathbf {w}) \leq \epsilon_ {\varphi} (\bar {\mathbf {w}}, \bar {\mathbf {q}}).
$$
Thus, for every distribution  $\mathcal{P}_i$ ,  $R_i(\bar{\mathbf{w}})$  can be bounded in the following way:
$$
R _ {i} (\bar {\mathbf {w}}) \leq \frac {1}{p _ {i}} \min  _ {\mathbf {w} \in \mathcal {W}} \max  _ {i \in [ m ]} p _ {i} R _ {i} (\mathbf {w}) + \frac {1}{p _ {i}} \epsilon_ {\varphi} (\bar {\mathbf {w}}, \bar {\mathbf {q}}).
$$
Taking the high probability bound in (57) as an example, we have with probability at  $1 - \delta$
$$
\begin{array}{l} R _ {i} (\bar {\mathbf {w}}) \leq \frac {1}{p _ {i}} \min  _ {\mathbf {w} \in \mathcal {W}} \max  _ {i \in [ m ]} p _ {i} R _ {i} (\mathbf {w}) + \frac {1}{p _ {i}} \left(8 + 2 \ln \frac {2}{\delta}\right) \sqrt {\frac {1 0 \left(D ^ {2} G ^ {2} + \ln m\right)}{n _ {1}}} \tag {58} \\ = \frac {n _ {1}}{n _ {i}} \min _ {\mathbf {w} \in \mathcal {W}} \max _ {i \in [ m ]} p _ {i} R _ {i} (\mathbf {w}) + \left(8 + 2 \ln \frac {2}{\delta}\right) \frac {\sqrt {1 0 (D ^ {2} G ^ {2} + \ln m) n _ {1}}}{n _ {i}}. \\ \end{array}
$$
B.6 Proof of Theorem 4
We first provide some simple facts that will be used later. From Assumption 3, we immediately know that each risk function  $R_{i}(\cdot)$  also belongs to [0, 1]. As a result, the difference between each risk function and its estimator is well-bounded, i.e., for all  $i \in [m]$ ,
$$
- 1 \leq R _ {i} (\mathbf {w}) - \ell (\mathbf {w}; \mathbf {z}) \leq 1, \forall \mathbf {w} \in \mathcal {W}, \mathbf {z} \sim \mathcal {P} _ {i}. \tag {59}
$$
From Assumption 4, we can prove that each risk function  $R_{i}(\cdot)$  is  $G$ -Lipschitz continuous. To see this, we have
$$
\left\| \nabla R _ {i} (\mathbf {w}) \right\| _ {w, *} = \left\| \mathrm {E} _ {\mathbf {z} \sim \mathcal {P} _ {i}} \nabla \ell (\mathbf {w}; \mathbf {z}) \right\| _ {w, *} \leq \left. \mathrm {E} _ {\mathbf {z} \sim \mathcal {P} _ {i}} \left\| \nabla \ell (\mathbf {w}; \mathbf {z}) \right\| _ {w, *} \right. ^ {(6)} \leq G, \forall \mathbf {w} \in \mathcal {W}, i \in [ m ]. \tag {60}
$$
$$
\left| R _ {i} (\mathbf {w}) - R _ {i} \left(w ^ {\prime}\right) \right| \leq G \| \mathbf {w} - \mathbf {w} ^ {\prime} \| _ {w}, \forall \mathbf {w}, \mathbf {w} ^ {\prime} \in \mathcal {W}, i \in [ m ]. \tag {61}
$$
Furthermore, the difference between the gradient of  $R_{i}(\cdot)$  and its estimator is also well-bounded, i.e., for all  $i\in [m]$ ,
$$
\left\| \nabla R _ {i} (\mathbf {w}) - \nabla \ell (\mathbf {w}; \mathbf {z}) \right\| _ {w, *} \leq \left\| \nabla R _ {i} (\mathbf {w}) \right\| _ {w, *} + \left\| \nabla \ell (\mathbf {w}; \mathbf {z}) \right\| _ {w, *} \stackrel {(6), (6 0)} {\leq} 2 G, \forall \mathbf {w} \in \mathcal {W}, \mathbf {z} \sim \mathcal {P} _ {i}. \tag {62}
$$
Recall the definition of the norm  $\| \cdot \|$  and dual norm  $\| \cdot \|_*$  for the space  $\mathcal{E} \times \mathbb{R}^m$  in (34), and the distance-generating function  $\nu(\cdot)$  in (35). Following the arguments in Section B.1, the two updating rules in (25) and (26) can be merged as
$$
\left[ \mathbf {w} _ {t + 1}; \mathbf {q} _ {t + 1} \right] = \underset {\mathbf {x} \in \mathcal {W} \times \Delta_ {m}} {\operatorname {a r g m i n}} \left\{\eta \left\langle \left[ \mathbf {g} _ {w} \left(\mathbf {w} _ {t} ^ {\prime}, \mathbf {q} _ {t} ^ {\prime}\right); - \mathbf {g} _ {q} \left(\mathbf {w} _ {t} ^ {\prime}, \mathbf {q} _ {t} ^ {\prime}\right) \right], \mathbf {x} - \left[ \mathbf {w} _ {t} ^ {\prime}; \mathbf {q} _ {t} ^ {\prime} \right] \right\rangle + B (\mathbf {x}, \left[ \mathbf {w} _ {t} ^ {\prime}; \mathbf {q} _ {t} ^ {\prime} \right]) \right\} \tag {63}
$$
where  $\eta_w = 2\eta D^2$  and  $\eta_q = 2\eta \ln m$ . Similarly, (28) and (29) are equivalent to
$$
\left[ \mathbf {w} _ {t + 1} ^ {\prime}; \mathbf {q} _ {t + 1} ^ {\prime} \right] = \underset {\mathbf {x} \in \mathcal {W} \times \Delta_ {m}} {\operatorname {a r g m i n}} \left\{\eta \left\langle \left[ \mathbf {g} _ {w} \left(\mathbf {w} _ {t + 1}, \mathbf {q} _ {t + 1}\right); - \mathbf {g} _ {q} \left(\mathbf {w} _ {t + 1}, \mathbf {q} _ {t + 1}\right) \right], \mathbf {x} - \left[ \mathbf {w} _ {t} ^ {\prime}; \mathbf {q} _ {t} ^ {\prime} \right] \right\rangle + B (\mathbf {x}, \left[ \mathbf {w} _ {t} ^ {\prime}; \mathbf {q} _ {t} ^ {\prime} \right]) \right\}. \tag {64}
$$
Let  $F([\mathbf{w}; \mathbf{q}])$  be the monotone operator associated with the weighted GDRO problem in (23), i.e.,
$$
\begin{array}{l} F ([ \mathbf {w}; \mathbf {q} ]) = [ \nabla_ {\mathbf {w}} \varphi (\mathbf {w}, \mathbf {q}); - \nabla_ {\mathbf {q}} \varphi (\mathbf {w}, \mathbf {q}) ] \\ = \left[ \sum_ {i = 1} ^ {m} q _ {i} p _ {i} \nabla R _ {i} (\mathbf {w}); - \left[ p _ {1} R _ {1} (\mathbf {w}), \dots , p _ {m} R _ {m} (\mathbf {w}) \right] ^ {\top} \right]. \tag {65} \\ \end{array}
$$
From our constructions of stochastic gradients in (24) and (27), we clearly have
$$
\begin{array}{l} \mathrm {E} _ {t - 1} \left\{\left[ \mathbf {g} _ {w} \left(\mathbf {w} _ {t} ^ {\prime}, \mathbf {q} _ {t} ^ {\prime}\right); - \mathbf {g} _ {q} \left(\mathbf {w} _ {t} ^ {\prime}, \mathbf {q} _ {t} ^ {\prime}\right) \right] \right\} = F \left(\left[ \mathbf {w} _ {t} ^ {\prime}; \mathbf {q} _ {t} ^ {\prime} \right]\right), \\ \left. \mathrm {E} _ {t - 1} \left\{\left[ \mathbf {g} _ {w} \left(\mathbf {w} _ {t + 1}, \mathbf {q} _ {t + 1}\right); - \mathbf {g} _ {q} \left(\mathbf {w} _ {t + 1}, \mathbf {q} _ {t + 1}\right) \right] \right\} = F \left(\left[ \mathbf {w} _ {t + 1}; \mathbf {q} _ {t + 1} \right]\right). \right. \\ \end{array}
$$
Thus, Algorithm 4 is indeed an instance of SMPA [Juditsky et al., 2011, Algorithm 1], and we can use their Theorem 1 and Corollary 1 to bound the optimization error.
Before applying their results, we show that all the preconditions are satisfied. The parameter  $\Omega$  defined in (16) of Juditsky et al. [2011] can be upper bounded by
$$
\begin{array}{l} \Omega = \sqrt {2 \max  _ {\mathbf {x} \in \mathcal {W} \times \Delta_ {m}} B (\mathbf {x} , [ \mathbf {w} _ {1} ^ {\prime} ; \mathbf {q} _ {1} ^ {\prime} ])} \stackrel {(3 6)} {=} \sqrt {\frac {1}{D ^ {2}} \max  _ {\mathbf {w} \in \mathcal {W}} B _ {w} (\mathbf {w} _ {1} , \mathbf {w} _ {1} ^ {\prime}) + \max  _ {\mathbf {q} \in \Delta_ {m}} \frac {1}{\ln m} B _ {q} (\mathbf {q} , \mathbf {q} _ {1} ^ {\prime})} \\ \stackrel {(4 0)} {\leq} \sqrt {\frac {1}{D ^ {2}} \left(\max  _ {\mathbf {w} \in \mathcal {W}} \nu_ {w} (\mathbf {w}) - \min  _ {\mathbf {w} \in \mathcal {W}} \nu_ {w} (\mathbf {w})\right) + \frac {1}{\ln m} \left(\max  _ {\mathbf {q} \in \Delta_ {m}} \nu_ {q} (\mathbf {q}) - \min  _ {\mathbf {q} \in \Delta_ {m}} \nu_ {q} (\mathbf {q})\right)} \stackrel {(4)} {=} \sqrt {2}. \tag {66} \\ \end{array}
$$
Next, we need to demonstrate that  $F([\mathbf{w}; \mathbf{q}])$  is continuous.
Lemma 3 For the monotone operator  $F([\mathbf{w}; \mathbf{q}])$ , we have
$$
\left\| F ([ \mathbf {w}; \mathbf {q} ]) - F ([ \mathbf {w} ^ {\prime}; \mathbf {q} ^ {\prime} ]) \right\| _ {*} \leq \widetilde {L} \left\| [ \mathbf {w} - \mathbf {w} ^ {\prime}; \mathbf {q} - \mathbf {q} ^ {\prime} ] \right\|
$$
where  $\widetilde{L}$  is defined in (31).
We proceed to show the variance of the stochastic gradients satisfies the light tail condition. To this end, we introduce the stochastic oracle used in Algorithm 4:
$$
\mathbf {g} ([ \mathbf {w}; \mathbf {q} ]) = [ \mathbf {g} _ {w} (\mathbf {w}, \mathbf {q}); - \mathbf {g} _ {q} (\mathbf {w}, \mathbf {q}) ] \tag {67}
$$
$$
\mathbf {g} _ {w} (\mathbf {w}, \mathbf {q}) = \sum_ {i = 1} ^ {m} q _ {i} p _ {i} \left(\frac {n _ {m}}{n _ {i}} \sum_ {j = 1} ^ {n _ {i} / n _ {m}} \nabla \ell (\mathbf {w}; \mathbf {z} ^ {(i, j)})\right),
$$
$$
\mathbf {g} _ {q} (\mathbf {w}, \mathbf {q}) = \left[ p _ {1} \frac {n _ {m}}{n _ {1}} \sum_ {j = 1} ^ {n _ {1} / n _ {m}} \ell (\mathbf {w}; \mathbf {z} ^ {(1, j)}), \ldots , p _ {m} \ell (\mathbf {w}; \mathbf {z} ^ {(m)}) \right] ^ {\top}
$$
and  $\mathbf{z}^{(i,j)}$  is the  $j$ -th sample drawn from distribution  $\mathcal{P}_i$ . The following lemma shows that the variance is indeed sub-Gaussian.
Lemma 4 For the stochastic oracle  $\mathbf{g}([w;q])$ , we have
$$
\left. \right. \operatorname {E} \left[ \exp \left(\frac {\| F ([ \mathbf {w} ; \mathbf {q} ]) - \mathbf {g} ([ \mathbf {w} ; \mathbf {q} ]) \| _ {*} ^ {2}}{\sigma^ {2}}\right)\right] \leq 2
$$
where  $\sigma^2$  is defined in (31).
Based on (66), Lemma 3, and Lemma 4, we can apply the theoretical guarantee of SMPA. Recall that the total number of iterations is  $n_m / 2$  in Algorithm 4. From Corollary 1 of Juditsky et al. [2011], by setting
$$
\eta = \min \left(\frac {1}{\sqrt {3} \widetilde {L}}, \frac {2}{\sqrt {7 \sigma^ {2} n _ {m}}}\right)
$$