SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the GDRO problem, we develop two SA approaches based on SMD and non-oblivious MAB. Both of them attain the nearly optimal  $O(m(\log m) / \epsilon^2)$  sample complexity, but with different number of samples used in each round, which are  $m$  and 1 respectively. Then, we formulate a weighted GDRO problem to handle the scenario in which different distributions have different sample budgets. We first incorporate non-uniform sampling into SMD to satisfy the sample budget in expectation, and deliver distribution-dependent convergence rates. Then, we propose to use mini-batches to meet the budget exactly, deploy SMPA to exploit the small variances, and establish nearly optimal rates for multiple distributions. We have conducted experiments to evaluate our proposed algorithms. The empirical results are presented in Appendix C, and align closely with our theories.
This work was partially supported by the National Key R&D Program of China (2022ZD0114801), NSFC (62122037, 61921006), National Postdoctoral Program for Innovative Talent, China Postdoctoral Science Foundation (2023M731597), and the major key project of PCL (PCL2021A12).
A. Agarwal and T. Zhang. Minimax regret optimization for robust machine learning under distribution shift. In Proceedings of 35th Conference on Learning Theory, pages 2704-2729, 2022.
D. Amodei, S. Ananthanarayanan, R. Anubhai, and et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In Proceedings of the 33rd International Conference on Machine Learning, pages 173-182, 2016.
P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1):48-77, 2002.
B. Becker and R. Kohavi. Adult. UCI Machine Learning Repository, 1996. DOI: https://doi.org/10.24432/C5XW20.
A. Ben-Tal, L. E. Ghaoui, and A. Nemirovski. *Robust Optimization*. Princeton University Press, 2009.
A. Ben-Tal, D. den Hertog, A. D. Waegenaere, B. Melenberg, and G. Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341-357, 2013.
A. Ben-Tal, E. Hazan, T. Koren, and S. Mannor. Oracle-based robust optimization via online learning. Operations Research, 63(3):628-638, 2015.
D. Bertsimas, V. Gupta, and N. Kallus. Robust sample average approximation. Mathematical Programming, 171:217-282, 2018.
A. Blum, N. Haghtalab, A. D. Procaccia, and M. Qiao. Collaborative PAC learning. In Advances in Neural Information Processing Systems 30, pages 2389-2398, 2017.
S. Bubeck and N. Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 5(1):1-122, 2012.
Y. Carmon and D. Hausler. Distributionally robust optimization via ball oracle acceleration. In Advances in Neural Information Processing Systems 35, pages 35866-35879, 2022.
N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
A. Cotter, O. Shamir, N. Srebro, and K. Sridharan. Better mini-batch algorithms via accelerated gradient methods. In Advances in Neural Information Processing Systems 24, pages 1647-1655, 2011.
E. Delage and Y. Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations Research, 58(3):595-612, 2010.
J. C. Duchi and H. Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378 - 1406, 2021.
J. C. Duchi, P. W. Glynn, and H. Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research, 46(3):946-969, 2021.
P. M. Esfahani and D. Kuhn. Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations. *Mathematical Programming*, 171: 115–166, 2018.
V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, second edition, 2000.
R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge University Press, 2018.
J. Wang, R. Gao, and Y. Xie. Sinkhorn distributionally robust optimization. ArXiv e-prints, arXiv:2109.11926, 2021.
Z. Xu, C. Dan, J. Khim, and P. Ravikumar. Class-weighted classification: Trade-offs and robust approaches. In Proceedings of the 37th International Conference on Machine Learning, pages 10544-10554, 2020.
J. Zhang, A. K. Menon, A. Veit, S. Bhojanapalli, S. Kumar, and S. Sra. Coping with label shift via distributionally robust optimisation. In International Conference on Learning Representations, 2021.
L. Zhang, T. Yang, R. Jin, and X. He.  $O(\log T)$  projections for stochastic optimization of smooth and strongly convex functions. In Proceedings of the 30th International Conference on Machine Learning, pages 1121-1129, 2013.
Z.-H. Zhou. A theoretical perspective of machine learning with computational resource concerns. ArXiv e-prints, arXiv:2305.02217, 2023.
Distributionally robust optimization (DRO) stems from the pioneering work of Scarf [1958], and has gained a lot of interest with the advancement of robust optimization [Ben-Tal et al., 2009, 2015]. It has been successfully applied to a variety of machine learning tasks, including adversarial training [Sinha et al., 2018], algorithmic fairness [Hashimoto et al., 2018], class imbalance [Xu et al., 2020], long-tail learning [Samuel and Chechik, 2021], label shift [Zhang et al., 2021], etc.
In general, DRO is formulated to reflect our uncertainty about the target distribution. To ensure good performance under distribution perturbations, it minimizes the risk w.r.t. the worst distribution in an uncertainty set, i.e.,
$$
\min  _ {\mathbf {w} \in \mathcal {W}} \sup  _ {\mathcal {P} \in \mathcal {S} \left(\mathcal {P} _ {0}\right)} \left\{\mathrm {E} _ {\mathbf {z} \sim \mathcal {P}} \left[ \ell (\mathbf {w}; \mathbf {z}) \right] \right\} \tag {33}
$$
where  $S(\mathcal{P}_0)$  denotes a set of probability distributions around  $\mathcal{P}_0$ . In the literature, there mainly exist three ways to construct  $S(\mathcal{P}_0)$ : (i) enforcing moment constraints [Delage and Ye, 2010], (ii) defining a neighborhood around  $\mathcal{P}_0$  by a distance function such as the  $f$ -divergence [Ben-Tal et al., 2013], the Wasserstein distance [Kuhn et al., 2019], and the Sinkhorn distance [Wang et al., 2021], and (iii) hypothesis testing of goodness-of-fit [Bertsimas et al., 2018].
By drawing a set of samples from  $\mathcal{P}_0$ , we can also define an empirical DRO problem, which can be regarded as an SAA approach for solving (33). When the uncertainty set is defined in terms of the Cressie-Read family of  $f$ -divergences, Duchi and Namkoong [2021] have studied finite sample and asymptotic properties of the empirical solution. Besides, it has been proved that empirical DRO can also benefit the risk minimization problem in (1). Namkoong and Duchi [2017] show that empirical DRO with the  $\chi^2$ -divergence has the effect of variance regularization, leading to better generalization w.r.t. distribution  $\mathcal{P}_0$ . Later, Duchi et al. [2021] demonstrate similar behaviors for the  $f$ -divergence constrained neighborhood, and provide one- and two-sided confidence intervals for the minimal risk in (1). Based on the Wasserstein distance, Esfahani and Kuhn [2018] establish an upper confidence bound on the risk of the empirical solution.
Since (33) is more complex than (1), considerable research efforts were devoted to develop efficient algorithms for DRO and its empirical version. For  $\mathcal{P}_0$  with finite support, Ben-Tal et al. [2013, Corollary 3] have demonstrated that (33) with  $f$ -divergences is equivalent to a convex optimization problem, provided that the loss  $\ell(\mathbf{w}; \mathbf{z})$  is convex in  $\mathbf{w}$ . Actually, this conclusion is true even when  $\mathcal{P}_0$  is continuous [Shapiro, 2017, §3.2]. Under mild assumptions, Esfahani and Kuhn [2018] show that DRO problems over Wasserstein balls can be reformulated as finite convex programs—in some cases even as linear programs. Besides the constrained formulation in (33), there also exists a penalized (or regularized) form of DRO [Sinha et al., 2018], which makes the optimization problem more tractable. In the past years, a series of SA methods have been proposed for empirical DRO with convex losses [Namkoong and Duchi, 2016], and DRO with convex loss [Levy et al., 2020] and non-convex losses [Jin et al., 2021, Qi et al., 2021, Rafique et al., 2022].
The main focus of this paper is the GDRO problem in (2)/(3), instead of the traditional DRO in (33). Sagawa et al. [2020] have applied SMD [Nemirovski et al., 2009] to (3), but only obtain a sub-optimal sample complexity of  $O(m^2 (\log m) / \epsilon^2)$ , because of the large variance in their gradients. In the sequel, Haghtalab et al. [2022] and Soma et al. [2022] have tried to improve the sample complexity by reusing samples and applying techniques from MAB respectively, but their analysis suffers dependency issues. Carmon and Hausler [2022, Proposition 2] successfully established an  $O(m(\log m) / \epsilon^2)$  sample complexity by combining SMD and gradient clipping, but their result holds only in expectation. To deal with heterogeneous noise in different distributions, Agarwal and Zhang [2022] propose a variant of GDRO named as minimax regret optimization (MRO), which replaces the risk  $R_{i}(\mathbf{w})$  with "excess risk"  $R_{i}(\mathbf{w}) - \min_{\mathbf{w}\in \mathcal{W}}R_{i}(\mathbf{w})$ . More generally, we can introduce calibration terms in DRO to prevent any single distribution to dominate the maximum [Slowik and Bottou, 2022].
Finally, we note that GDRO has a similar spirit with collaborative PAC learning [Blum et al., 2017, Nguyen and Zakynthinou, 2018, Rothblum and Yona, 2021] in the sense that both aim to find a single model that performs well on multiple distributions.
In this section, we present proofs of main theorems.
The proof is based on Lemma 3.1 and Proposition 3.2 of Nemirovski et al. [2009]. To apply them, we show that their preconditions are satisfied under our assumptions.
Although two instances of SMD are invoked to update  $\mathbf{w}$  and  $\mathbf{q}$  separately, they can be merged as 1 instance by concatenating  $\mathbf{w}$  and  $\mathbf{q}$  as a single variable  $[\mathbf{w};\mathbf{q}]\in \mathcal{W}\times \Delta_{m}$ , and redefine the norm and the distance-generating function [Nemirovski et al., 2009, §3.1]. Let  $\mathcal{E}$  be the space that  $\mathcal{W}$  lies in. We equip the Cartesian product  $\mathcal{E}\times \mathbb{R}^m$  with the following norm and dual norm:
$$
\left\| [ \mathbf {w}; \mathbf {q} ] \right\| = \sqrt {\frac {1}{2 D ^ {2}} \| \mathbf {w} \| _ {w} ^ {2} + \frac {1}{2 \ln m} \| \mathbf {q} \| _ {1} ^ {2}}, \text {a n d} \left\| [ \mathbf {u}; \mathbf {v} ] \right\| _ {*} = \sqrt {2 D ^ {2} \| \mathbf {u} \| _ {w , *} ^ {2} + 2 \| \mathbf {v} \| _ {\infty} ^ {2} \ln m}. \tag {34}
$$
We use the notation  $\mathbf{x} = [\mathbf{w};\mathbf{q}]$ , and equip the set  $\mathcal{W}\times \Delta_m$  with the distance-generating function
$$
\nu (\mathbf {x}) = \nu ([ \mathbf {w}; \mathbf {q} ]) = \frac {1}{2 D ^ {2}} \nu_ {w} (\mathbf {w}) + \frac {1}{2 \ln m} \nu_ {q} (\mathbf {q}). \tag {35}
$$
It is easy to verify that  $\nu (\mathbf{x})$  is 1-strongly convex w.r.t. the norm  $\| \cdot \|$ . Let  $B(\cdot ,\cdot)$  be the Bregman distance associated with  $\nu (\cdot)$ :
$$
\begin{array}{l} B (\mathbf {x}, \mathbf {x} ^ {\prime}) = \nu (\mathbf {x}) - \left[ \nu (\mathbf {x} ^ {\prime}) + \langle \nabla \nu (\mathbf {x} ^ {\prime}), \mathbf {x} - \mathbf {x} ^ {\prime} \rangle \right] \\ = \frac {1}{2 D ^ {2}} \left(\nu_ {w} (\mathbf {w}) - \left[ \nu_ {w} \left(\mathbf {w} ^ {\prime}\right) + \langle \nabla \nu_ {w} \left(\mathbf {w} ^ {\prime}\right), \mathbf {w} - \mathbf {w} ^ {\prime} \rangle \right]\right) \\ + \frac {1}{2 \ln m} \left(\nu_ {q} (\mathbf {q}) - \left[ \nu_ {q} \left(\mathbf {q} ^ {\prime}\right) + \langle \nabla \nu_ {q} \left(\mathbf {q} ^ {\prime}\right), \mathbf {q} - \mathbf {q} ^ {\prime} \rangle \right]\right) \tag {36} \\ = \frac {1}{2 D ^ {2}} B _ {w} (\mathbf {w}, \mathbf {w} ^ {\prime}) + \frac {1}{2 \ln m} B _ {q} (\mathbf {q}, \mathbf {q} ^ {\prime}) \\ \end{array}
$$
where  $\mathbf{x}' = [\mathbf{w}';\mathbf{q}']$
Then, we consider the following version of SMD for updating  $\mathbf{x}_t$ :
$$
\mathbf {x} _ {t + 1} = \underset {\mathbf {x} \in \mathcal {W} \times \Delta_ {m}} {\operatorname {a r g m i n}} \left\{\eta \langle [ \mathbf {g} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}); - \mathbf {g} _ {q} (\mathbf {w} _ {t}, \mathbf {q} _ {t}) ], \mathbf {x} - \mathbf {x} _ {t} \rangle + B (\mathbf {x}, \mathbf {x} _ {t}) \right\} \tag {37}
$$
where  $\eta > 0$  is the step size. In the beginning, we set  $\mathbf{x}_1 = \mathrm{argmin}_{\mathbf{x} \in \mathcal{W} \times \Delta_m} \nu(\mathbf{x}) = [\mathbf{w}_1; \mathbf{q}_1]$ . From the decomposition of the Bregman distance in (36), we observe that (37) is equivalent to (11) and (12) by setting
$$
\eta_ {w} = 2 \eta D ^ {2}, \text {a n d} \eta_ {q} = 2 \eta \ln m.
$$
Next, we show that the stochastic gradients are well-bounded. Under our assumptions, we have
$$
\left\| \mathbf {g} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \right\| _ {w, *} = \left\| \sum_ {i = 1} ^ {m} q _ {t, i} \nabla \ell (\mathbf {w} _ {t}; \mathbf {z} _ {t} ^ {(i)}) \right\| _ {w, *} \leq \sum_ {i = 1} ^ {m} q _ {t, i} \left\| \nabla \ell (\mathbf {w} _ {t}; \mathbf {z} _ {t} ^ {(i)}) \right\| _ {w, *} \stackrel {(6)} {\leq} \sum_ {i = 1} ^ {m} q _ {t, i} G = G,
$$
$$
\left\| \mathbf {g} _ {q} (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \right\| _ {\infty} = \left\| \left[ \ell (\mathbf {w} _ {t}; \mathbf {z} _ {t} ^ {(1)}), \dots , \ell (\mathbf {w} _ {t}; \mathbf {z} _ {t} ^ {(m)}) \right] ^ {\top} \right\| _ {\infty} \overset {(5)} {\leq} 1.
$$
As a result, the concatenated gradients used in (37) is also bounded in term of the dual norm  $\| \cdot \|_{*}$ :
$$
\begin{array}{l} \left\| \left[ \mathbf {g} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}); - \mathbf {g} _ {q} (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \right] \right\| _ {*} = \sqrt {2 D ^ {2} \| \mathbf {g} _ {w} (\mathbf {w} _ {t} , \mathbf {q} _ {t}) \| _ {w , *} ^ {2} + 2 \| \mathbf {g} _ {q} (\mathbf {w} _ {t} , \mathbf {q} _ {t}) \| _ {\infty} ^ {2} \ln m} \\ \leq \underbrace {\sqrt {2 D ^ {2} G ^ {2} + 2 \ln m}} _ {:= M}. \\ \end{array}
$$
Now, we are ready to state our theoretical guarantees. By setting
$$
\eta = \frac {2}{M \sqrt {5 T}} = \sqrt {\frac {2}{5 T (D ^ {2} G ^ {2} + \ln m)}},
$$
Lemma 3.1 and (3.13) of Nemirovski et al. [2009] imply that
$$
\operatorname {E} \left[ \epsilon_ {\phi} (\bar {\mathbf {w}}, \bar {\mathbf {q}}) \right] \leq 2 M \sqrt {\frac {5}{T}} = 2 \sqrt {\frac {1 0 (D ^ {2} G ^ {2} + \ln m)}{T}}
$$
Furthermore, from Proposition 3.2 of Nemirovski et al. [2009], we have, for any  $\Omega >1$
$$
\operatorname * {P r} \left[ \epsilon_ {\phi} (\bar {\mathbf {w}}, \bar {\mathbf {q}}) \geq (8 + 2 \Omega) M \sqrt {\frac {5}{T}} = (8 + 2 \Omega) \sqrt {\frac {1 0 (D ^ {2} G ^ {2} + \ln m)}{T}} \right] \leq 2 \exp (- \Omega).
$$
We complete the proof by setting  $\delta = 2\exp (-\Omega)$
We first bound the regret of the 1st player. In the analysis, we address the non-obliviousness by the "ghost iterate" technique of Nemirovski et al. [2009].
Theorem 5 Under Assumptions 1, 2 and 4, by setting  $\eta_w = \frac{2D}{G\sqrt{5T}}$ , we have
$$
\mathrm {E} \left[ \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \min  _ {\mathbf {w} \in \mathcal {W}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w}, \mathbf {q} _ {t}) \right] \leq 2 D G \sqrt {5 T}
$$
and with probability at least  $1 - \delta$
$$
\sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \min  _ {\mathbf {w} \in \mathcal {W}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w}, \mathbf {q} _ {t}) \leq D G \sqrt {T} \left(2 \sqrt {5} + 8 \sqrt {\ln \frac {1}{\delta}}\right).
$$
By extending Exp3-IX to stochastic rewards, we have the following bound for the 2nd player.
Theorem 6 Under Assumption 3, by setting  $\eta_q = \sqrt{\frac{\ln m}{mT}}$  and the IX coefficient  $\gamma = \frac{\eta_q}{2}$ , we have
$$
\mathrm {E} \left[ \max  _ {q \in \Delta_ {m}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q}) - \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \right] \leq 3 \sqrt {m T \ln m} + \sqrt {\frac {T}{2}} + 3 \left(\sqrt {\frac {m T}{\ln m}} + \sqrt {\frac {T}{2}} + 1\right)
$$
$$
\max  _ {q \in \Delta_ {m}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q}) - \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \leq 3 \sqrt {m T \ln m} + \sqrt {\frac {T}{2}} + \left(\sqrt {\frac {m T}{\ln m}} + \sqrt {\frac {T}{2}} + 1\right) \ln \frac {3}{\delta}.
$$
From Jensen's inequality and the outputs  $\bar{\mathbf{w}} = \frac{1}{T}\sum_{t=1}^{T}\mathbf{w}_t$  and  $\bar{\mathbf{q}} = \frac{1}{T}\sum_{t=1}^{T}\mathbf{q}_t$ , we have
$$
\begin{array}{l} \epsilon_ {\phi} (\bar {\mathbf {w}}, \bar {\mathbf {q}}) = \max  _ {\mathbf {q} \in \Delta_ {m}} \phi (\bar {\mathbf {w}}, \mathbf {q}) - \min  _ {\mathbf {w} \in \mathcal {W}} \phi (\mathbf {w}, \bar {\mathbf {q}}) \\ \leq \frac {1}{T} \left(\max  _ {\mathbf {q} \in \Delta_ {m}} \sum_ {t = 1} ^ {T} \phi \left(\mathbf {w} _ {t}, \mathbf {q}\right) - \min  _ {\mathbf {w} \in \mathcal {W}} \sum_ {t = 1} ^ {T} \phi \left(\mathbf {w}, \mathbf {q} _ {t}\right)\right) \tag {38} \\ = \frac {1}{T} \left(\max  _ {\mathbf {q} \in \Delta_ {m}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q}) - \sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t})\right) + \frac {1}{T} \left(\sum_ {t = 1} ^ {T} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \min  _ {\mathbf {w} \in \mathcal {W}} \sum_ {t = 1} ^ {T} \phi (\mathbf {w}, \mathbf {q} _ {t})\right). \\ \end{array}
$$
We obtain (19) by substituting the high probability bound in Theorems 5 and 6 into (38), and taking the union bound.
Our goal is to analyze SMD for non-oblivious OCO with stochastic gradients. In the literature, we did not find a convenient reference for it. A very close one is the Lemma 3.2 of Flaxman et al. [2005], which bounds the expected regret of SGD for non-oblivious OCO. But it is insufficient for our purpose, so we provide our proof by following the analysis of SMD for stochastic convex-concave
optimization [Nemirovski et al., 2009, §3]. Notice that we cannot use the theoretical guarantee of SMD for SCO [Nemirovski et al., 2009, §2.3], because the objective function is fixed in SCO.
From the standard analysis of mirror descent, e.g., Lemma 2.1 of Nemirovski et al. [2009], we have
$$
\langle \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}), \mathbf {w} _ {t} - \mathbf {w} \rangle \leq \frac {B _ {w} (\mathbf {w} , \mathbf {w} _ {t}) - B _ {w} (\mathbf {w} , \mathbf {w} _ {t + 1})}{\eta_ {w}} + \frac {\eta_ {w}}{2} \| \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \| _ {w, *} ^ {2}.
$$
Summing the above inequality over  $t = 1,\ldots ,T$  , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left\langle \tilde {\mathbf {g}} _ {w} \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right), \mathbf {w} _ {t} - \mathbf {w} \right\rangle \leq \frac {B _ {w} \left(\mathbf {w} , \mathbf {w} _ {1}\right)}{\eta_ {w}} + \frac {\eta_ {w}}{2} \sum_ {t = 1} ^ {T} \| \tilde {\mathbf {g}} _ {w} \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right) \| _ {w, *} ^ {2} \tag {39} \\ \stackrel {(6), (1 0)} {\leq} \frac {B _ {w} (\mathbf {w} , \mathbf {w} _ {1})}{\eta_ {w}} + \frac {\eta_ {w} T G ^ {2}}{2} \leq \frac {D ^ {2}}{\eta_ {w}} + \frac {\eta_ {w} T G ^ {2}}{2} \\ \end{array}
$$
where the last step is due to [Nemirovski et al., 2009, (2.42)]
$$
\max  _ {\mathbf {w} \in \mathcal {W}} B _ {w} (\mathbf {w}, \mathbf {w} _ {1}) \leq \max  _ {\mathbf {w} \in \mathcal {W}} \nu_ {w} (\mathbf {w}) - \min  _ {\mathbf {w} \in \mathcal {W}} \nu_ {w} (\mathbf {w}) \overset {(4)} {\leq} D ^ {2}. \tag {40}
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left[ \phi \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right) - \phi \left(\mathbf {w}, \mathbf {q} _ {t}\right) \right] \leq \sum_ {t = 1} ^ {T} \left\langle \nabla_ {\mathbf {w}} \phi \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right), \mathbf {w} _ {t} - \mathbf {w} \right\rangle \\ = \sum_ {t = 1} ^ {T} \langle \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}), \mathbf {w} _ {t} - \mathbf {w} \rangle + \sum_ {t = 1} ^ {T} \langle \nabla_ {\mathbf {w}} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}), \mathbf {w} _ {t} - \mathbf {w} \rangle \\ \stackrel {(3 9)} {\leq} \frac {D ^ {2}}{\eta_ {w}} + \frac {\eta_ {w} T G ^ {2}}{2} + \sum_ {t = 1} ^ {T} \langle \nabla_ {\mathbf {w}} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}), \mathbf {w} _ {t} - \mathbf {w} \rangle . \\ \end{array}
$$
Maximizing each side over  $\mathbf{w} \in \mathcal{W}$ , we arrive at
$$
\begin{array}{l} \max  _ {\mathbf {w} \in \mathcal {W}} \sum_ {t = 1} ^ {T} \left[ \phi \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right) - \phi (\mathbf {w}, \mathbf {q} _ {t}) \right] = \sum_ {t = 1} ^ {T} \phi \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right) - \min  _ {\mathbf {w} \in \mathcal {W}} \sum_ {t = 1} ^ {T} \phi \left(\mathbf {w}, \mathbf {q} _ {t}\right) \\ \leq \frac {D ^ {2}}{\eta_ {w}} + \frac {\eta_ {w} T G ^ {2}}{2} + \max  _ {\mathbf {w} \in \mathcal {W}} \left\{\underbrace {\sum_ {t = 1} ^ {T} \langle \nabla_ {\mathbf {w}} \phi (\mathbf {w} _ {t} , \mathbf {q} _ {t}) - \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t} , \mathbf {q} _ {t}) , \mathbf {w} _ {t} - \mathbf {w} \rangle} _ {:= F (\mathbf {w})} \right\}. \tag {41} \\ \end{array}
$$
Next, we bound the last term in (41), i.e.,  $\max_{\mathbf{w} \in \mathcal{W}} F(\mathbf{w})$ . Because  $\mathrm{E}_{t-1}[\tilde{\mathbf{g}}_w(\mathbf{w}_t, \mathbf{q}_t)] = \nabla_{\mathbf{w}} \phi(\mathbf{w}_t, \mathbf{q}_t)$ ,  $F(\mathbf{w})$  is the sum of a martingale difference sequence for any fixed  $\mathbf{w}$ . However, it is not true for  $\widetilde{\mathbf{w}} = \operatorname{argmax}_{\mathbf{w} \in \mathcal{W}} F(\mathbf{w})$ , because  $\widetilde{\mathbf{w}}$  depends on the randomness of the algorithm. Thus, we cannot directly apply techniques for martingales to bounding  $\max_{\mathbf{w} \in \mathcal{W}} F(\mathbf{w})$ . This is the place where the analysis differs from that of SCO.
To handle the above challenge, we introduce a virtual sequence of variables to decouple the dependency [Nemirovski et al., 2009, proof of Lemma 3.1]. Imagine there is an online algorithm which performs SMD by using  $\nabla_{\mathbf{w}}\phi (\mathbf{w}_t,\mathbf{q}_t) - \tilde{\mathbf{g}}_w(\mathbf{w}_t,\mathbf{q}_t)$  as the gradient:
$$
\mathbf {v} _ {t + 1} = \underset {\mathbf {w} \in \mathcal {W}} {\operatorname {a r g m i n}} \left\{\eta_ {w} \left\langle \nabla_ {\mathbf {w}} \phi \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right) - \tilde {\mathbf {g}} _ {w} \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right), \mathbf {w} - \mathbf {v} _ {t} \right\rangle + B _ {w} (\mathbf {w}, \mathbf {v} _ {t}) \right\} \tag {42}
$$
where  $\mathbf{v}_1 = \mathbf{w}_1$ . By repeating the derivation of (39), we can show that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left\langle \nabla_ {\mathbf {w}} \phi \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right) - \tilde {\mathbf {g}} _ {w} \left(\mathbf {w} _ {t}, \mathbf {q} _ {t}\right), \mathbf {v} _ {t} - \mathbf {w} \right\rangle \tag {43} \\ \leq \frac {B _ {w} (\mathbf {w} , \mathbf {w} _ {1})}{\eta_ {w}} + \frac {\eta_ {w}}{2} \sum_ {t = 1} ^ {T} \| \nabla_ {\mathbf {w}} \phi (\mathbf {w} _ {t}, \mathbf {q} _ {t}) - \tilde {\mathbf {g}} _ {w} (\mathbf {w} _ {t}, \mathbf {q} _ {t}) \| _ {w, *} ^ {2} \leq \frac {D ^ {2}}{\eta_ {w}} + 2 \eta_ {w} T G ^ {2} \\ \end{array}
$$
where in the last inequality, we make use of (40) and