SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The efficient version is concluded in Algorithm 1 with Line 10-11, which uses only one gradient  $\nabla f_t(\mathbf{x}_t)$ . The only algorithmic modification is that base learners update on the carefully designed surrogate functions, not the original one. We provide the regret guarantee below, which achieves the same guarantees as Theorem 1 with only one gradient per round. The proof is in Appendix C.2.
Theorem 2. Under Assumptions 1 and 2, efficient Algorithm 1 enjoys  $\mathcal{O}(\log V_T)$ ,  $\mathcal{O}(d\log V_T)$  and  $\hat{\mathcal{O}} (\sqrt{V_T})$  for strongly convex, exp-concave and convex functions, using only one gradient per round.
As a byproduct, we show that this idea can be used to recover the optimal worst-case universal guarantees using one gradient with a simpler approach and analysis, with proof in Appendix C.1.
Proposition 1. Under Assumption 1, using the above surrogate loss functions for base learners, and running ADAPT-ML-PROD as the meta learner guarantees  $\mathcal{O}(\log T)$ ,  $\mathcal{O}(d\log T)$  and  $\mathcal{O}(\sqrt{T})$  regret bounds for strongly convex, exp-concave and convex functions, using one gradient per round.
In this paper, we obtain universal gradient-variation guarantees via a multi-layer online ensemble approach. We first propose a novel optimism design to unify various kinds of functions. Then we analyze the negative terms of the meta algorithm MsMwC and inject cascaded correction terms to improve the algorithmic stability to realize effective cancellations in the multi-layer structure. Furthermore, we provide a novel regret decomposition combined with carefully designed surrogate functions to achieve one gradient query per round. Finally, we deploy the our approach into two applications, including the stochastically extended adversarial (SEA) model and two-player zero-sum games, to validate its effectiveness, and obtain best known universal guarantees therein. Due to page limits, the applications are deferred to Appendix A. Two byproducts rise in our work. The first one is negative stability terms in the analysis of MsMwC. And the second one contains a simple approach and analysis for the optimal worst-case universal guarantees, using one gradient per round.
An important open problem lies in optimality and efficiency. In the convex case, our results still exhibit an  $\mathcal{O}(\log V_T)$  gap from the optimal  $\mathcal{O}(\sqrt{V_T})$  result. Moreover, our algorithm necessitates  $\mathcal{O}(\log^2 T)$  base learners, as opposed to  $\mathcal{O}(\log T)$  base learners in two-layer structures. Whether it is possible to achieve the optimal results for all kinds of functions (strongly convex, exp-concave, convex) using a two-layer algorithm remains as an important problem for future investigation.
This research was supported by National Key R&D Program of China (2022ZD0114800), NSFC (62206125, 61921006), and JiangsuSF (BK20220776). The authors would thank Reviewer #ZhYs of OpenReview for the thorough review and insightful suggestions.
N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
L. Chen, H. Luo, and C. Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Proceedings of the 34th Annual Conference Computational Learning Theory (COLT), pages 1216-1259, 2021.
S. Chen, W.-W. Tu, P. Zhao, and L. Zhang. Optimistic online mirror descent for bridging stochastic and adversarial online convex optimization. In Proceedings of the 40th International Conference on Machine Learning (ICML), pages 5002-5035, 2023a.
S. Chen, Y.-J. Zhang, W.-W. Tu, P. Zhao, and L. Zhang. Optimistic online mirror descent for bridging stochastic and adversarial online convex optimization. ArXiv preprint, arXiv:2302.04552, 2023b.
C. Chiang, T. Yang, C. Lee, M. Mahdavi, C. Lu, R. Jin, and S. Zhu. Online optimization with gradual variations. In Proceedings of the 25th Annual Conference Computational Learning Theory (COLT), pages 6.1-6.20, 2012.
P. Gaillard, G. Stoltz, and T. van Erven. A second-order bound with excess losses. In Proceedings of the 27th Annual Conference Computational Learning Theory (COLT), pages 176-196, 2014.
E. Hazan. Introduction to Online Convex Optimization. Foundations and Trends in Optimization, 2 (3-4):157-325, 2016.
E. Hazan, A. Agarwal, and S. Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning, 69(2-3):169-192, 2007.
S. Ji and J. Ye. An accelerated gradient method for trace norm minimization. In Proceedings of the 26th International Conference on Machine Learning (ICML), pages 457-464, 2009.
M. Li, T. Zhang, Y. Chen, and A. J. Smola. Efficient mini-batch training for stochastic optimization. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), pages 661-670, 2014.
F. Orabona. A modern introduction to online learning. ArXiv preprint, arXiv:1912.13213, 2019.
F. Orabona, N. Cesa-Bianchi, and C. Gentile. Beyond logarithmic bounds in online learning. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 823-831, 2012.
M. S. Pinsker. Information and information stability of random variables and processes. Holden-Day, 1964.
R. Pogodin and T. Lattimore. On first-order bounds, variance and gap-dependent bounds for adversarial bandits. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence (UAI), pages 894-904, 2019.
A. Rakhlin and K. Sridharan. Online learning with predictable sequences. In Proceedings of the 26th Annual Conference Computational Learning Theory (COLT), pages 993-1019, 2013a.
A. Rakhlin and K. Sridharan. Optimization, learning, and games with predictable sequences. In Advances in Neural Information Processing Systems 26 (NIPS), pages 3066-3074, 2013b.
S. Sachs, H. Hadiji, T. van Erven, and C. Guzmán. Between stochastic and adversarial online convex optimization: Improved regret bounds via smoothness. In Advances in Neural Information Processing Systems 35 (NeurIPS), pages 691-702, 2022.
S. Sachs, H. Hadiji, T. van Erven, and C. Guzmán. Accelerated rates between stochastic and adversarial online convex optimization. ArXiv preprint, arXiv:2303.03272, 2023.
N. Srebro, K. Sridharan, and A. Tewari. Smoothness, low noise and fast rates. In Advances in Neural Information Processing Systems 23 (NIPS), pages 2199-2207, 2010.
V. Syrgkanis, A. Agarwal, H. Luo, and R. E. Schapire. Fast convergence of regularized learning in games. In Advances in Neural Information Processing Systems 28 (NIPS), pages 2989-2997, 2015.
T. van Erven and W. M. Koolen. Metagrad: Multiple learning rates in online learning. In Advances in Neural Information Processing Systems 29 (NIPS), pages 3666-3674, 2016.
G. Wang, D. Zhao, and L. Zhang. Minimizing adaptive regret with one gradient per iteration. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), pages 2762-2768, 2018.
G. Wang, S. Lu, and L. Zhang. Adaptivity and optimality: A universal algorithm for online convex optimization. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence (UAI), pages 659-668, 2019.
C. Wei, Y. Hong, and C. Lu. Tracking the best expert in non-stationary stochastic environments. In Advances in Neural Information Processing Systems 29 (NIPS), pages 3972-3980, 2016.
L. Zhang, S. Lu, and Z.-H. Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems 31 (NeurIPS), pages 1330-1340, 2018.
L. Zhang, G. Wang, J. Yi, and T. Yang. A simple yet universal strategy for online convex optimization. In Proceedings of the 39th International Conference on Machine Learning (ICML), pages 26605-26623, 2022a.
M. Zhang, P. Zhao, H. Luo, and Z.-H. Zhou. No-regret learning in time-varying zero-sum games. In Proceedings of the 39th International Conference on Machine Learning (ICML), pages 26772-26808, 2022b.
P. Zhao, Y.-J. Zhang, L. Zhang, and Z.-H. Zhou. Dynamic regret of convex and smooth functions. In Advances in Neural Information Processing Systems 33 (NeurIPS), pages 12510-12520, 2020.
P. Zhao, Y.-J. Zhang, L. Zhang, and Z.-H. Zhou. Adaptivity and non-stationarity: Problem-dependent dynamic regret for online convex optimization. ArXiv preprint, arXiv:2112.14368, 2021.
M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML), pages 928-936, 2003.
A Applications
In this section, we validate the importance and the practical potential of our approach by applying it in two problems, including stochastically extended adversarial (SEA) model in Appendix A.1 and two-player zero-sum games in Appendix A.2.
A.1 Application I: Stochastically Extended Adversarial Model
Stochastically extended adversarial (SEA) model [Sachs et al., 2022] serves as an interpolation between stochastic and adversarial online convex optimization, where the environments choose the loss function  $f_{t}$  from a distribution  $\mathcal{D}_t$  (i.e.,  $f_{t} \sim \mathcal{D}_{t}$ ). They proposed cumulative stochastic variance  $\sigma_{1:T}^{2}$ , the gap between  $f_{t}$  and its expected version  $F_{t}(\cdot) \triangleq \mathbb{E}_{f_{t} \sim \mathcal{D}_{t}}[f_{t}(\cdot)]$ , and cumulative adversarial variation  $\Sigma_{1:T}^{2}$ , the gap between  $F_{t}(\cdot)$  and  $F_{t-1}(\cdot)$ , to bound the expected regret. Formally,
$$
\sigma_ {1: T} ^ {2} \triangleq \sum_ {t = 1} ^ {T} \max _ {\mathbf {x} \in \mathcal {X}} \mathbb {E} _ {f _ {t} \sim \mathcal {D} _ {t}} \left[ \| \nabla f _ {t} (\mathbf {x}) - \nabla F _ {t} (\mathbf {x}) \| ^ {2} \right], \Sigma_ {1: T} ^ {2} \triangleq \mathbb {E} \left[ \sum_ {t = 2} ^ {T} \sup _ {\mathbf {x} \in \mathcal {X}} \| \nabla F _ {t} (\mathbf {x}) - \nabla F _ {t - 1} (\mathbf {x}) \| ^ {2} \right].
$$
By specializing different  $\sigma_{1:T}^{2}$  and  $\Sigma_{1:T}^{2}$ , the SEA model can recover the adversarial and stochastic OCO setup, respectively. Specifically, setting  $\sigma_{1:T}^{2} = 0$  recovers the adversarial setup, where  $\Sigma_{1:T}^{2}$  equals to the gradient variation  $V_{T}$ . Besides, choosing  $\Sigma_{1:T}^{2} = 0$  recovers the stochastic setup, where  $\sigma_{1:T}^{2}$  stands for the variance in stochastic optimization.
For smooth expected function  $F_{t}(\cdot)$ , Sachs et al. [2022] obtained an optimal  $\mathcal{O}(\sqrt{\sigma_{1:T}^{2} + \Sigma_{1:T}^{2}})$  expected regret for convex functions and an  $\mathcal{O}((\sigma_{\max}^2 + \Sigma_{\max}^2)\log T)$  regret for strongly convex functions, where  $\sigma_{\max}^2 \triangleq \max_{t \in [T]} \max_{\mathbf{x} \in \mathcal{X}} \mathbb{E}_{f_t \sim \mathcal{D}_t}[\|\nabla f_t(\mathbf{x}) - \nabla F_t(\mathbf{x})\|^2]$  and  $\Sigma_{\max}^2 \triangleq \max_{t \in [T]} \sup_{\mathbf{x} \in \mathcal{X}}\| \nabla F_t(\mathbf{x}) - \nabla F_{t-1}(\mathbf{x})\|^2$ . Later, the results are improved by Chen et al. [2023b], where the authors achieved a better  $\mathcal{O}((\sigma_{\max}^2 + \Sigma_{\max}^2)\log (\sigma_{1:T}^2 + \Sigma_{1:T}^2))$  regret for strongly convex functions and a new  $\mathcal{O}(d\log (\sigma_{1:T}^2 + \Sigma_{1:T}^2))$  regret for exp-concave functions, while maintaining the optimal guarantee for convex functions. Their key observation is that the gradient variation in optimistic algorithms encodes the information of the cumulative stochastic variance  $\sigma_{1:T}^2$  and cumulative adversarial variation  $\Sigma_{1:T}^2$ , restated in Lemma 5.
A major open problem remains in the work of Chen et al. [2023b] that whether it is possible to get rid of different parameter configurations and obtain universal guarantees in the problem. In the following, we show that our approach can be directly applied and achieves almost the same guarantees as those in Chen et al. [2023b], up to a logarithmic factor in leading term. Notably, our algorithm does not require different parameter setups, thus resolving the open problem proposed by the authors. We conclude our results in Theorem 3 and the proof can be found in Appendix D.1.
Theorem 3. Under the same assumptions as Chen et al. [2023b], the efficient version of Algorithm 1 obtains  $\mathcal{O}((\sigma_{\max}^2 + \Sigma_{\max}^2) \log (\sigma_{1:T}^2 + \Sigma_{1:T}^2))$  regret for strongly convex functions,  $\mathcal{O}(d \log (\sigma_{1:T}^2 + \Sigma_{1:T}^2))$  regret for exp-concave functions and  $\widehat{\mathcal{O}}(\sqrt{\sigma_{1:T}^2 + \Sigma_{1:T}^2})$  regret for convex functions, where the  $\widehat{\mathcal{O}}(\cdot)$ -notation omits logarithmic factors in leading terms.
Note that Sachs et al. [2023] also considered the problem of universal learning and obtained an  $\mathcal{O}(\sqrt{T\log T})$  regret for convex functions and an  $\mathcal{O}((\sigma_{\max}^2 + \Sigma_{\max}^2 + D^2 L^2)\log^2 T)$  regret for strongly convex functions simultaneously. We conclude the existing results in Table 3. Our results are better than theirs in two aspects: (i) for strongly convex and convex functions, our guarantees are adaptive with the problem-dependent quantities  $\sigma_{1:T}$  and  $\Sigma_{1:T}$  while theirs depend on the time horizon  $T$ ; and (ii) our algorithm achieves an additional guarantee for exp-concave functions.
A.2 Application II: Two-player Zero-sum Games
Multi-player online games [Cesa-Bianchi and Lugosi, 2006] is a versatile model that depicts the interaction of multiple players over time. Since each player is facing similar players like herself, the theoretical guarantees, e.g., the summation of all players' regret, can be better than the minimax optimal  $\sqrt{T}$  bound in adversarial environments, thus achieving fast rates.
This bound improves the result of  $\mathcal{O}\big(\min \{G^2\log (\sigma_{1:T}^2 +\Sigma_{1:T}^2),(\sigma_{\max}^2 +\Sigma_{\max}^2)\log T\} \big)$  in their conference version [Chen et al., 2023a].
Table 3: Comparisons of our results with existing ones. The second column presents the regret bounds for various kinds of loss functions, where  $\sigma_{1:T}^2$  and  $\Sigma_{1:T}^2$  are the cumulative stochastic variance and adversarial variation, which are at most  $\mathcal{O}(T)$  and can be much smaller in benign environments. The  $\widehat{\mathcal{O}} (\cdot)$ -notation omits logarithmic factors on leading terms. The third column indicates whether the results for different kinds of functions are achieved by a single algorithm.
Works	Regret Bounds	Single Algorithm?
Strongly Convex	Exp-concave	Convex
Sachs et al. [2022]	O((σ2max+Σ2max) log T)	X	O(√σ1:T+Σ1:T)	No
Chen et al. [2023b]	O((σ2max+Σ2max) log(σ1:T+Σ1:T))	O(d log(σ1:T+Σ1:T))	O(√σ1:T+Σ1:T)	No
Sachs et al. [2023]	O((σ2max+Σ2max+D2L2) log2T)	X	O(√T)	Yes
Ours	O((σ2max+Σ2max) log(σ1:T+Σ1:T))	O(d log(σ1:T+Σ1:T))	O(√σ1:T+Σ1:T)	Yes
Table 4: Comparisons of our results with existing ones. In the honest case, the results are measured by the summation of all players' regret and in the dishonest case, the results are in terms of the individual regret of each player. Bilinear and strongly convex-concave games are considered inside each case.  $\bullet$  denotes the best result in each case (row).
Games	Syrgkanis et al. [2015]	Zhang et al. [2022a]	Ours
Honest	bilinear	O(1) •	O(√T)	O(1) •
strongly convex-concave	O(1) •	O(log T)	O(1) •
Dishonest	bilinear	O(√T) •	O(√T) •	O(√T) •
strongly convex-concave	O(√T)	O(log T) •	O(log T) •
The pioneering works of Rakhlin and Sridharan [2013b], Syrgkanis et al. [2015] investigated optimistic algorithms in multi-player online games and illuminated the importance of the gradient variation. Specifically, Syrgkanis et al. [2015] proved that when each player runs an optimistic algorithm (optimistic OMD or optimistic follow the regularized leader), the summation of the regret, which serves as an upper bound for some performance measures in games, can be bounded by  $\mathcal{O}(1)$ . The above results assume that the players are honest, i.e., they agree to run the same algorithm distributedly. In the dishonest case, where there exist players who do not follow the agreed protocol, the guarantees will degenerate to the minimax result of  $\mathcal{O}(\sqrt{T})$ .
In this part, we consider the simple two-player zero-sum games as an example to validate the effectiveness of our proposed algorithm. The game can be formulated as a min-max optimization problem of  $\min_{\mathbf{x} \in \mathcal{X}} \max_{\mathbf{y} \in \mathcal{Y}} f(\mathbf{x}, \mathbf{y})$ . We consider the case that the game is either bilinear (i.e.,  $f(\mathbf{x}, \mathbf{y}) = \mathbf{x}^{\top} A \mathbf{y}$ ), or strongly convex-concave (i.e.,  $f(\mathbf{x}, \mathbf{y})$  is  $\lambda$ -strongly convex in  $\mathbf{x}$  and  $\lambda$ -strongly concave in  $\mathbf{y}$ ). Our algorithm can guarantee the regret summation in the honest case and the individual regret of each player in the dishonest case, without knowing the type of games in advance. We conclude our results in Theorem 4, and the proof can be found in Appendix D.2.
Theorem 4. Under Assumptions 1 and 2, for bilinear and strongly convex-concave games, the efficient version of Algorithm 1 enjoys  $\mathcal{O}(1)$  regret summation in the honest case,  $\widehat{\mathcal{O}}(\sqrt{T})$  and  $\mathcal{O}(\log T)$  bounds in the dishonest case, where the  $\widehat{\mathcal{O}}(\cdot)$ -notation omits logarithmic factors in leading terms.
Table 4 compares our approach with Syrgkanis et al. [2015], Zhang et al. [2022a]. Specifically, ours is better than Syrgkanis et al. [2015] in the strongly convex-concave games in the dishonest case due to its universality, and better than Zhang et al. [2022a] in the honest case since our approach enjoys gradient-variation bounds that are essential in achieving fast rates for regret summation.
In this section, we provide proofs for Section 3, including Lemma 1, Lemma 2 and Lemma 3. For simplicity, we introduce the following notations denoting the stability of the final and intermediate decisions of the algorithm. Specifically, for any  $k \in [K], i \in [N]$ , we define
$$
S _ {T} ^ {\mathbf {x}} \triangleq \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2}, S _ {T, k} ^ {\mathbf {x}} \triangleq \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t, k} - \mathbf {x} _ {t - 1, k} \| ^ {2}, S _ {T, k, i} ^ {\mathbf {x}} \triangleq \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t, k, i} - \mathbf {x} _ {t - 1, k, i} \| ^ {2},
$$
$$
S _ {T} ^ {\boldsymbol {q}} \triangleq \sum_ {t = 2} ^ {T} \| \boldsymbol {q} _ {t} - \boldsymbol {q} _ {t - 1} \| _ {1} ^ {2}, \text {a n d} S _ {T, k} ^ {\boldsymbol {p}} \triangleq \sum_ {t = 2} ^ {T} \| \boldsymbol {p} _ {t, k} - \boldsymbol {p} _ {t - 1, k} \| _ {1} ^ {2}. \tag {B.1}
$$
Proof. For simplicity, we define  $\mathbf{g}_t \triangleq \nabla f_t(\mathbf{x}_t)$ . For exp-concave functions, we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(r _ {t, i ^ {\star}} - m _ {t, i ^ {\star}}\right) ^ {2} = \sum_ {t = 1} ^ {T} \left(\left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} - \mathbf {x} _ {t, i ^ {\star}} \right\rangle - \left\langle \mathbf {g} _ {t - 1}, \mathbf {x} _ {t - 1} - \mathbf {x} _ {t - 1, i ^ {\star}} \right\rangle\right) ^ {2} \\ \leq 2 \sum_ {t = 1} ^ {T} \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} - \mathbf {x} _ {t, i ^ {\star}} \right\rangle^ {2} + 2 \sum_ {t = 1} ^ {T} \left\langle \mathbf {g} _ {t - 1}, \mathbf {x} _ {t - 1} - \mathbf {x} _ {t - 1, i ^ {\star}} \right\rangle^ {2} \\ \leq 4 \sum_ {t = 1} ^ {T} \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} - \mathbf {x} _ {t, i ^ {\star}} \right\rangle^ {2} + \mathcal {O} (1). \\ \end{array}
$$
For convex function, it holds that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(r _ {t, i ^ {\star}} - m _ {t, i ^ {\star}}\right) ^ {2} = \sum_ {t = 1} ^ {T} \left(\left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} - \mathbf {x} _ {t, i ^ {\star}} \right\rangle - \left\langle \mathbf {g} _ {t - 1}, \mathbf {x} _ {t - 1} - \mathbf {x} _ {t - 1, i ^ {\star}} \right\rangle\right) ^ {2} \\ \leq 2 \sum_ {t = 1} ^ {T} \left\langle \mathbf {g} _ {t} - \mathbf {g} _ {t - 1}, \mathbf {x} _ {t} - \mathbf {x} _ {t, i ^ {\star}} \right\rangle^ {2} + 2 \sum_ {t = 1} ^ {T} \left\langle \mathbf {g} _ {t - 1}, \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} + \mathbf {x} _ {t, i ^ {\star}} - \mathbf {x} _ {t - 1, i ^ {\star}} \right\rangle^ {2} \\ \leq 2 D ^ {2} \sum_ {t = 1} ^ {T} \| \mathbf {g} _ {t} - \mathbf {g} _ {t - 1} \| ^ {2} + 2 G ^ {2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} + \mathbf {x} _ {t, i ^ {\star}} - \mathbf {x} _ {t - 1, i ^ {\star}} \| ^ {2} \quad (\text {b y}) \\ \leq 4 D ^ {2} V _ {T} + 4 (D ^ {2} L ^ {2} + G ^ {2}) \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2} + 4 G ^ {2} \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t, i ^ {\star}} - \mathbf {x} _ {t - 1, i ^ {\star}} \| ^ {2}, \\ \end{array}
$$
where the last step is due to the definition of the gradient variation, finishing the proof.
In this part, we analyze the negative stability terms in the MsMwC algorithm [Chen et al., 2021]. For self-containedness, we restate the update rule of MsMwC in the following:
$$
\boldsymbol {p} _ {t} = \underset {\boldsymbol {p} \in \Delta_ {d}} {\arg \min } \left\{\langle \boldsymbol {m} _ {t}, \boldsymbol {p} \rangle + \mathcal {D} _ {\psi_ {t}} (\boldsymbol {p}, \widehat {\boldsymbol {p}} _ {t}) \right\}, \quad \widehat {\boldsymbol {p}} _ {t + 1} = \underset {\boldsymbol {p} \in \Delta_ {d}} {\arg \min } \left\{\langle \boldsymbol {\ell} _ {t} + \boldsymbol {a} _ {t}, \boldsymbol {p} \rangle + \mathcal {D} _ {\psi_ {t}} (\boldsymbol {p}, \widehat {\boldsymbol {p}} _ {t}) \right\},
$$
where the bias term  $a_{t,i} = 16\eta_{t,i}(\ell_{t,i} - m_{t,i})^2$ . Below, we give a detailed proof of Lemma 2, following a similar logic flow as Lemma 1 of Chen et al. [2021], while illustrating the negative stability terms. Moreover, for generality, we investigate a more general setting of an arbitrary comparator  $\pmb{u} \in \Delta_d$  and changing step sizes  $\eta_{t,i}$ . This was done hoping that the negative stability term analysis would be comprehensive enough for readers interested solely in the MsMwC algorithm.
Proof of Lemma 2. To begin with, the regret with correction can be analyzed as follows:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {\ell} _ {t} + \boldsymbol {a} _ {t}, \boldsymbol {p} _ {t} - \boldsymbol {e} _ {i ^ {\star}} \right\rangle \leq \sum_ {t = 1} ^ {T} \left(\mathcal {D} _ {\psi_ {t}} (\boldsymbol {u}, \widehat {\boldsymbol {p}} _ {t}) - \mathcal {D} _ {\psi_ {t}} (\boldsymbol {u}, \widehat {\boldsymbol {p}} _ {t + 1})\right) + \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {\ell} _ {t} + \boldsymbol {a} _ {t} - \boldsymbol {m} _ {t}, \boldsymbol {p} _ {t} - \widehat {\boldsymbol {p}} _ {t + 1} \right\rangle \\ - \sum_ {t = 1} ^ {T} \left(\mathcal {D} _ {\psi_ {t}} \left(\widehat {\boldsymbol {p}} _ {t + 1}, \boldsymbol {p} _ {t}\right) + \mathcal {D} _ {\psi_ {t}} \left(\boldsymbol {p} _ {t}, \widehat {\boldsymbol {p}} _ {t}\right)\right) \\ \leq \underbrace {\sum_ {t = 1} ^ {T} \left(\mathcal {D} _ {\psi_ {t}} (\boldsymbol {u} , \widehat {\boldsymbol {p}} _ {t}) - \mathcal {D} _ {\psi_ {t}} (\boldsymbol {u} , \widehat {\boldsymbol {p}} _ {t + 1})\right)} _ {\text {T E R M (A)}} + \underbrace {\sum_ {t = 1} ^ {T} \left(\left\langle \boldsymbol {\ell} _ {t} + \boldsymbol {a} _ {t} - \boldsymbol {m} _ {t} , \boldsymbol {p} _ {t} - \widehat {\boldsymbol {p}} _ {t + 1} \right\rangle - \frac {1}{2} \mathcal {D} _ {\psi_ {t}} (\widehat {\boldsymbol {p}} _ {t + 1} , \boldsymbol {p} _ {t})\right)} _ {\text {T E R M (B)}} \\ \end{array}
$$
$$
- \frac {1}{2} \underbrace {\sum_ {t = 1} ^ {T} \left(\mathcal {D} _ {\psi_ {t}} \left(\widehat {\boldsymbol {p}} _ {t + 1} , \boldsymbol {p} _ {t}\right) + \mathcal {D} _ {\psi_ {t}} \left(\boldsymbol {p} _ {t} , \widehat {\boldsymbol {p}} _ {t}\right)\right)} _ {\text {T E R M (C)}},
$$