SlowGuess/NeurIPS_2023_OCR huggingface dataset:
second one measures the base learner stability, which can be canceled by optimistic algorithms [Rakhlin and Sridharan, 2013a]. We provide a self-contained analysis of the stability of optimistic online mirror descent (OMD) in Appendix E.2. At last, if the stability term of the final decisions (i.e.,  $\| \mathbf{x}_t - \mathbf{x}_{t - 1}\|^2$ ) can be canceled, an  $\mathcal{O}(\sqrt{V_T})$  meta regret for convex functions is achievable.
To this end, we give the positive term  $\| \mathbf{x}_t - \mathbf{x}_{t-1}\|^2$  a more detailed decomposition, due to the fact that the final decision is the weighted combination of base learners' decisions (i.e.,  $\mathbf{x}_t = \sum_{i} p_{t,i} \mathbf{x}_{t,i}$ ). A detailed proof is deferred to Lemma 6. Specifically, it holds that
$$
\left\| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \right\| ^ {2} \lesssim \left\| \boldsymbol {p} _ {t} - \boldsymbol {p} _ {t - 1} \right\| _ {1} ^ {2} + \sum_ {i = 1} ^ {N} p _ {t, i} \left\| \mathbf {x} _ {t, i} - \mathbf {x} _ {t - 1, i} \right\| ^ {2}, \tag {3.1}
$$
where the first part represents the meta learner's stability while the second one is a weighted version of the base learners' stability. In Section 3.2, we cancel the two parts respectively.
3.2 Negative Terms for Cancellation
In this part, we propose negative terms to cancel the two parts of (3.1). Specifically, Section 3.2.1 analyzes the endogenous negative stability terms of the meta learner to handle the first part, and Section 3.2.2 proposes artificially-injected cascaded corrections to cancel the second part exogenously.
3.2.1 Endogenous Negativity: Stability Analysis of Meta Algorithms
In this part, we aim to control the meta learner's stability, measured by  $\| \pmb{p}_t - \pmb{p}_{t-1}\|_1^2$ . To this end, we leverage the two-layer meta algorithm proposed by Chen et al. [2021], where each layer runs the MsMwC algorithm [Chen et al., 2021] but with different parameter configurations. Specifically, a single MsMwC updates via the following rule:
$$
\boldsymbol {p} _ {t} = \underset {\boldsymbol {p} \in \Delta_ {d}} {\arg \min } \left\{\langle \boldsymbol {m} _ {t}, \boldsymbol {p} \rangle + \mathcal {D} _ {\psi_ {t}} (\boldsymbol {p}, \widehat {\boldsymbol {p}} _ {t}) \right\}, \quad \widehat {\boldsymbol {p}} _ {t + 1} = \underset {\boldsymbol {p} \in \Delta_ {d}} {\arg \min } \left\{\langle \ell_ {t} + \boldsymbol {a} _ {t}, \boldsymbol {p} \rangle + \mathcal {D} _ {\psi_ {t}} (\boldsymbol {p}, \widehat {\boldsymbol {p}} _ {t}) \right\}, \tag {3.2}
$$
where  $\Delta_d$  denotes a  $d$ -dimensional simplex,  $\psi_t(\pmb{p}) = \sum_{i=1}^d \eta_{t,i}^{-1} p_i \ln p_i$  is the weighted negative entropy regularizer with time-coordinate-varying step size  $\eta_{t,i}$ ,  $\mathcal{D}_{\psi_t}(\pmb{p}, \pmb{q}) = \psi_t(\pmb{p}) - \psi_t(\pmb{q}) - \langle \nabla \psi_t(\pmb{q}), \pmb{p} - \pmb{q} \rangle$  is the induced Bregman divergence for any  $\pmb{p}, \pmb{q} \in \Delta_d$ ,  $m_t$  is the optimism,  $\ell_t$  is the loss vector and  $\pmb{a}_t$  is a bias term.
It is worth noting that MsMwC is based on OMD, which is well-studied and proved to enjoy negative stability terms in analysis. However, the authors omitted them, which turns out to be crucial for our purpose. In Lemma 2 below, we extend Lemma 1 of Chen et al. [2021] by explicitly exhibiting the negative terms in MsMwC. The proof is deferred to Appendix B.2.
Lemma 2. If  $\max_{t\in [T],i\in [d]}\{|\ell_{t,i}|,|m_{t,i}|\} \leq 1$ , then MSMwC (3.2) with time-invariant step sizes (i.e.,  $\eta_{t,i} = \eta_i$  for any  $t\in [T])^3$  enjoys the following guarantee if  $\eta_i\leq 1 / 32$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \langle \boldsymbol {\ell} _ {t}, \boldsymbol {p} _ {t} \rangle - \sum_ {t = 1} ^ {T} \ell_ {t, i ^ {\star}} \leq \frac {1}{\eta_ {i ^ {\star}}} \ln \frac {1}{\widehat {p} _ {1 , i ^ {\star}}} + \sum_ {i = 1} ^ {d} \frac {\widehat {p} _ {1 , i}}{\eta_ {i}} - 8 \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {d} \eta_ {i} p _ {t, i} (\ell_ {t, i} - m _ {t, i}) ^ {2} \\ + 1 6 \eta_ {i ^ {\star}} \sum_ {t = 1} ^ {T} (\ell_ {t, i ^ {\star}} - m _ {t, i ^ {\star}}) ^ {2} - 4 \sum_ {t = 2} ^ {T} \| \boldsymbol {p} _ {t} - \boldsymbol {p} _ {t - 1} \| _ {1} ^ {2}. \\ \end{array}
$$
The two-layer meta algorithm is constructed by MsMwC in the following way. Briefly, both layers run MsMwC, but with different parameter configurations. Specifically, the top MsMwC (indicated by MsMwC-TOP) connects with  $K = \mathcal{O}(\log T)$  MsMwCs (indicated by MsMwC-MID), and each MsMwC-MID is further connected with  $N$  base learners (as specified in Section 2). The specific parameter configurations will be illuminated later. The two-layer meta algorithm is provable to enjoy a regret guarantee of  $\mathcal{O}(\sqrt{V_{\star}\ln V_{\star}})$  (Theorem 5 of Chen et al. [2021]), where  $V_{\star}$  is analogous to  $\sum_{t}(\ell_{t,i^{\star}} - m_{t,i^{\star}})^{2}$ , but within a multi-layer context (a formal definition will be shown later). By choosing the optimism as  $m_{t,i} = \langle \ell_t,p_t\rangle$ , it can recover the guarantee of ADAPT-ML-PROD,
3We only focus on the proof with fixed learning rate, since it is sufficient for our analysis.
Although  $\pmb{p}_t$  is unknown when using  $m_{t,i}$  due to (3.2),  $m_{t,i} = \langle \ell_t, \pmb{p}_t \rangle$  remains the same for all dimension  $i \in [d]$  and can thus omitted in the algorithm, i.e., it only suffices to update as  $\pmb{p}_t = \arg \min_{\pmb{p} \in \Delta_d} \mathcal{D}_{\psi_t}(\pmb{p}, \hat{\pmb{p}}_t)$ . Interested readers can refer to Section 2.1 of Chen et al. [2021] for more details.
Table 2: Notations of the three-layer structure. The first column presents the index of layers. The rest columns illuminate the notations for the algorithms, losses, optimisms, decisions and outputs of each layer.
Layer	Algorithm	Loss	Optimism	Decision	Output
Top (Meta)	MsMWC	\( \ell_t \)	\( m_t \)	\( q_t \in \Delta_K \)	\( \mathbf{x}_t = \sum_k q_{t,k} \mathbf{x}_{t,k} \)
Middle (Meta)	MsMWC	\( \ell_{t,k} \)	\( m_{t,k} \)	\( p_{t,k} \in \Delta_N \)	\( \mathbf{x}_{t,k} = \sum_i p_{t,k,i} \mathbf{x}_{t,k,i} \)
Bottom (Base)	Optimistic OMD	\( f_t(\cdot) \)	\( \nabla f_{t-1}(\cdot) \)	\( \mathbf{x}_{t,k,i} \in \mathcal{X} \)	\( \mathbf{x}_{t,k,i} \)
although up to an  $\mathcal{O}(\ln V_{\star})$  factor (leading to an  $\hat{\mathcal{O}} (\sqrt{V_T})$  meta regret), but with additional negative terms in analysis. Note that single MsMwC enjoys an  $\mathcal{O}(\sqrt{V_{\star}\ln T})$  bound, where the extra  $\ln T$  factor would ruin the desired  $\mathcal{O}(\log V_T)$  bound for exp-concave and strongly convex functions. This is the reason for choosing a second-layer meta algorithm, overall resulting in a three-layer structure.
For clarity, we summarize the notations of the three-layer structure in Table 2. Besides, previous notions need to be extended analogously. Specifically, instead of (2.2), regret is now decomposed as
$$
\operatorname {R E G} _ {T} = \left[ \sum_ {t = 1} ^ {T} f _ {t} (\mathbf {x} _ {t}) - \sum_ {t = 1} ^ {T} f _ {t} \left(\mathbf {x} _ {t, k ^ {\star}, i ^ {\star}}\right) \right] + \left[ \sum_ {t = 1} ^ {T} f _ {t} \left(\mathbf {x} _ {t, k ^ {\star}, i ^ {\star}}\right) - \min  _ {\mathbf {x} \in \mathcal {X}} \sum_ {t = 1} ^ {T} f _ {t} (\mathbf {x}) \right].
$$
The quantity  $V_{\star}$  is defined as  $V_{\star} \triangleq \sum_{t} (\ell_{t,k^{\star},i^{\star}} - m_{t,k^{\star},i^{\star}})^{2}$ , where  $\ell_{t,k,i} = \langle \nabla f_t(\mathbf{x}_t), \mathbf{x}_{t,k,i} \rangle$  and  $m_{t,k,i} = \langle \nabla f_t(\mathbf{x}_t), \mathbf{x}_t \rangle - \langle \nabla f_{t-1}(\mathbf{x}_{t-1}), \mathbf{x}_{t-1} - \mathbf{x}_{t-1,k,i} \rangle$  follows the same sprit as Lemma 1.
At the end of this part, we explain why we choose MsMwC as the meta algorithm. Apparently, a direct try is to keep using ADAPT-ML-PROD following Zhang et al. [2022a]. However, it is still an open problem to determine whether ADAPT-ML-PROD contains negative stability terms in the analysis, which is essential to realize effective cancellation in our problem. Another try is to explore the titled exponentially weighted average (TEWA) as the meta algorithm, following another line of research [van Erven and Koolen, 2016], as introduced in Section 1.1. Unfortunately, its stability property is also unclear. Investigating the negative stability terms in these algorithms is an important open problem, but beyond the scope of this work.
3.2.2 Exogenous Negativity: Cascaded Correction Terms
In this part, we aim to deal with the second term in (3.1) (i.e.,  $\sum_{i}p_{t,i}\| \mathbf{x}_{t,i} - \mathbf{x}_{t - 1,i}\| ^2$ ). Inspired by the work of Zhao et al. [2021] on the gradient-variation dynamic regret in non-stationary online learning, we incorporate exogenous correction terms for cancellation. Concretely, we inject cascaded correction terms to both top and middle layers. Specifically, the loss  $\ell_t$  and the optimism  $m_{t}$  of MSmwC-TOP are chosen as
$$
\ell_ {t, k} \triangleq \left\langle \nabla f _ {t} \left(\mathbf {x} _ {t}\right), \mathbf {x} _ {t, k} \right\rangle + \lambda_ {1} \left\| \mathbf {x} _ {t, k} - \mathbf {x} _ {t - 1, k} \right\| ^ {2}, m _ {t, k} \triangleq \left\langle \widehat {\boldsymbol {m}} _ {t, k}, \boldsymbol {p} _ {t, k} \right\rangle + \lambda_ {1} \left\| \mathbf {x} _ {t, k} - \mathbf {x} _ {t - 1, k} \right\| ^ {2}. \tag {3.3}
$$
The loss  $\ell_{t,k}$  and optimism  $m_{t,k}$  of MSMWC-MID are chosen similarly as
$$
\ell_ {t, k, i} \triangleq \left\langle \nabla f _ {t} (\mathbf {x} _ {t}), \mathbf {x} _ {t, k, i} \right\rangle + \lambda_ {2} \left\| \mathbf {x} _ {t, k, i} - \mathbf {x} _ {t - 1, k, i} \right\| ^ {2}, m _ {t, k, i} \triangleq \widehat {m} _ {t, k, i} + \lambda_ {2} \left\| \mathbf {x} _ {t, k, i} - \mathbf {x} _ {t - 1, k, i} \right\| ^ {2}, \tag {3.4}
$$
where  $\widehat{\pmb{m}}_{t,k} \triangleq (\widehat{m}_{t,k,1},\dots,\widehat{m}_{t,k,N})$  and  $\widehat{m}_{t,k,i} = \langle \nabla f_t(\mathbf{x}_t),\mathbf{x}_t\rangle -\langle \nabla f_{t - 1}(\mathbf{x}_{t - 1}),\mathbf{x}_{t - 1} - \mathbf{x}_{t - 1,k,i}\rangle$  denotes our optimism, which uses the same idea as Lemma 1 in Section 3.1.
To see how the correction term works, consider a simpler problem with regret  $\sum_{t} \langle \ell_t, \mathbf{q}_t \rangle - \sum_{t} \ell_{t,k^*}$ . If we instead optimize the corrected loss  $\ell_t + b_t$  and obtain a regret bound of  $R_T$ , then moving the correction terms to the right-hand side, the original regret is at most  $R_T - \sum_{t} \sum_{k} q_{t,k} b_{t,k} + \sum_{t} b_{t,k^*}$  where the negative term of  $-\sum_{t} \sum_{k} q_{t,k} b_{t,k}$  can be leveraged for cancellation. Meanwhile, the algorithm is required to handle an extra term of  $\sum_{t} b_{t,k^*}$ , which only relies on the  $k^*$ -th dimension. In the next part, we will discuss about how cascaded corrections and negative stability terms of meta algorithms realize effective collaboration for cancellation in the multi-layer online ensemble.
It is noteworthy that our work is the first to introduce correction mechanisms to universal online learning, whereas prior works use it for different purposes, such as non-stationary online learn-
This work proposes an improved dynamic regret minimization algorithm compared to its conference version [Zhao et al., 2020], which introduces the correction terms to the meta-base online ensemble structure and thus improves the gradient query complexity from  $\mathcal{O}(\log T)$  to 1 within each round.
Algorithm 1 Universal OCO with Gradient-variation Guarantees
Input: Curvature coefficient pool  $\mathcal{H}$ , MSMWC-MID number  $K$ , base learner number  $N$   
1: Initialize: Top layer:  $\mathcal{A}^{\mathrm{top}}$  — MSMWC-TOP with  $\eta_{k} = (C_{0} \cdot 2^{k})^{-1}$  and  $\widehat{q}_{1,k} = \eta_{k}^{2} / \sum_{k=1}^{K} \eta_{k}^{2}$   
Middle layer:  $\{\mathcal{A}_{k}^{\mathrm{mid}}\}_{k \in [K]}$  — MSMWC-MID with step size  $2\eta_{k}$  and  $\widehat{p}_{1,k,i} = 1/N$   
Bottom layer:  $\{\mathcal{B}_{k,i}\}_{k \in [K], i \in [N]}$  — base learners as specified in Section 2  
2: for  $t = 1$  to  $T$  do  
3: Receive  $\mathbf{x}_{t,k,i}$  from  $\mathcal{B}_{k,i}$ , obtain  $\mathbf{x}_{t,k} = \sum_{i} p_{t,k,i} \mathbf{x}_{t,k,i}$  and submit  $\mathbf{x}_t = \sum_k q_{t,k} \mathbf{x}_{t,k}$   
4: Suffer  $f_{t}(\mathbf{x}_{t})$  and observe the gradient information  $\nabla f_{t}(\cdot)$   
5: Construct  $(\ell_t, m_t)$  (3.3) for  $\mathcal{A}^{\mathrm{top}}$  and  $(\ell_{t,k}, m_{t,k})$  (3.4) for  $\mathcal{A}_k^{\mathrm{mid}}$   
6:  $\mathcal{A}^{\mathrm{top}}$  updates to  $q_{t+1}$  and  $\mathcal{A}_k^{\mathrm{mid}}$  updates to  $p_{t+1,k}$   
7: Multi-gradient feedback model:  
8: Send gradient  $\nabla f_{t}(\cdot)$  to  $\mathcal{B}_{k,i}$  for update  
9: One-gradient feedback model:  
10: Construct surrogates  $h_{t,i}^{\mathrm{sc}}(\cdot)$ ,  $h_{t,i}^{\mathrm{exp}}(\cdot)$ ,  $h_{t,i}^{\mathrm{c}}(\mathbf{x})$  using only  $\nabla f_{t}(\mathbf{x}_{t})$   
11: Send the surrogate functions to  $\mathcal{B}_{k,i}$  for update  
12: end for
ing [Zhao et al., 2021] and the multi-scale expert problem [Chen et al., 2021]. Distinctively, different from the conventional two-layer algorithmic frameworks seen in prior studies, deploying this technique to a three-layer structure necessitates a comprehensive use and extensive adaptation of it.
3.3 Overall Algorithm: A Multi-layer Online Ensemble Structure
In this part, we conclude our three-layer online ensemble approach in Algorithm 1. In Line 3, the decisions are aggregated from bottom to top for the final output. In Line 4, the learner suffers the loss of the decision, and the environments return the gradient information of the loss function. In Line 5, the algorithm constructs the surrogate losses and optimisms for the two-layer meta learner. In Line 6-11, the update is conducted from top to bottom. Note that in Line 8, our algorithm requires multiple, concretely  $\mathcal{O}(\log^2 T)$ , gradient queries per round since it needs to query  $\nabla f_t(\mathbf{x}_{t,k,i})$  for each base learner, making it inefficient when the gradient evaluation is costly. To this end, in Section 4, we improve the algorithm's gradient query complexity to 1 per round, corresponding to Line 10-11, via a novel regret decomposition and carefully designed surrogate loss functions.
In Figure 1, we illustrate the detailed procedure of the collaboration in our multi-layer online ensemble approach. Specifically, we aim to deal with the positive term of  $\| \mathbf{x}_t - \mathbf{x}_{t - 1}\|^2$ , which stems from the universal optimism design proposed in Section 3.1. Since  $\mathbf{x}_t = \sum_k q_{t,k} \mathbf{x}_{t,k}$ , the positive term can be further decomposed into two parts:  $\| \pmb{q}_t - \pmb{q}_{t - 1}\|_1^2$  and  $\sum_k q_{t,k} \| \mathbf{x}_{t,k} - \mathbf{x}_{t - 1,k}\|^2$ , which can be canceled by the negative and correction terms in MsMWC-TOP, respectively. Note that the correction comes at the expense of an extra term of  $\| \mathbf{x}_{t,k^*} - \mathbf{x}_{t - 1,k^*}\|^2$ , which can be decomposed similarly into two parts:  $\| \pmb{p}_{t,k^*} - \pmb{p}_{t - 1,k^*}\|_1^2$  and  $\sum_i p_{t,k^*,i} \| \mathbf{x}_{t,k^*,i} - \mathbf{x}_{t - 1,k^*,i}\|^2$  because of  $\mathbf{x}_{t,k} = \sum_i p_{t,k,i} \mathbf{x}_{t,k,i}$ . The negative term and corrections in the middle layer (specifically, the  $k^*$ -th MsMWC-MID) can be leveraged for cancellation. This correction finally generate  $\| \mathbf{x}_{t,k^*,i^*} - \mathbf{x}_{t - 1,k^*,i^*}\|^2$ , which can be handled by choosing optimistic OMD as base algorithms.
As a final remark, although it is possible to treat the two-layer meta algorithm as a single layer, its analysis will become much more complicated than that in one layer and is unsuitable for extending to more layers. In contrast, our layer-by-layer analytical approach paves a systematic and principled way for analyzing the dynamics of the online ensemble framework with even more layers.
3.4 Universal Regret Guarantees
In this part, we conclude our main theoretical result and provide several implications and applications to validate its importance and practical potential. Theorem 1 summarizes the main result, universal regret guarantees in terms of the gradient variation. The proof is deferred to Appendix B.4.
Theorem 1. Under Assumptions 1 and 2, Algorithm 1 obtains  $\mathcal{O}(\log V_T)$ ,  $\mathcal{O}(d\log V_T)$  and  $\widehat{\mathcal{O}} (\sqrt{V_T})$  regret bounds for strongly convex, exp-concave and convex functions, respectively.
Figure 1: Decomposition of the positive term  $\| \mathbf{x}_t - \mathbf{x}_{t - 1}\|^2$  and how it is handled by the multi-layer online ensemble via endogenous negativity from meta algorithm and exogenous negativity from cascaded corrections.
Theorem 1 improves the results of Zhang et al. [2022a] by not only maintaining the optimal rates for strongly convex and exp-concave functions but also taking advantage of the small gradient variation for convex functions when  $V_{T} \ll F_{T}$ . For example, if  $f_{1} = \ldots = f_{T} = f$  and  $\min_{\mathbf{x} \in \mathcal{X}} f(\mathbf{x}) = 1$ , our bound is much better since  $V_{T} = 0$  while  $F_{T} = T$ . Moreover, Algorithm 1 also provably achieves universal small-loss guarantees without any algorithmic modification and thus safeguards the case when  $F_{T} \leq V_{T}$ . We conclude the result below and provide the proof in Appendix B.5.
Corollary 1. Under Assumptions 1 and 2, if  $f_{t}(\cdot) \geq 0$ , Algorithm 1 obtains  $\mathcal{O}(\log F_T)$ ,  $\mathcal{O}(d\log F_T)$  and  $\widehat{\mathcal{O}} (\sqrt{F_T})$  regret bounds for strongly convex, exp-concave and convex functions.
Due to the connection of the gradient-variation with stochastic and adversarial OCO [Sachs et al., 2022] and game theory [Syrgkanis et al., 2015], our results can be immediately applied and achieve best known universal guarantees therein. Due to page limits, we defer applications to Appendix A.
4 Improved Gradient Query Complexity
Though achieving favorable theoretical guarantees in Section 3, one caveat is that our algorithm requires  $\mathcal{O}(\log^2 T)$  gradient queries per round since it needs to query  $\nabla f_t(\mathbf{x}_{t,k,i})$  for all  $k\in [K],i\in [N]$ , making it computational-inefficient when the gradient evaluation is costly, e.g., in nuclear norm optimization [Ji and Ye, 2009] and mini-batch optimization [Li et al., 2014]. The same concern also appears in the approach of Zhang et al. [2022a], who provided small-loss and worst-case regret guarantees for universal online learning. By contrast, traditional algorithms such as OGD typically work under the one-gradient feedback setup, namely, they only require one gradient  $\nabla f_t(\mathbf{x}_t)$  for the update. In light of this, it is natural to ask whether there is a universal algorithm that can maintain the desired regret guarantees while using only one gradient query per round.
We answer the question affirmatively by reducing the gradient query complexity to 1 per round. To describe the high-level idea, we first consider the case of known  $\lambda$ -strong convexity within a two-layer structure, e.g., adaptive regret minimization for strongly convex functions [Wang et al., 2018]. The regret can be upper-bounded as  $\mathrm{REG}_T \leq [\sum_t h_t(\mathbf{x}_t) - \sum_t h_t(\mathbf{x}_{t,i^*})] + [\sum_t h_t(\mathbf{x}_{t,i^*}) - \sum_t h_t(\mathbf{x}^\star)]$ , where  $\mathbf{x}^\star \in \arg \min_{\mathbf{x} \in \mathcal{X}} \sum_t f_t(\mathbf{x})$  and  $h_t(\mathbf{x}) \triangleq \langle \nabla f_t(\mathbf{x}_t), \mathbf{x} \rangle + \lambda \| \mathbf{x} - \mathbf{x}_t \|^2 / 2$  is a second-order surrogate of the original function  $f_t$ . This yields a homogeneous surrogate for both meta and base algorithms — the meta algorithm uses the same evaluation function across all the base learners, who in turn use this function to update their base-level decisions.
In the universal online learning (i.e., unknown  $\lambda$ ), a natural adaptation would be a heterogeneous surrogate  $h_{t,i}(\mathbf{x}) \triangleq \langle \nabla f_t(\mathbf{x}_t), \mathbf{x} \rangle + \lambda_i \| \mathbf{x} - \mathbf{x}_t \|^2 / 2$  for the  $i$ -th base learner, where  $\lambda_i$  is a guess of the true curvature  $\lambda$  from the candidate pool  $\mathcal{H}$  (2.1). Consequently, the regret decomposition with respect to this surrogate becomes  $\mathrm{REG}_T \leq [\sum_t h_{t,i^*}(\mathbf{x}_t) - \sum_t h_{t,i^*}(\mathbf{x}_{t,i^*})] + [\sum_t h_{t,i^*}(\mathbf{x}_{t,i^*}) - \sum_t h_{t,i^*}(\mathbf{x}^\star)]$ , where  $i^*$  denotes the index of the best base learner whose strong convexity coefficient satisfies  $\lambda_{i^*} \leq \lambda \leq 2\lambda_{i^*}$ . This admits heterogeneous surrogates for both meta and base regret, which
necessitates designing expert-tracking algorithms with heterogeneous inputs for different experts as required in MetaGrad [van Erven and Koolen, 2016], which is not flexible enough.
To this end, we propose a novel regret decomposition that admits homogeneous surrogates for the meta regret, making our algorithm as flexible as Zhang et al. [2022a], and heterogeneous surrogate for the base regret, making it as efficient as van Erven and Koolen [2016]. Specifically, we remain the heterogeneous surrogates  $h_{t,i}(\mathbf{x})$  defined above for the base regret. For the meta regret, we define a homogeneous linear surrogate  $\ell_t(\mathbf{x}) \triangleq \langle \nabla f_t(\mathbf{x}_t),\mathbf{x}\rangle$  such that the meta regret is bounded by  $\sum_{t}\ell_{t}(\mathbf{x}_{t}) - \sum_{t}\ell_{t}(\mathbf{x}_{t,i^{\star}}) - \lambda_{i^{\star}}\sum_{t}\| \mathbf{x}_{t} - \mathbf{x}_{t,i^{\star}}\|^{2} / 2$ . As long as we can obtain a second-order bound for the regret defined on this surrogate loss, i.e.,  $\sum_{t}\ell_{t}(\mathbf{x}_{t}) - \sum_{t}\ell_{t}(\mathbf{x}_{t,i^{\star}})$ , it can be canceled by the negative term from strong convexity. Overall, we decompose the regret in the following way:
$$
\operatorname {R E G} _ {T} \leq \left[ \sum_ {t = 1} ^ {T} \ell_ {t} (\mathbf {x} _ {t}) - \sum_ {t = 1} ^ {T} \ell_ {t} \left(\mathbf {x} _ {t, i ^ {\star}}\right) - \frac {\lambda_ {i ^ {\star}}}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t, i ^ {\star}} \| ^ {2} \right] + \left[ \sum_ {t = 1} ^ {T} h _ {t, i ^ {\star}} \left(\mathbf {x} _ {t, i ^ {\star}}\right) - \sum_ {t = 1} ^ {T} h _ {t, i ^ {\star}} \left(\mathbf {x} ^ {\star}\right) \right].
$$
For clarity, we denote this surrogate for strongly convex functions by  $h_{t,i}^{\mathrm{sc}}(\mathbf{x})$ . Similarly, we define the surrogates  $h_{t,i}^{\mathrm{exp}}(\mathbf{x}) \triangleq \langle \nabla f_t(\mathbf{x}_t),\mathbf{x}\rangle +\alpha_i\langle \nabla f_t(\mathbf{x}_t),\mathbf{x} - \mathbf{x}_t\rangle^2 /2$  and  $h_{t,i}^{\mathrm{c}}(\mathbf{x}) \triangleq \langle \nabla f_t(\mathbf{x}_t),\mathbf{x}\rangle$  for  $\alpha$ -exp-concave and convex functions, respectively. These surrogates require only one gradient  $\nabla f_{t}(\mathbf{x}_{t})$  within each round, thus successfully reducing the gradient query complexity.
Note that the base regret optimization requires controlling the algorithmic stability, because the empirical gradient variation  $\nabla h_{t,i}^{\mathrm{sc}}(\mathbf{x}_{t,i}) - \nabla h_{t - 1,i}^{\mathrm{sc}}(\mathbf{x}_{t - 1,i}) = \nabla f_t(\mathbf{x}_t) + \lambda_i(\mathbf{x}_{t,i} - \mathbf{x}_t) - \nabla f_{t - 1}(\mathbf{x}_{t - 1})-$ $\lambda_{i}(\mathbf{x}_{t - 1,i} - \mathbf{x}_{t - 1})$  not only contains the desired gradient variation, but also includes the positive stability terms of base and final decisions. Fortunately, as discussed earlier, these stability terms can be effectively addressed through our cancellation mechanism within the multi-layer online ensemble. This stands in contrast to previous two-layer algorithms with worst-case regret bounds [Zhang et al., 2018, Wang et al., 2018], where the algorithmic stability is not examined.