Title: Universal Online Learning with Gradient Variations: A Multi-layer Online Ensemble Approach
Venue: NeurIPS 2023
Accepted: yes
Keywords: online learning
Abstract: In this paper, we propose an online convex optimization approach with two different levels of adaptivity. On a higher level, our approach is agnostic to the unknown types and curvatures of the online functions, while at a lower level, it can exploit the unknown niceness of the environments and attain problem-dependent guarantees. Specifically, we obtain $\mathcal{O}(\log V_T)$, $\mathcal{O}(d \log V_T)$ and $\hat{\mathcal{O}}(\sqrt{V_T})$ regret bounds for strongly convex, exp-concave and convex loss functions, respectively, where $d$ is the dimension, $V_T$ denotes problem-dependent gradient variations and the $\hat{\mathcal{O}}(\cdot)$-notation omits $\log V_T$ factors. Our result not only safeguards the worst-case guarantees but also directly implies the small-loss bounds in analysis. Moreover, when applied to adversarial/stochastic convex optimization and game theory problems, our result enhances the existing universal guarantees. Our approach is based on a multi-layer online ensemble framework incorporating novel ingredients, including a carefully designed optimism for unifying diverse function types and cascaded corrections for algorithmic stability. Notably, despite its multi-layer structure, our algorithm necessitates only one gradient query per round, making it favorable when the gradient evaluation is time-consuming. This is facilitated by a novel regret decomposition equipped with carefully designed surrogate losses.

Universal Online Learning with Gradient Variations: A Multi-layer Online Ensemble Approach

Abstract

In this paper, we propose an online convex optimization approach with two different levels of adaptivity. On a higher level, our approach is agnostic to the unknown types and curvatures of the online functions, while at a lower level, it can exploit the unknown niceness of the environments and attain problem-dependent guarantees. Specifically, we obtain O (log V T ) , O ( d log V T ) and ̂ O ( √ V T ) regret bounds for strongly convex, exp-concave and convex loss functions, respectively, where d is the dimension, V T denotes problem-dependent gradient variations and the ̂ O ( · ) -notation omits log V T factors. Our result not only safeguards the worstcase guarantees but also directly implies the small-loss bounds in analysis. Moreover, when applied to adversarial/stochastic convex optimization and game theory problems, our result enhances the existing universal guarantees. Our approach is based on a multi-layer online ensemble framework incorporating novel ingredients, including a carefully designed optimism for unifying diverse function types and cascaded corrections for algorithmic stability. Notably, despite its multi-layer structure, our algorithm necessitates only one gradient query per round, making it favorable when the gradient evaluation is time-consuming. This is facilitated by a novel regret decomposition equipped with carefully designed surrogate losses.

1 Introduction

Online convex optimization (OCO) is a versatile model that depicts the interaction between a learner and the environments over time [Hazan, 2016, Orabona, 2019]. In each round t ∈ [ T ] , the learner selects a decision x t from a convex compact set X ⊆ R d , and simultaneously the environments choose a convex loss function f t : X ↦→ R . Subsequently, the learner incurs a loss f t ( x t ) , obtains information about the online function, and updates the decision to x t +1 , aiming to optimize the game-theoretical performance measure known as regret [Cesa-Bianchi and Lugosi, 2006]:

which represents the learner's excess loss compared to the best fixed comparator in hindsight.

In OCO, the type and curvature of online functions significantly impact the minimax regret bounds. Specifically, for convex functions, online gradient descent (OGD) can achieve an O ( √ T ) regret guarantee [Zinkevich, 2003]. For α -exp-concave functions, online Newton step (ONS), with prior knowledge of the curvature coefficient α , attains an O ( d log T ) regret [Hazan et al., 2007]. For λ -strongly convex functions, OGD with prior knowledge of the curvature coefficient λ and a different parameter configuration enjoys an O (log T ) regret [Hazan et al., 2007]. Note that the above

O

{

}

O

{

}

̂

O

}

algorithms require the function type and curvature beforehand and does not consider the niceness of environments. Recent studies further strengthen the algorithms and results with two levels of adaptivity . The higher-level adaptivity requires an algorithm to be agnostic to the unknown types and curvatures of the online functions. And the lower-level adaptivity requires an algorithm to exploit the unknown niceness of the environments within a specific function family. In the following, we delve into an extensive discussion on these two levels of adaptivity.

1.1 High Level: Adaptive to Unknown Curvature of Online Functions

Traditionally, the learner needs to know the function type and curvature in advance to select suitable algorithms (and parameter configurations), which can be burdensome in practice. Universal online learning [van Erven and Koolen, 2016, Wang et al., 2019, Zhang et al., 2022a] aims to develop a single algorithm agnostic to the specific function type and curvature while achieving the same regret guarantees as if they were known. The pioneering work of van Erven and Koolen [2016] proposed a single algorithm called MetaGrad that achieves an O ( √ T ) regret for convex functions and an O ( d log T ) regret for exp-concave functions. Later, Wang et al. [2019] further obtained the optimal O (log T ) regret for strongly convex functions. Notably, these approaches are efficient regarding the gradient query complexity, by using only one gradient within each round.

However, the above approaches are not flexible enough since they have to optimize a group of heterogeneous and carefully-designed surrogate loss functions, which can be cumbersome and challenging. To this end, Zhang et al. [2022a] introduced a simple framework that operates on the original online functions with the same optimal results at the expense of O (log T ) gradient queries.

1.2 Low Level: Adaptive to Unknown Niceness of Online Environments

Within a specific function family, the algorithm's performance is also substantially influenced by the niceness of environments. This concept is usually captured through problem-dependent quantities in the literature. Therefore, it becomes essential to develope adaptive algorithms with problemdependent regret guarantees. Specifically, we consider the following problem-dependent quantities:

where the small loss F T represents the cumulative loss of the best comparator [Srebro et al., 2010, Orabona et al., 2012] and the gradient variation V T characterizes the variation of the function gradients [Chiang et al., 2012]. In particular, the gradient-variation bound demonstrates its fundamental importance in modern online learning from the following three aspects: (i) it safeguards the worstcase guarantees in terms of T and implies the small-loss bounds in analysis directly; (ii) it draws a profound connection between adversarial and stochastic convex optimization; and (iii) it is crucial for fast convergence rates in game theory. We will explain the three aspects in detail in the next part.

1.3 Our Contributions and Techniques

In this paper, we consider the two levels of adaptivity simultaneously and propose a novel universal approach that achieves O (log V T ) , O ( d log V T ) and ̂ O ( √ V T ) regret bounds for strongly convex,

exp-concave and convex loss functions, respectively, using only one gradient query per round, where ̂ O ( · ) -notation omits factors on log V T . Table 1 compares our results with existing ones. In summary, relying on the basic idea of online ensemble [Zhao et al., 2021], our approach primarily admits a multi-layer online ensemble structure with several important novel ingredients. Specifically, we propose a carefully designed optimism, a hyper-parameter encoding historical information, to handle different kinds of functions universally, particularly exp-concave functions. Nevertheless, it necessitates careful management of the stability of final decisions, which is complicated in the multi-layer structure. To this end, we analyze the negative stability terms in the algorithm and propose cascaded correction terms to realize effective collaboration among layers, thus enhancing the algorithmic stability. Moreover, we facilitate a novel regret decomposition equipped with carefully designed surrogate losses to achieve only one gradient query per round, making our algorithm as efficient as van Erven and Koolen [2016] regarding the gradient complexity. Our result resolves an open problem proposed by Zhang et al. [2022a], who have obtained partial results for exp-concave and strongly convex functions and asked whether it is possible to design designing a single algorithm with universal gradient-variation bounds. Among them, the convex case is particularly important because the improvement from T to V T is polynomial, whereas logarithmic in the other cases.

Next, we shed light on some applications of our approach. First, it safeguards the worst-case guarantees [van Erven and Koolen, 2016, Wang et al., 2019] and directly implies the small-loss bounds of Zhang et al. [2022a] in analysis. Second, gradient variation is shown to play an essential role in the stochastically extended adversarial (SEA) model [Sachs et al., 2022, Chen et al., 2023b], an interpolation between stochastic and adversarial OCO. Our approach resolves a major open problem left in Chen et al. [2023b] on whether it is possible to develop a single algorithm with universal guarantees for strongly convex, exp-concave, and convex functions in the SEA model. Third, in game theory, gradient variation encodes the changes in other players' actions and can thus lead to fast convergence rates [Rakhlin and Sridharan, 2013b, Syrgkanis et al., 2015, Zhang et al., 2022b]. We demonstrate the universality of our approach by taking two-player zero-sum games as an example.

Technical Contributions. Our first contribution is proposing a multi-layer online ensemble approach with effective collaboration among layers, which is achieved by a carefully-designed optimism to unify different kinds of functions and cascaded correction terms to improve the algorithmic stability within the multi-layer structure. The second contribution arises from efficiency. Although there are multiple layers, our algorithm only requires one gradient query per round, which is achieved by a novel regret decomposition equipped with carefully designed surrogate losses. Two interesting byproducts rises in our approach. The first one is the negative stability term in the analysis of MSMWC [Chen et al., 2021], which serves as an important building block of our algorithm. And the second byproduct contains a simple approach and analysis for the optimal worst-case universal guarantees, using one gradient query within each round.

Organization. The rest of the paper is structured as follows. Section 2 provides preliminaries. Section 3 proposes our multi-layer online ensemble approach for universal gradient-variation bounds. Section 4 further improves the gradient query complexity. Due to page limits, the applications of our proposed algorithm are deferred to Appendix A. All the proofs can be found in the appendices.

2 Preliminaries

In this section, we introduce some preliminary knowledge, including our assumptions, the definitions, the formal problem setup, and a review of the latest progress of Zhang et al. [2022a].

̂ Assumption 1 (Boundedness) . For any x , y ∈ X and t ∈ [ T ] , the domain diameter satisfies ∥ x -y ∥ ≤ D , and the gradient norm of the online functions is bounded by ∥∇ f t ( x ) ∥ ≤ G .

To begin with, we list some notations. Specifically, we use ∥·∥ for ∥·∥ 2 in default and use ∑ t , ∑ k , ∑ i as abbreviations for ∑ t ∈ [ T ] , ∑ k ∈ [ K ] and ∑ i ∈ [ N ] . a ≲ b represents a ≤ O ( b ) . ̂ O ( · ) -notation omits logarithmic factors on leading terms. For example, O ( √ V ) omits the dependence of log V .

Assumption 2 (Smoothness) . All online functions are L -smooth: ∥∇ f t ( x ) -∇ f t ( y ) ∥ ≤ L ∥ x -y ∥ for any x , y ∈ X and t ∈ [ T ] .

Both assumptions are common in the literature. Specifically, the boundedness assumption is common in OCO [Hazan, 2016]. The smoothness assumption is essential for first-order algorithms to

achieve gradient-variation bounds [Chiang et al., 2012]. Strong convexity and exp-concavity are defined as follows. For any x , y ∈ X , a function f is λ -strongly convex if f ( x ) -f ( y ) ≤ ⟨∇ f ( x ) , x -y ⟩-λ 2 ∥ x -y ∥ 2 , and is α -exp-concave if f ( x ) -f ( y ) ≤ ⟨∇ f ( x ) , x -y ⟩-α 2 ⟨∇ f ( x ) , x -y ⟩ 2 . Note that the formal definition of β -exp-concavity states that exp( -βf ( · )) is concave. Under Assumption 1, β -exp-concavity leads to our definition with α = 1 2 min { 1 4 GD , β } [Hazan, 2016, Lemma 4.3]. For simplicity, we use it as an alternative definition for exp-concavity.

In the following, we formally describe the problem setup and briefly review the key insight of Zhang et al. [2022a]. Concretely, we consider the problem where the learner has no prior knowledge about the function type (strongly convex, exp-concave, or convex) or the curvature coefficient ( α or λ ). Without loss of generality, we study the case where the curvature coefficients α, λ ∈ [ 1 / T , 1] . This requirement is natural because if α ( or λ ) < 1 / T , even the optimal regret is Ω( T ) [Hazan et al., 2007], which is vacuous. Conversely, functions with α ( or λ ) > 1 are also 1-exp-concave (or 1strongly convex). Thus using α ( or λ ) = 1 will only worsen the regret by a constant factor, which can be omitted. This condition is also used in previous works [Zhang et al., 2022a].

ABrief Review of Zhang et al. [2022a]. Ageneral solution to handle the uncertainty is to leverage a two-layer framework, which consists of a group of base learners exploring the environments and a meta learner tracking the best base learner on the fly. To handle the unknown curvature coefficients α and λ , the authors discretize them into the following candidate pool:

(i) about log T base learners, each of which runs the algorithm for strongly convex functions with a guess λ i ∈ H of the strong convexity coefficient λ ;

where |H| ≈ log T . Consequently, they design three groups of base learners:

(ii) about log T base learners, each of which runs the algorithm for exp-concave functions with a guess α i ∈ H of the exp-concavity coefficient α ;

(iii) 1 base learner that runs the algorithm for convex functions.

Overall, they maintain N ≈ log T +log T +1 base learners. Denoting by p t ≜ ( p t, 1 , . . . , p t,N ) the meta learner's weights and x t,i the i -th base learner's decision, the learner submits x t = ∑ i p t,i x t,i . In the two-layer framework, the regret (1.1) can be decomposed into two terms:

where the meta regret (first term) assesses how well the algorithm tracks the best base learner, and the base regret (second term) measures the performance of it. The best base learner is the one which runs the algorithm matching the ground-truth function type with the most accurate guess of the curvature - taking α -exp-concave functions as an example, there must exist a base learner indexed by i ⋆ , whose coefficient α i ⋆ ∈ H satisfies α i ⋆ ≤ α ≤ 2 α i ⋆ .

A direct benefit of the above decomposition is that the meta regret can be bounded by a constant O (1) for exp-concave and strongly convex functions, allowing the algorithm to perfectly inherit the gradient-variation bound from the base learner. Taking α -exp-concave functions as an example, by definition, the meta regret can be bounded by ∑ t r t,i ⋆ -α 2 ∑ t r 2 t,i ⋆ , where the first term r t,i ≜ ⟨∇ f t ( x t ) , x t -x t,i ⟩ denotes the linearized regret, and the second one is a negative term from exp-concavity. Choosing ADAPT-ML-PROD [Gaillard et al., 2014] as the meta algorithm bounds the first term ∑ t r t,i ⋆ by O ( √∑ t ( r t,i ⋆ ) 2 ) , which can be canceled by the negative term, leading to an O (1) meta regret. Due to the meta algorithm's benefits, their approach can inherit the gradient-variation guarantees from the base learner. Similar derivation also applies to strongly convex functions. However, their approach is not favorable enough in the convex case and is not efficient enough in terms of the gradient query complexity. We will give more discussions about the above issues in Section 3.1 and Section 4.

3 Our Approach

This section presents our multi-layer online ensemble approach with universal gradient-variation bounds. Specifically, in Section 3.1, we provide a novel optimism to unify different kinds of functions. In Section 3.2, we exploit two types of negative terms to cancel the positive term caused by

the optimism design. We summarize the overall algorithm in Section 3.3. Finally, in Section 3.4, we present the main results and list several applications of our approach.

3.1 Universal Optimism Design

In this part, we propose a novel optimism that simultaneously unifies various kinds of functions. We start by observing that Zhang et al. [2022a] does not enjoy gradient-variation bounds for convex functions, where the main challenge lies in obtaining an O ( √ V T ) meta regret for convex functions while simultaneously maintaining an O (1) regret for exp-concave and strongly convex functions. In the following, we focus on the meta regret because the base regret optimization is straightforward by employing the optimistic online learning technique [Rakhlin and Sridharan, 2013a,b] in a blackbox fashion. Optimistic online learning is essential in our problem since it can utilize the historical information, e.g., ∇ f t -1 ( · ) for our purpose due to the definition of the gradient variation.

Shifting our focus to the meta regret, we consider upper-bounding it by a second-order bound of O ( √∑ t ( r t,i ⋆ -m t,i ⋆ ) 2 ) , where r t,i = ⟨∇ f t ( x t ) , x t -x t,i ⟩ and the optimism m t,i ⋆ can encode historical information. Such a second-order bound can be easily obtained using existing prediction with expert advice algorithms, e.g., ADAPT-ML-PROD [Wei et al., 2016]. Nevertheless, as we will demonstrate in the following, designing an optimism m t,i that effectively unifies various function types is not straightforward, thereby requiring novel ideas in the optimism design.

To begin with, a natural impulse is to choose the optimism as m t,i = ⟨∇ f t -1 ( x t -1 ) , x t,i -x t ⟩ , 2 which yields the following second-order bound:

where the inequality is due to the boundedness assumption (Assumption 1). This optimism design handles the strongly convex functions well since the bound can be canceled by the negative term imported by strong convexity (i.e., -∥ x t -x t,i ⋆ ∥ 2 ). Moreover, it is quite promising for convex functions because the bound essentially consists of the desired gradient variation and a positive term of ∥ x t -x t -1 ∥ 2 (we will deal with it later). However, it fails for exp-concave functions because the negative term imported by exp-concavity (i.e., -⟨∇ f t ( x t ) , x t -x t,i ⋆ ⟩ 2 ) cannot be used for cancellation due to the mismatch of the formulation.

To unify various kinds of functions, we propose a novel optimism design defined by m t,i = r t -1 ,i . This design aims to secure a second-order bound of O ( √∑ t ( r t,i ⋆ -r t -1 ,i ⋆ ) 2 ) , which is sufficient to achieve an O (1) meta regret for exp-concave functions (with strong convexity being a subcategory thereof) while maintaining an O ( √ V T ) meta regret for convex functions. The high-level intuition behind this approach is as follows: although the bound cannot be canceled exactly by the negative term imported by exp-concavity (i.e., -r 2 t,i ⋆ ) within each round, it becomes manageable when aggregated across the whole time horizon as ∑ t ( r t,i ⋆ -r t -1 ,i ⋆ ) 2 ≲ 4 ∑ t r 2 t,i ⋆ , because r t,i ⋆ and r t -1 ,i ⋆ differs by merely a single time step. In the following, we propose the key lemma of the universal optimism design and defer the proof to Appendix B.1.

Lemma 1 (Key Lemma) . Under Assumptions 1 and 2, if the optimism is chosen as m t,i = r t -1 ,i = ⟨∇ f t -1 ( x t -1 ) , x t -1 -x t -1 ,i ⟩ , it holds that

Moreover, the second part of Lemma 1 shows that the bound for convex functions is also controllable by being further decomposed into three terms. The first term is the desired gradient variation. The

second one measures the base learner stability, which can be canceled by optimistic algorithms [Rakhlin and Sridharan, 2013a]. We provide a self-contained analysis of the stability of optimistic online mirror descent (OMD) in Appendix E.2. At last, if the stability term of the final decisions (i.e., ∥ x t -x t -1 ∥ 2 ) can be canceled, an O ( √ V T ) meta regret for convex functions is achievable.

To this end, we give the positive term ∥ x t -x t -1 ∥ 2 a more detailed decomposition, due to the fact that the final decision is the weighted combination of base learners' decisions (i.e., x t = ∑ i p t,i x t,i ). A detailed proof is deferred to Lemma 6. Specifically, it holds that where the first part represents the meta learner's stability while the second one is a weighted version of the base learners' stability. In Section 3.2, we cancel the two parts respectively.

3.2 Negative Terms for Cancellation

In this part, we propose negative terms to cancel the two parts of (3.1). Specifically, Section 3.2.1 analyzes the endogenous negative stability terms of the meta learner to handle the first part, and Section 3.2.2 proposes artificially-injected cascaded corrections to cancel the second part exogenously .

3.2.1 Endogenous Negativity: Stability Analysis of Meta Algorithms

In this part, we aim to control the meta learner's stability, measured by ∥ p t -p t -1 ∥ 2 1 . To this end, we leverage the two-layer meta algorithm proposed by Chen et al. [2021], where each layer runs the MSMWC algorithm [Chen et al., 2021] but with different parameter configurations. Specifically, a single MSMWC updates via the following rule:

where ∆ d denotes a d -dimensional simplex, ψ t ( p ) = ∑ d i =1 η -1 t,i p i ln p i is the weighted negative entropy regularizer with time-coordinate-varying step size η t,i , D ψ t ( p , q ) = ψ t ( p ) -ψ t ( q ) -⟨∇ ψ t ( q ) , p -q ⟩ is the induced Bregman divergence for any p , q ∈ ∆ d , m t is the optimism, ℓ t is the loss vector and a t is a bias term.

It is worth noting that MSMWC is based on OMD, which is well-studied and proved to enjoy negative stability terms in analysis. However, the authors omitted them, which turns out to be crucial for our purpose. In Lemma 2 below, we extend Lemma 1 of Chen et al. [2021] by explicitly exhibiting the negative terms in MSMWC. The proof is deferred to Appendix B.2.

Lemma 2. If max t ∈ [ T ] ,i ∈ [ d ] {| ℓ t,i | , | m t,i |} ≤ 1 , then MSMWC (3.2) with time-invariant step sizes (i.e., η t,i = η i for any t ∈ [ T ] ) 3 enjoys the following guarantee if η i ≤ 1 / 32

The two-layer meta algorithm is constructed by MSMWC in the following way. Briefly, both layers run MSMWC, but with different parameter configurations. Specifically, the top MSMWC (indicated by MSMWC-TOP) connects with K = O (log T ) MSMWCs (indicated by MSMWC-MID), and each MSMWC-MID is further connected with N base learners (as specified in Section 2). The specific parameter configurations will be illuminated later. The two-layer meta algorithm is provable to enjoy a regret guarantee of O ( √ V ⋆ ln V ⋆ ) (Theorem 5 of Chen et al. [2021]), where V ⋆ is analogous to ∑ t ( ℓ t,i ⋆ -m t,i ⋆ ) 2 , but within a multi-layer context (a formal definition will be shown later). By choosing the optimism as m t,i = ⟨ ℓ t , p t ⟩ , 4 , it can recover the guarantee of ADAPT-ML-PROD,

·

∇

-

·

∈ X

although up to an O (ln V ⋆ ) factor (leading to an ̂ O ( √ V T ) meta regret), but with additional negative terms in analysis. Note that single MSMWC enjoys an O ( √ V ⋆ ln T ) bound, where the extra ln T factor would ruin the desired O (log V T ) bound for exp-concave and strongly convex functions. This is the reason for choosing a second-layer meta algorithm, overall resulting in a three-layer structure.

For clarity, we summarize the notations of the three-layer structure in Table 2. Besides, previous notions need to be extended analogously. Specifically, instead of (2.2), regret is now decomposed as

At the end of this part, we explain why we choose MSMWC as the meta algorithm. Apparently, a direct try is to keep using ADAPT-ML-PROD following Zhang et al. [2022a]. However, it is still an open problem to determine whether ADAPT-ML-PROD contains negative stability terms in the analysis, which is essential to realize effective cancellation in our problem. Another try is to explore the titled exponentially weighted average (TEWA) as the meta algorithm, following another line of research [van Erven and Koolen, 2016], as introduced in Section 1.1. Unfortunately, its stability property is also unclear. Investigating the negative stability terms in these algorithms is an important open problem, but beyond the scope of this work.

The quantity V ⋆ is defined as V ⋆ ≜ ∑ t ( ℓ t,k ⋆ ,i ⋆ -m t,k ⋆ ,i ⋆ ) 2 , where ℓ t,k,i = ⟨∇ f t ( x t ) , x t,k,i ⟩ and m t,k,i = ⟨∇ f t ( x t ) , x t ⟩ - ⟨∇ f t -1 ( x t -1 ) , x t -1 -x t -1 ,k,i ⟩ follows the same sprit as Lemma 1.

3.2.2 Exogenous Negativity: Cascaded Correction Terms

In this part, we aim to deal with the second term in (3.1) (i.e., ∑ i p t,i ∥ x t,i -x t -1 ,i ∥ 2 ). Inspired by the work of Zhao et al. [2021] on the gradient-variation dynamic regret in non-stationary online learning, 5 we incorporate exogenous correction terms for cancellation. Concretely, we inject cascaded correction terms to both top and middle layers. Specifically, the loss ℓ t and the optimism m t of MSMWC-TOP are chosen as

ℓ t,k ≜ ⟨∇ f t ( x t ) , x t,k ⟩ + λ 1 ∥ x t,k -x t -1 ,k ∥ 2 , m t,k ≜ ⟨ ̂ m t,k , p t,k ⟩ + λ 1 ∥ x t,k -x t -1 ,k ∥ 2 . (3.3) The loss ℓ t,k and optimism m t,k of MSMWC-MID are chosen similarly as

To see how the correction term works, consider a simpler problem with regret ∑ t ⟨ ℓ t , q t ⟩-∑ t ℓ t,k ⋆ . If we instead optimize the corrected loss ℓ t + b t and obtain a regret bound of R T , then moving the correction terms to the right-hand side, the original regret is at most R T -∑ t ∑ k q t,k b t,k + ∑ t b t,k ⋆ , where the negative term of -∑ t ∑ k q t,k b t,k can be leveraged for cancellation. Meanwhile, the algorithm is required to handle an extra term of ∑ t b t,k ⋆ , which only relies on the k ⋆ -th dimension. In the next part, we will discuss about how cascaded corrections and negative stability terms of meta algorithms realize effective collaboration for cancellation in the multi-layer online ensemble.

ℓ t,k,i ≜ ⟨∇ f t ( x t ) , x t,k,i ⟩ + λ 2 ∥ x t,k,i -x t -1 ,k,i ∥ 2 , m t,k,i ≜ ̂ m t,k,i + λ 2 ∥ x t,k,i -x t -1 ,k,i ∥ 2 , (3.4) where ̂ m t,k ≜ ( ̂ m t,k, 1 , . . . , ̂ m t,k,N ) and ̂ m t,k,i = ⟨∇ f t ( x t ) , x t ⟩ - ⟨∇ f t -1 ( x t -1 ) , x t -1 -x t -1 ,k,i ⟩ denotes our optimism, which uses the same idea as Lemma 1 in Section 3.1.

It is noteworthy that our work is the first to introduce correction mechanisms to universal online learning, whereas prior works use it for different purposes, such as non-stationary online learn-

ing [Zhao et al., 2021] and the multi-scale expert problem [Chen et al., 2021]. Distinctively, different from the conventional two-layer algorithmic frameworks seen in prior studies, deploying this technique to a three-layer structure necessitates a comprehensive use and extensive adaptation of it.

3.3 Overall Algorithm: A Multi-layer Online Ensemble Structure

In this part, we conclude our three-layer online ensemble approach in Algorithm 1. In Line 3, the decisions are aggregated from bottom to top for the final output. In Line 4, the learner suffers the loss of the decision, and the environments return the gradient information of the loss function. In Line 5, the algorithm constructs the surrogate losses and optimisms for the two-layer meta learner. In Line 611, the update is conducted from top to bottom. Note that in Line 8, our algorithm requires multiple, concretely O (log 2 T ) , gradient queries per round since it needs to query ∇ f t ( x t,k,i ) for each base learner, making it inefficient when the gradient evaluation is costly. To this end, in Section 4, we improve the algorithm's gradient query complexity to 1 per round, corresponding to Line 10-11, via a novel regret decomposition and carefully designed surrogate loss functions.

In Figure 1, we illustrate the detailed procedure of the collaboration in our multi-layer online ensemble approach. Specifically, we aim to deal with the positive term of ∥ x t -x t -1 ∥ 2 , which stems from the universal optimism design proposed in Section 3.1. Since x t = ∑ k q t,k x t,k , the positive term can be further decomposed into two parts: ∥ q t -q t -1 ∥ 2 1 and ∑ k q t,k ∥ x t,k -x t -1 ,k ∥ 2 , which can be canceled by the negative and correction terms in MSMWC-TOP, respectively. Note that the correction comes at the expense of an extra term of ∥ x t,k ⋆ -x t -1 ,k ⋆ ∥ 2 , which can be decomposed similarly into two parts: ∥ p t,k ⋆ -p t -1 ,k ⋆ ∥ 2 1 and ∑ i p t,k ⋆ ,i ∥ x t,k ⋆ ,i -x t -1 ,k ⋆ ,i ∥ 2 because of x t,k = ∑ i p t,k,i x t,k,i . The negative term and corrections in the middle layer (specifically, the k ⋆ -th MSMWC-MID) can be leveraged for cancellation. This correction finally generate ∥ x t,k ⋆ ,i ⋆ -x t -1 ,k ⋆ ,i ⋆ ∥ 2 , which can be handled by choosing optimistic OMD as base algorithms.

As a final remark, although it is possible to treat the two-layer meta algorithm as a single layer, its analysis will become much more complicated than that in one layer and is unsuitable for extending to more layers. In contrast, our layer-by-layer analytical approach paves a systematic and principled way for analyzing the dynamics of the online ensemble framework with even more layers.

3.4 Universal Regret Guarantees

In this part, we conclude our main theoretical result and provide several implications and applications to validate its importance and practical potential. Theorem 1 summarizes the main result, universal regret guarantees in terms of the gradient variation. The proof is deferred to Appendix B.4.

Theorem 1. Under Assumptions 1 and 2, Algorithm 1 obtains O (log V T ) , O ( d log V T ) and ̂ O ( √ V T ) regret bounds for strongly convex, exp-concave and convex functions, respectively.

Positive Term:

‖

x

t

-

x

t

-

1

‖

2

K

∑

k

=1

q

t,k

‖

x

t,k

-

x

t

-

1

,k

‖

2

︸

︷︷

︸

canceled by

-

∑

K

k

=1

q

t,k

b

t,k

+

λ

1

‖

x

t,k

/star

-

x

t

-

1

,k

/star

‖

2

‖

q

t

-

q

t

-

1

‖

2

1

︸

︷︷

︸

canceled by MSMWC-TOP

‖

p

t,k

/star

-

p

t

-

1

,k

/star

‖

2

1

︸

︷︷

︸

canceled by MSMWC-MID

N

∑

i

=1

p

t,k

/star

,i

‖

x

t,k

/star

,i

-

x

t

-

1

,k

/star

,i

‖

2

︸

︷︷

︸

canceled by

-

∑

N

i

=1

p

t,k /star ,i

b

t,k /star ,i

+

λ

2

‖

x

t,k

/star

,i

/star

-

x

t

-

1

,k

/star

,i

/star

‖

2

Middle Layer

Theorem 1 improves the results of Zhang et al. [2022a] by not only maintaining the optimal rates for strongly convex and exp-concave functions but also taking advantage of the small gradient variation for convex functions when V T ≪ F T . For example, if f 1 = . . . = f T = f and min x ∈X f ( x ) = 1 , our bound is much better since V T = 0 while F T = T . Moreover, Algorithm 1 also provably achieves universal small-loss guarantees without any algorithmic modification and thus safeguards the case when F T ≤ V T . We conclude the result below and provide the proof in Appendix B.5.

̂ Due to the connection of the gradient-variation with stochastic and adversarial OCO [Sachs et al., 2022] and game theory [Syrgkanis et al., 2015], our results can be immediately applied and achieve best known universal guarantees therein. Due to page limits, we defer applications to Appendix A.

Corollary 1. Under Assumptions 1 and 2, if f t ( · ) ≥ 0 , Algorithm 1 obtains O (log F T ) , O ( d log F T ) and O ( √ F T ) regret bounds for strongly convex, exp-concave and convex functions.

4 Improved Gradient Query Complexity

Though achieving favorable theoretical guarantees in Section 3, one caveat is that our algorithm requires O (log 2 T ) gradient queries per round since it needs to query ∇ f t ( x t,k,i ) for all k ∈ [ K ] , i ∈ [ N ] , making it computational-inefficient when the gradient evaluation is costly, e.g., in nuclear norm optimization [Ji and Ye, 2009] and mini-batch optimization [Li et al., 2014]. The same concern also appears in the approach of Zhang et al. [2022a], who provided small-loss and worst-case regret guarantees for universal online learning. By contrast, traditional algorithms such as OGD typically work under the one-gradient feedback setup, namely, they only require one gradient ∇ f t ( x t ) for the update. In light of this, it is natural to ask whether there is a universal algorithm that can maintain the desired regret guarantees while using only one gradient query per round.

We answer the question affirmatively by reducing the gradient query complexity to 1 per round. To describe the high-level idea, we first consider the case of known λ -strong convexity within a two-layer structure, e.g., adaptive regret minimization for strongly convex functions [Wang et al., 2018]. The regret can be upper-bounded as REG T ≤ [ ∑ t h t ( x t ) -∑ t h t ( x t,i ⋆ )] + [ ∑ t h t ( x t,i ⋆ ) -∑ t h t ( x ⋆ )] , where x ⋆ ∈ arg min x ∈X ∑ t f t ( x ) and h t ( x ) ≜ ⟨∇ f t ( x t ) , x ⟩ + λ ∥ x -x t ∥ 2 / 2 is a second-order surrogate of the original function f t . This yields a homogeneous surrogate for both meta and base algorithms - the meta algorithm uses the same evaluation function across all the base learners, who in turn use this function to update their base-level decisions.

In the universal online learning (i.e., unknown λ ), a natural adaptation would be a heterogeneous surrogate h t,i ( x ) ≜ ⟨∇ f t ( x t ) , x ⟩ + λ i ∥ x -x t ∥ 2 / 2 for the i -th base learner, where λ i is a guess of the true curvature λ from the candidate pool H (2.1). Consequently, the regret decomposition with respect to this surrogate becomes REG T ≤ [ ∑ t h t,i ⋆ ( x t ) -∑ t h t,i ⋆ ( x t,i ⋆ )] + [ ∑ t h t,i ⋆ ( x t,i ⋆ ) -∑ t h t,i ⋆ ( x ⋆ )] , where i ⋆ denotes the index of the best base learner whose strong convexity coefficient satisfies λ i ⋆ ≤ λ ≤ 2 λ i ⋆ . This admits heterogeneous surrogates for both meta and base regret, which

necessitates designing expert-tracking algorithms with heterogeneous inputs for different experts as required in MetaGrad [van Erven and Koolen, 2016], which is not flexible enough.

To this end, we propose a novel regret decomposition that admits homogeneous surrogates for the meta regret, making our algorithm as flexible as Zhang et al. [2022a], and heterogeneous surrogate for the base regret, making it as efficient as van Erven and Koolen [2016]. Specifically, we remain the heterogeneous surrogates h t,i ( x ) defined above for the base regret. For the meta regret, we define a homogeneous linear surrogate ℓ t ( x ) ≜ ⟨∇ f t ( x t ) , x ⟩ such that the meta regret is bounded by ∑ t ℓ t ( x t ) -∑ t ℓ t ( x t,i ⋆ ) -λ i ⋆ ∑ t ∥ x t -x t,i ⋆ ∥ 2 / 2 . As long as we can obtain a second-order bound for the regret defined on this surrogate loss, i.e., ∑ t ℓ t ( x t ) -∑ t ℓ t ( x t,i ⋆ ) , it can be canceled by the negative term from strong convexity. Overall, we decompose the regret in the following way:

For clarity, we denote this surrogate for strongly convex functions by h sc t,i ( x ) . Similarly, we define the surrogates h exp t,i ( x ) ≜ ⟨∇ f t ( x t ) , x ⟩ + α i ⟨∇ f t ( x t ) , x -x t ⟩ 2 / 2 and h c t,i ( x ) ≜ ⟨∇ f t ( x t ) , x ⟩ for α -exp-concave and convex functions, respectively. These surrogates require only one gradient ∇ f t ( x t ) within each round, thus successfully reducing the gradient query complexity.

Note that the base regret optimization requires controlling the algorithmic stability, because the empirical gradient variation ∇ h sc t,i ( x t,i ) -∇ h sc t -1 ,i ( x t -1 ,i ) = ∇ f t ( x t )+ λ i ( x t,i -x t ) -∇ f t -1 ( x t -1 ) -λ i ( x t -1 ,i -x t -1 ) not only contains the desired gradient variation, but also includes the positive stability terms of base and final decisions. Fortunately, as discussed earlier, these stability terms can be effectively addressed through our cancellation mechanism within the multi-layer online ensemble. This stands in contrast to previous two-layer algorithms with worst-case regret bounds [Zhang et al., 2018, Wang et al., 2018], where the algorithmic stability is not examined.

The efficient version is concluded in Algorithm 1 with Line 10-11, which uses only one gradient ∇ f t ( x t ) . The only algorithmic modification is that base learners update on the carefully designed surrogate functions, not the original one. We provide the regret guarantee below, which achieves the same guarantees as Theorem 1 with only one gradient per round. The proof is in Appendix C.2.

Theorem 2. Under Assumptions 1 and 2, efficient Algorithm 1 enjoys O (log V T ) , O ( d log V T ) and O ( √ V T ) for strongly convex, exp-concave and convex functions, using only one gradient per round.

̂ As a byproduct, we show that this idea can be used to recover the optimal worst-case universal guarantees using one gradient with a simpler approach and analysis, with proof in Appendix C.1.

Proposition 1. Under Assumption 1, using the above surrogate loss functions for base learners, and running ADAPT-ML-PROD as the meta learner guarantees O (log T ) , O ( d log T ) and O ( √ T ) regret bounds for strongly convex, exp-concave and convex functions, using one gradient per round.

5 Conclusion

In this paper, we obtain universal gradient-variation guarantees via a multi-layer online ensemble approach. We first propose a novel optimism design to unify various kinds of functions. Then we analyze the negative terms of the meta algorithm MSMWC and inject cascaded correction terms to improve the algorithmic stability to realize effective cancellations in the multi-layer structure. Furthermore, we provide a novel regret decomposition combined with carefully designed surrogate functions to achieve one gradient query per round. Finally, we deploy the our approach into two applications, including the stochastically extended adversarial (SEA) model and two-player zerosum games, to validate its effectiveness, and obtain best known universal guarantees therein. Due to page limits, the applications are deferred to Appendix A. Two byproducts rise in our work. The first one is negative stability terms in the analysis of MSMWC. And the second one contains a simple approach and analysis for the optimal worst-case universal guarantees, using one gradient per round.

An important open problem lies in optimality and efficiency. In the convex case, our results still exhibit an O (log V T ) gap from the optimal O ( √ V T ) result. Moreover, our algorithm necessitates O (log 2 T ) base learners, as opposed to O (log T ) base learners in two-layer structures. Whether it is possible to achieve the optimal results for all kinds of functions (strongly convex, exp-concave, convex) using a two-layer algorithm remains as an important problem for future investigation.

References

Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

Chen, H. Luo, and C. Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Proceedings of the 34th Annual Conference Computational Learning Theory (COLT) , pages 1216-1259, 2021.

Chen, W.-W. Tu, P. Zhao, and L. Zhang. Optimistic online mirror descent for bridging stochastic and adversarial online convex optimization. In Proceedings of the 40th International Conference on Machine Learning (ICML) , pages 5002-5035, 2023a.

Chen, Y.-J. Zhang, W.-W. Tu, P. Zhao, and L. Zhang. Optimistic online mirror descent for bridging stochastic and adversarial online convex optimization. ArXiv preprint , arXiv:2302.04552, 2023b.

Chiang, T. Yang, C. Lee, M. Mahdavi, C. Lu, R. Jin, and S. Zhu. Online optimization with gradual variations. In Proceedings of the 25th Annual Conference Computational Learning Theory (COLT) , pages 6.1-6.20, 2012.

Gaillard, G. Stoltz, and T. van Erven. A second-order bound with excess losses. In Proceedings of the 27th Annual Conference Computational Learning Theory (COLT) , pages 176-196, 2014.

Hazan. Introduction to Online Convex Optimization. Foundations and Trends in Optimization , 2 (3-4):157-325, 2016.

Hazan, A. Agarwal, and S. Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2-3):169-192, 2007.

Ji and J. Ye. An accelerated gradient method for trace norm minimization. In Proceedings of the 26th International Conference on Machine Learning (ICML) , pages 457-464, 2009.

Li, T. Zhang, Y. Chen, and A. J. Smola. Efficient mini-batch training for stochastic optimization. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) , pages 661-670, 2014.

Orabona. A modern introduction to online learning. ArXiv preprint , arXiv:1912.13213, 2019.

Orabona, N. Cesa-Bianchi, and C. Gentile. Beyond logarithmic bounds in online learning. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 823-831, 2012.

S. Pinsker. Information and information stability of random variables and processes. HoldenDay , 1964.

Pogodin and T. Lattimore. On first-order bounds, variance and gap-dependent bounds for adversarial bandits. In Proceedings of the 35th Conference on Uncertainty in Artifificial Intelligence (UAI) , pages 894-904, 2019.

Rakhlin and K. Sridharan. Online learning with predictable sequences. In Proceedings of the 26th Annual Conference Computational Learning Theory (COLT) , pages 993-1019, 2013a.

Rakhlin and K. Sridharan. Optimization, learning, and games with predictable sequences. In Advances in Neural Information Processing Systems 26 (NIPS) , pages 3066-3074, 2013b.

Sachs, H. Hadiji, T. van Erven, and C. Guzm´ an. Between stochastic and adversarial online convex optimization: Improved regret bounds via smoothness. In Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 691-702, 2022.

Sachs, H. Hadiji, T. van Erven, and C. Guzm´ an. Accelerated rates between stochastic and adversarial online convex optimization. ArXiv preprint , arXiv:2303.03272, 2023.

Srebro, K. Sridharan, and A. Tewari. Smoothness, low noise and fast rates. In Advances in Neural Information Processing Systems 23 (NIPS) , pages 2199-2207, 2010.

Syrgkanis, A. Agarwal, H. Luo, and R. E. Schapire. Fast convergence of regularized learning in games. In Advances in Neural Information Processing Systems 28 (NIPS) , pages 2989-2997, 2015.

van Erven and W. M. Koolen. Metagrad: Multiple learning rates in online learning. In Advances in Neural Information Processing Systems 29 (NIPS) , pages 3666-3674, 2016.

Wang, D. Zhao, and L. Zhang. Minimizing adaptive regret with one gradient per iteration. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI) , pages 2762-2768, 2018.

Wang, S. Lu, and L. Zhang. Adaptivity and optimality: A universal algorithm for online convex optimization. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence (UAI) , pages 659-668, 2019.

Wei, Y. Hong, and C. Lu. Tracking the best expert in non-stationary stochastic environments. In Advances in Neural Information Processing Systems 29 (NIPS) , pages 3972-3980, 2016.

Zhang, S. Lu, and Z.-H. Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems 31 (NeurIPS) , pages 1330-1340, 2018.

Zhang, G. Wang, J. Yi, and T. Yang. A simple yet universal strategy for online convex optimization. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 26605-26623, 2022a.

Zhang, P. Zhao, H. Luo, and Z.-H. Zhou. No-regret learning in time-varying zero-sum games. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 2677226808, 2022b.

Zhao, Y.-J. Zhang, L. Zhang, and Z.-H. Zhou. Dynamic regret of convex and smooth functions. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 12510-12520, 2020.

Zhao, Y.-J. Zhang, L. Zhang, and Z.-H. Zhou. Adaptivity and non-stationarity: Problemdependent dynamic regret for online convex optimization. ArXiv preprint , arXiv:2112.14368, 2021.

Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML) , pages 928-936, 2003.

A Applications

In this section, we validate the importance and the practical potential of our approach by applying it in two problems, including stochastically extended adversarial (SEA) model in Appendix A.1 and two-player zero-sum games in Appendix A.2.

A.1 Application I: Stochastically Extended Adversarial Model

Stochastically extended adversarial (SEA) model [Sachs et al., 2022] serves as an interpolation between stochastic and adversarial online convex optimization, where the environments choose the loss function f t from a distribution D t (i.e., f t ∼ D t ). They proposed cumulative stochastic variance σ 2 1: T , the gap between f t and its expected version F t ( · ) ≜ E f t ∼D t [ f t ( · )] , and cumulative adversarial variation Σ 2 1: T , the gap between F t ( · ) and F t -1 ( · ) , to bound the expected regret. Formally,

By specializing different σ 2 1: T and Σ 2 1: T , the SEA model can recover the adversarial and stochastic OCO setup, respectively. Specifically, setting σ 2 1: T = 0 recovers the adversarial setup, where Σ 2 1: T equals to the gradient variation V T . Besides, choosing Σ 2 1: T = 0 recovers the stochastic setup, where σ 2 1: T stands for the variance in stochastic optimization.

For smooth expected function F t ( · ) , Sachs et al. [2022] obtained an optimal O ( √ σ 2 1: T +Σ 2 1: T ) expected regret for convex functions and an O (( σ 2 max + Σ 2 max ) log T ) regret for strongly convex functions, where σ 2 max ≜ max t ∈ [ T ] max x ∈X E f t ∼D t [ ∥∇ f t ( x ) - ∇ F t ( x ) ∥ 2 ] and Σ 2 max ≜ max t ∈ [ T ] sup x ∈X ∥∇ F t ( x ) -∇ F t -1 ( x ) ∥ 2 . Later, the results are improved by Chen et al. [2023b], where the authors achieved a better O (( σ 2 max +Σ 2 max ) log( σ 2 1: T +Σ 2 1: T )) regret 6 for strongly convex functions and a new O ( d log( σ 2 1: T +Σ 2 1: T )) regret for exp-concave functions, while maintaining the optimal guarantee for convex functions. Their key observation is that the gradient variation in optimistic algorithms encodes the information of the cumulative stochastic variance σ 2 1: T and cumulative adversarial variation Σ 2 1: T , restated in Lemma 5.

A major open problem remains in the work of Chen et al. [2023b] that whether it is possible to get rid of different parameter configurations and obtain universal guarantees in the problem. In the following, we show that our approach can be directly applied and achieves almost the same guarantees as those in Chen et al. [2023b], up to a logarithmic factor in leading term. Notably, our algorithm does not require different parameter setups, thus resolving the open problem proposed by the authors. We conclude our results in Theorem 3 and the proof can be found in Appendix D.1.

Theorem 3. Under the same assumptions as Chen et al. [2023b], the efficient version of Algorithm 1 obtains O (( σ 2 max +Σ 2 max ) log( σ 2 1: T +Σ 2 1: T )) regret for strongly convex functions, O ( d log( σ 2 1: T + Σ 2 1: T )) regret for exp-concave functions and ̂ O ( √ σ 2 1: T +Σ 2 1: T ) regret for convex functions, where the O ( · ) -notation omits logarithmic factors in leading terms.

̂ Note that Sachs et al. [2023] also considered the problem of universal learning and obtained an O ( √ T log T ) regret for convex functions and an O (( σ 2 max + Σ 2 max + D 2 L 2 ) log 2 T ) regret for strongly convex functions simultaneously. We conclude the existing results in Table 3. Our results are better than theirs in two aspects: (i) for strongly convex and convex functions, our guarantees are adaptive with the problem-dependent quantities σ 1: T and Σ 1: T while theirs depends on the time horizon T ; and (ii) our algorithm achieves an additional guarantee for exp-concave functions.

A.2 Application II: Two-player Zero-sum Games

Multi-player online games [Cesa-Bianchi and Lugosi, 2006] is a versatile model that depicts the interaction of multiple players over time. Since each player is facing similar players like herself, the theoretical guarantees, e.g., the summation of all players' regret, can be better than the minimax optimal √ T bound in adversarial environments, thus achieving fast rates.

The pioneering works of Rakhlin and Sridharan [2013b], Syrgkanis et al. [2015] investigated optimistic algorithms in multi-player online games and illuminated the importance of the gradient variation. Specifically, Syrgkanis et al. [2015] proved that when each player runs an optimistic algorithm (optimistic OMD or optimistic follow the regularized leader), the summation of the regret, which serves as an upper bound for some performance measures in games, can be bounded by O (1) . The above results assume that the players are honest, i.e., they agree to run the same algorithm distributedly. In the dishonest case, where there exist players who do not follow the agreed protocol, the guarantees will degenerate to the minimax result of O ( √ T ) .

In this part, we consider the simple two-player zero-sum games as an example to validate the effectiveness of our proposed algorithm. The game can be formulated as a min-max optimization problem of min x ∈X max y ∈Y f ( x , y ) . We consider the case that the game is either bilinear (i.e., f ( x , y ) = x ⊤ A y ), or strongly convex-concave (i.e., f ( x , y ) is λ -strongly convex in x and λ -strongly concave in y ). Our algorithm can guarantee the regret summation in the honest case and the individual regret of each player in the dishonest case, without knowing the type of games in advance. We conclude our results in Theorem 4, and the proof can be found in Appendix D.2.

Theorem 4. Under Assumptions 1 and 2, for bilinear and strongly convex-concave games, the efficient version of Algorithm 1 enjoys O (1) regret summation in the honest case, ̂ O ( √ T ) and O (log T ) bounds in the dishonest case, where the O ( · ) -notation omits logarithmic factors in leading terms.

̂ Table 4 compares our approach with Syrgkanis et al. [2015], Zhang et al. [2022a]. Specifically, ours is better than Syrgkanis et al. [2015] in the strongly convex-concave games in the dishonest case due to its universality, and better than Zhang et al. [2022a] in the honest case since our approach enjoys gradient-variation bounds that are essential in achieving fast rates for regret summation.

B Proofs for Section 3

In this section, we provide proofs for Section 3, including Lemma 1, Lemma 2 and Lemma 3. For simplicity, we introduce the following notations denoting the stability of the final and intermediate decisions of the algorithm. Specifically, for any k ∈ [ K ] , i ∈ [ N ] , we define

B.1 Proof of Lemma 1

Proof. For simplicity, we define g t ≜ ∇ f t ( x t ) . For exp-concave functions, we have

For convex function, it holds that

where the last step is due to the definition of the gradient variation, finishing the proof.

B.2 Proof of Lemma 2

In this part, we analyze the negative stability terms in the MSMWC algorithm [Chen et al., 2021]. For self-containedness, we restate the update rule of MSMWC in the following:

where the bias term a t,i = 16 η t,i ( ℓ t,i -m t,i ) 2 . Below, we give a detailed proof of Lemma 2, following a similar logic flow as Lemma 1 of Chen et al. [2021], while illustrating the negative stability terms. Moreover, for generality, we investigate a more general setting of an arbitrary comparator u ∈ ∆ d and changing step sizes η t,i . This was done hoping that the negative stability term analysis would be comprehensive enough for readers interested solely in the MsMwC algorithm.

Proof of Lemma 2. To begin with, the regret with correction can be analyzed as follows:

̂ To begin with, we require an upper bound of η t,i ≤ 1 / 32 for the step sizes. To give a lower bound for TERM ( C ) , we notice that for any a , b ∈ ∆ d , where the first step follows the standard analysis of optimistic OMD, e.g., Theorem 1 of Zhao et al. [2021]. One difference of our analysis from the previous one lies in the second step, where previous work dropped the D ψ t ( p t , p t ) term while we keep it to generate the desired negative terms.

where the first inequality is due to x ln x -x +1 ≥ 0 for all x > 0 , leading to

For TERM ( B ) , the proof is similar to the previous work, where only some constants are modified. For self-containedness, we give the analysis below. Treating p t +1 as a free variable and defining where the first step is from the above derivation, the second step is due to the Pinsker's inequality [Pinsker, 1964]: KL ( a , b ) ≥ 1 2 ln 2 ∥ a -b ∥ 2 1 for any a , b ∈ ∆ d and the last step is true because of 2 x 2 +2 y 2 ≥ ( x + y ) 2 for any x, y ∈ R and ∥ a + b ∥ ≤ ∥ a ∥ + ∥ b ∥ for any a , b ∈ R d .

by the optimality of p ⋆ , we have

Since [ ∇ ψ t ( p )] i = 1 η t,i (ln p i +1) , it holds that

Therefore we have

where the first and second steps use the optimality of p ⋆ , the last inequality uses e -x -1 + x ≤ x 2 for all x ≥ -1 , requiring | 2 η t,i ( ℓ t,i -m t,i + a t,i ) | ≤ 1 . It can be satisfied by η t,i ≤ 1 / 32 and | ℓ t,i -m t,i + a t,i | ≤ 16 , where the latter requirement can be satisfied by setting a t,i = 16 η t,i ( ℓ t,i -m t,i ) 2 :

As a result, we have where the last step holds because | ℓ t,i | , | m t,i | ≤ 1 and η t,i ≤ 1 / 32 . Finally, it holds that

As for TERM ( A ) , following the same argument as Lemma 1 of Chen et al. [2021], we have

where f KL ( a, b ) ≜ a ln( a/b ) -a + b . Combining all three terms, we have

Moving the correction term ∑ T t =1 ⟨ a t , p t -u ⟩ to the right-hand side gives:

Finally, choosing u = e i ⋆ and η t,i = η i for all t ∈ [ T ] finishes the proof.

B.3 Proof of Lemma 3

In this part, we give a self-contained analysis of the two-layer meta learner, which mainly follows the Theorem 4 and Theorem 5 of Chen et al. [2021], but with additional negative stability terms.

Lemma 3. If | ℓ t,k | , | m t,k | , | ℓ t,k,i | , | m t,k,i | ≤ 1 and ( ℓ t,k -m t,k ) 2 = ⟨ ℓ t,k -m t,k , p t,k ⟩ 2 for any k ∈ [ K ] and i ∈ [ N ] , then the two-layer MSMWC algorithm satisfies where V ⋆ ≜ ∑ t ( ℓ t,k ⋆ ,i ⋆ -m t,k ⋆ ,i ⋆ ) 2 , S q T ≜ ∑ T t =2 ∥ q t -q t -1 ∥ 2 1 and S p T,k ≜ ∑ T t =2 ∥ p t,k -p t -1 ,k ∥ 2 1 measure the stability of MSMWC-TOP and MSMWC-MID .

Note that we leverage a condition of ( ℓ t,k -m t,k ) 2 = ⟨ ℓ t,k -m t,k , p t,k ⟩ 2 in Lemma 3 only to ensure a self-contained result. When using Lemma 3 (more specifically, in Theorem 2), we will verify that this condition is inherently satisfied by our algorithm.

Proof. Using Lemma 2, the regret of MSMWC-TOP can be bounded as

where the first step comes from f KL ( a, b ) = a ln a / b -a + b ≤ a ln a / b + b for a, b > 0 . The second term above (corresponding to the second term in Lemma 2) is zero since the step size is time-invariant. The first term above can be further bounded as

̂ where the first step is due to the initialization of ̂ q 1 ,k = η 2 k / ∑ K k =1 η 2 k . Plugging in the setting of η k = 1 / ( C 0 · 2 k ) , the second step holds since

Since 1 /η k = C 0 · 2 k ≥ 2 C 0 , the regret of MSMWC-TOP can be bounded by

Next, using Lemma 2 again, the regret of the k ⋆ -th MSMWC-MID, whose step size is 2 η k ⋆ , can be bounded as

where the first step is due to the initialization of p 1 ,k,i = 1 /N . Based on the observation of

where the last step use Cauchy-Schwarz inequality, combining the regret of MSMWC-TOP and the k ⋆ -th MSMWC-MID finishes the proof.

B.4 Proof of Theorem 1

In Appendix C.2, we provide the proof of Theorem 2, the gradient-variation guarantees of the efficient version of Algorithm 1. Therefore, the proof of Theorem 1 can be directly extracted from that of Theorem 2. In the following, we give a proof sketch.

Proof Sketch. The proof of the meta regret is the same as Theorem 2 since the meta learners (top and middle layer) are both optimizing the linearized regret of ∑ T t =1 ⟨∇ f t ( x t ) , x t -x t,k ⋆ ,i ⋆ ⟩ . The only difference lies in the base regret. In Theorem 1, since the base regret is defined on the original function f t , the gradient-variation bound of the base learner can be directly obtained via a blackbox analysis of the optimistic OMD algorithm, e.g., Theorem B.1 of Zhang et al. [2022a] (strongly convex), Theorem 15 (exp-concave) and Theorem 11 (convex) of Chiang et al. [2012]. The only requirement on the base learners is that they need to cancel the positive term of ∥ x t,k ⋆ ,i ⋆ -x t -1 ,k ⋆ ,i ⋆ ∥ 2 caused by the universal optimism design (Lemma 1) and cascaded correction terms (Figure 1). The above base algorithms satisfy the requirement. The negative stability term in the optimistic OMD analysis is provided in Appendix E.2.

B.5 Proof of Corollary 1

In Appendix C.3, we provide Corollary 2, the small-loss guarantees of the efficient version of Algorithm 1. The proof of Corollary 1 can be directly extracted from that of Corollary 2. We give a proof sketch below.

Proof Sketch. The proof sketch is similar to that of Theorem 1 in Appendix B.4. Specifically, the meta regret can be bounded in the same way. Since gradient-variation bounds naturally implies small-loss ones due to (E.3), the small-loss bound of the base learner can be directly obtained via a black-box analysis of the base algorithms mentioned in Appendix B.4, for different kinds of functions. The requirement of being capable of handling ∥ x t,k ⋆ ,i ⋆ -x t -1 ,k ⋆ ,i ⋆ ∥ 2 can be still satisfied, which finishes the proof sketch.

C Proofs for Section 4

In this section, we give the proofs for Section 4, including Proposition 1, Theorem 2 and Corollary 2.

C.1 Proof of Proposition 1

Proof. To handle the meta regret, we use ADAPT-ML-PROD [Gaillard et al., 2014] to optimize the linear loss ℓ t = ( ℓ t, 1 , . . . , ℓ t,N ) , where ℓ t,i ≜ ⟨∇ f t ( x t ) , x t,i ⟩ , and obtain the following secondorder bound by Corollary 4 of Gaillard et al. [2014],

For α -exp-concave functions, it holds that

where the second step uses AM-GM inequality: √ xy ≤ ax 2 + y 2 a for any x, y, a > 0 with a = α i ⋆ . To handle the base regret, by optimizing the surrogate loss function h exp t,i ⋆ using online Newton step [Hazan et al., 2007], it holds that

where G exp ≜ max x ∈X ,t ∈ [ T ] ,i ∈ [ N ] ∥∇ h exp t,i ( x ) ∥ ≤ G + GD represents the maximum gradient norm the last step is because α ≤ 2 α i ⋆ . Combining the meta and base regret, the regret can be bounded by O ( d log T ) . For λ -strongly convex functions, since it is also α = λ/G 2 exp-concave under Assumption 1 [Hazan et al., 2007, Section 2.2], the above meta regret analysis is still applicable. To optimize the base regret, by optimizing the surrogate loss function h sc t,i ⋆ using online gradient descent [Hazan et al., 2007], it holds that

where G sc ≜ max x ∈X ,t ∈ [ T ] ,i ∈ [ N ] ∥∇ h sc t,i ( x ) ∥ ≤ G + D represents the maximum gradient norm and the last step is because λ ≤ 2 λ i ⋆ . Thus the overall regret can be bounded by O (log T ) . For convex functions, the meta regret can be bounded by O ( √ T ln ln T ) , where the ln ln T factor can be omitted in the O ( · ) -notation, and the base regret can be bounded by O ( √ T ) using OGD, resulting in an O ( √ T ) regret overall, which completes the proof.

C.2 Proof of Theorem 2

Proof. We first give different decompositions for the regret, then analyze the meta regret, and finally provide the proofs for different kinds of loss functions. Some abbreviations of the stability terms are defined in (B.1).

Regret Decomposition. Denoting by x ⋆ ∈ arg min x ∈X ∑ T t =1 f t ( x ) , for exp-concave functions, where the second step uses the strong exp-concavity and the last step holds by defining surrogate loss functions h exp t,i ( x ) ≜ ⟨∇ f t ( x t ) , x ⟩ + α i 2 ⟨∇ f t ( x t ) , x -x t ⟩ 2 . Similarly, for strongly convex functions, the regret can be upper-bounded by

by defining surrogate loss functions h sc t,i ( x ) ≜ ⟨∇ f t ( x t ) , x ⟩ + λ i 2 ∥ x -x t ∥ 2 . For convex functions, the regret can be decomposed as:

Meta Regret Analysis. For the meta regret, we focus on the linearized term since the negative term by exp-concavity or strong convexity only exists in analysis. Specifically,

Next, we investigate the regret of the two-layer meta algorithm (i.e., the first two terms above). To begin with, we validate the conditions of Lemma 3. First, since

rescaling the ranges of losses and optimisms to [ -1 , 1] only add a constant multiplicative factors on the final result ( λ 1 and λ 2 only consist of constants). Second, by the definition of the losses and the optimisms, it holds that

̂ where ̂ ℓ t,k,i ≜ ⟨∇ f t ( x t ) , x t,k,i ⟩ . Using Lemma 3, it holds that where the last step holds by requiring C 0 ≥ 4 λ 1 D 2 .

Exp-concave Functions. Using Lemma 1, it holds that

As a result, the meta regret can be bounded by

where the last step holds by α i ⋆ ≤ α ≤ 2 α i ⋆ . For the base regret, according to Lemma 11, the i ⋆ -th base learner guarantees

where ¯ V T,k ⋆ ,i ⋆ = ∑ T t =2 ∥∇ h exp t,i ⋆ ( x t,k ⋆ ,i ⋆ ) -∇ h exp t -1 ,i ⋆ ( x t -1 ,k ⋆ ,i ⋆ ) ∥ 2 denotes the empirical gradient variation of the i ⋆ -th base learner. For simplicity, we denote by g t ≜ ∇ f t ( x t ) , and this term can be further decomposed and bounded as:

where the first step uses the definition of ∇ h exp t,i ( x ) = g t + α i g t g ⊤ t ( x -x t ) and the last step holds by setting C 1 = 4 + 32 D 2 G 2 and C 2 = 4 L 2 +32 D 2 G 2 L 2 +8 G 4 . Then we obtain

Plugging the above result and due to α i ⋆ ≤ α ≤ 2 α i ⋆ , the base regret can be bounded by

Combining the meta regret and base regret, the overall regret can be bounded by

Strongly Convex Functions. Since a λ -strongly convex function is also α = λ/G 2 exp-concave under Assumption 1 [Hazan et al., 2007, Section 2.2], plugging α = λ/G 2 into the above analysis, the meta regret can be bounded by

For the base regret, according to Lemma 12, the i ⋆ -th base learner guarantees

where ¯ V T,k ⋆ ,i ⋆ = ∑ T t =2 ∥∇ h sc t,i ⋆ ( x t,k ⋆ ,i ⋆ ) -∇ h sc t -1 ,i ⋆ ( x t -1 ,k ⋆ ,i ⋆ ) ∥ 2 is the empirical gradient variation and G sc ≜ max x ∈X ,t ∈ [ T ] ,i ∈ [ N ] ∥∇ h sc t,i ( x ) ∥ ≤ G + D represents the maximum gradient norm. For simplicity, we denote by g t ≜ ∇ f t ( x t ) , and the empirical gradient variation ¯ V T,k ⋆ ,i ⋆ can be further decomposed as

where the first step uses the definition of ∇ h sc ( x ) = g t + λ i ( x -x t ) . Consequently,

Plugging the above result and due to λ i ⋆ ≤ λ ≤ 2 λ i ⋆ , the base regret can be bounded by where C 3 = (4 + 4 L 2 ) G 2 sc . Combining the meta regret and base regret, the overall regret satisfies

where the first step is due to η k = 1 / ( C 0 · 2 k ) ≤ 1 / (2 C 0 ) and requiring C 0 ≥ 1 and the last step is by Lemma 7 and requiring C 0 ≥ 8 D . Thus the meta regret can be bounded by

For the base regret, according to Lemma 10, the convex base learner guarantees

where ¯ V T,k ⋆ ,i ⋆ = ∑ T t =2 ∥∇ f t ( x t ) -∇ f t -1 ( x t -1 ) ∥ 2 denotes the empirical gradient variation of the base learner. Via (E.2), the base regret can be bounded by

Combining the meta regret and base regret, the overall regret can be bounded by

where C 4 = 2 D 2 L 2 +2 G 2 +10 DL 2 .

Overall. At last, we determine the specific values of C 0 , λ 1 and λ 2 . These parameters need to satisfy the following requirements:

As a result, we set C 0 , λ 1 , λ 2 to be the minimum constants satisfying the above conditions., which completes the proof.

C.3 Proof of Corollary 2

In this part, we provide Corollary 2, which obtains the same small-loss bounds as Corollary 1, but only requires one gradient query within each round.

Corollary 2. Under Assumptions 1 and 2, if f t ( · ) ≥ 0 for all t ∈ [ T ] , the efficient version of Algorithm 1 obtains O (log F T ) , O ( d log F T ) and ̂ O ( √ F T ) regret bounds for strongly convex, expconcave and convex functions, using only one gradient per round.

Exp-concave Functions. The meta regret can be bounded the same as Theorem 1 (especially (C.1)), and thus we directly move on to the base regret. For simplicity, we denote by g t ≜ ∇ f t ( x t ) and give a different decomposition for ¯ V T,k ⋆ ,i ⋆ = ∑ T t =2 ∥∇ h exp t,i ⋆ ( x t,k ⋆ ,i ⋆ ) -∇ h exp t -1 ,i ⋆ ( x t -1 ,k ⋆ ,i ⋆ ) ∥ 2 , the empirical gradient variation of the base learner.

Proof. For simplicity, we define F x T ≜ ∑ T t =1 f t ( x t ) . Some abbreviations are given in (B.1).

where the first step by using the definition of ∇ h exp t,i ( x ) = g t + α i g t g ⊤ t ( x -x t ) , the third step uses the assumption that α ∈ [ 1 / T , 1] without loss of generality, and the last step sets C 5 = 16 L (2 + 16 D 2 G 2 ) . Consequently, it holds that

Plugging the above result and due to α i ⋆ ≤ α ≤ 2 α i ⋆ , the base regret can be bounded by

Combining the meta regret (C.1) and base regret, the overall regret can be bounded by

where the last uses the following lemma.

Lemma 4 (Corollary 5 of Orabona et al. [2012]) . If a, b, c, d, x > 0 satisfy x -d ≤ a ln( bx + c ) ,

Strongly Convex Functions. The meta regret can be bounded the same as Theorem 1 (especially (C.3)), and thus we directly move on to the base regret. For simplicity, we denote by g t ≜ ∇ f t ( x t ) and give a different decomposition for ¯ V T,k ⋆ ,i ⋆ = ∑ T t =2 ∥∇ h sc t,i ⋆ ( x t,k ⋆ ,i ⋆ ) -∇ h sc t -1 ,i ⋆ ( x t -1 ,k ⋆ ,i ⋆ ) ∥ 2 , the empirical gradient variation of the base learner.

where the first step uses the definition of ∇ h sc t,i ( x ) = g t + λ i ( x -x t ) . Consequently,

Plugging the above result and due to λ i ⋆ ≤ λ ≤ 2 λ i ⋆ , the base regret can be bounded by

Combining the meta regret (C.3) and base regret, the overall regret can be bounded by

Convex Functions. We first give a different analysis for V ⋆ ≜ ∑ T t =1 ( ℓ t,k ⋆ ,i ⋆ -m t,k ⋆ ,i ⋆ ) 2 . From Lemma 1, it holds that

Plugging the above analysis back to the meta regret, we obtain

For the base regret, using Lemma 10 and (E.3), it holds that

Combining the meta regret and base regret, the overall regret can be bounded by

Overall. At last, we determine the specific values of C 0 , λ 1 and λ 2 . These parameters need to satisfy the following requirements:

where the second step holds by requiring C 0 ≥ 8 D 2 G 2 , λ 1 ≥ 4 G 2 and γ ≥ 4 λ 2 + 8 G 2 , and the third step uses AM-GM inequality: √ xy ≤ ax 2 + y 2 a for any x, y, a > 0 with a = 1 / (2 Dη k ⋆ ) . The fourth step is by requiring 1 -1044 η k ⋆ D 2 L ≥ 1 / 2 , i.e., η k ≤ 1 / (2088 D 2 L ) for any k ∈ [ K ] , which can be satisfied by requiring C 0 ≥ 1044 D 2 L .

As a result, we set C 0 , λ 1 , λ 2 to be the minimum constants satisfying the above conditions. Besides, since the absolute values of the surrogate losses and the optimisms are bounded by problemindependent constants, as shown in the proof of Theorem 1, rescaling them to [ -1 , 1] only add a constant multiplicative factors on the final result.

D Proofs for Appendix A

This section provides the proofs for Appendix A, including Theorem 3 and Theorem 4.

D.1 Proof of Theorem 3

A key result in the work of Chen et al. [2023b] is the following decomposition for the empirical gradient variation ¯ V T ≜ ∑ T t =2 ∥∇ f t ( x t ) -∇ f t -1 ( x t -1 ) ∥ 2 , restated as follows. Lemma 5 (Lemma 4 of Chen et al. [2023b]) . Under the same assumptions as Chen et al. [2023b],

Proof of Theorem 3. The analysis is almost the same as the proof of Theorem 2. Some abbreviations of the stability terms are defined in (B.1).

Exp-concave Functions. The meta regret remains the same as (C.1) and the only difference is a slightly different decomposition of the empirical gradient variation of the base learner, i.e., ¯ V T,k ⋆ ,i ⋆ in (C.2). Specifically, it holds that

where C 5 = 8(1+8 D 2 G 2 ) , C 6 = 8 L 2 +64 D 2 G 2 L 2 +8 G 4 and the last step is by taking expectation on both sides and Lemma 5, leading to the following upper bound of the base algorithm:

Combining the meta regret and base regret, the overall regret can be bounded by where the last step holds by requiring C 0 ≥ 8 C 6 D 2 /γ , λ 1 ≥ 4 C 6 /γ and γ ≥ λ 2 +8 G 2 .

Strongly Convex Functions. The meta regret remains the same as (C.3) and the only difference is a slightly different decomposition of the empirical gradient variation of the base learner:

where σ 2 t ≜ max x ∈X E f t ∼D t [ ∥∇ f t ( x ) -∇ F t ( x ) ∥ 2 ] , Σ 2 t ≜ E [sup x ∈X ∥∇ F t ( x ) -∇ F t -1 ( x ) ∥ 2 ] , and the last step is by taking expectation on both sides and Lemma 5. Plugging the above empirical gradient variation decomposition into Lemma 12, we aim to control the following term:

The last step is due to Lemma 9, which is a generalization of Lemma 5 of Chen et al. [2023b]. Combining the meta regret and base regret, the overall regret can be bounded by

Convex Functions. We first give a slightly different decomposition for V ⋆ ≜ ∑ t ( ℓ t,k ⋆ ,i ⋆ -m t,k ⋆ ,i ⋆ ) 2 . Starting from Lemma 5, it holds that where C 7 = 64 D 2 ( L 2 + 1) 2 and the last step holds by requiring C 0 ≥ 4 C 7 D 2 , λ 1 ≥ 2 C 7 and γ ≥ 8 λ 2 +256 D 2 ( L 2 +1) .

where the last step by taking expectation on both sides. Following the same proof as in Theorem 2, taking expectation on both sides, the expected meta regret can be bounded by

where the last step is due to Lemma 7 by requiring C 0 ≥ 8 D .

For the base regret, the only difference is a slightly different decomposition of the empirical gradient variation of the base learner, i.e., ¯ V T,k ⋆ ,i ⋆ in (C.5). Specifically, using Lemma 5, it holds that

Combining the meta regret and base regret, the overall regret can be bounded by

where C 9 = 128(2 D 2 L 2 + G 2 )+10 DL and the last step holds by requiring C 0 ≥ 4 D 2 C 9 , λ 1 ≥ 2 C 9 and γ ≥ 4 λ 2 +512 G 2 .

Overall. At last, we determine the specific values of C 0 , λ 1 and λ 2 . These parameters need to satisfy the following requirements:

As a result, we set C 0 , λ 1 , λ 2 to be the minimum constants satisfying the above conditions. Besides, since the absolute values of the surrogate losses and the optimisms are bounded by problemindependent constants, as shown in the proof of Theorem 1, rescaling them to [ -1 , 1] only add a constant multiplicative factors on the final result.

D.2 Proof of Theorem 4

Proof. The proof of the dishonest case is straightforward by directly applying Theorem 2. In the following, we mainly focus on the honest case. Consider a bilinear game of f ( x , y ) ≜ x ⊤ A y and denote by g x t ≜ A y t , g y t ≜ A x t the gradients received by the x -player and y -player. The only difference from the proof of Theorem 2 is that the gradient variation can be now decomposed

where the last step holds under the mild assumption of ∥ A ∥ ≤ 1 . The gradient variation of the x -player is associated with the stability term of the y -player. When summing the regret of the players, the negative terms in the x -player's algorithm can be leveraged to cancel the gradient variation of the y -player and vise versa. As a result, all gradient variations can be canceled and the summation of regret is bounded by O (1) . As for strongly convex-concave games, since it is a special case of the bilinear games, the above derivations still hold, which completes the proof.

E Supporting Lemmas

In this section, we present several supporting lemmas used in proving our theoretical results. In Appendix E.1, we provide useful lemmas for the decomposition of two combined decisions and the parameter tuning. And in Appendix E.2, we analyze the stability of the base algorithms for different kinds of loss functions.

E.1 Useful Lemmas

In this part, we conclude some useful lemmas for bounding the gap between two combined decisions (Lemma 6), tuning the parameter (Lemma 7 and Lemma 8), and a useful summation (Lemma 9).

Lemma 6. Under Assumption 1, if x = ∑ N i =1 p i x i , y = ∑ N i =1 q i y i , where p , q ∈ ∆ N , x i , y i ∈ X for any i ∈ [ N ] , then it holds that

Lemma 7. For a step size pool of H η = { η k } k ∈ [ K ] , where η 1 = 1 2 C 0 ≥ . . . ≥ η K = 1 2 C 0 T , if C 0 ≥ √ X 2 T , there exists η ∈ H η such that

Lemma 8. Denoting by η ⋆ the optimal step size, for a step size pool of H η = { η k } k ∈ [ K ] , where η 1 = 1 2 C 0 ≥ . . . ≥ η K = 1 2 C 0 T , if C 0 ≥ 1 2 η ⋆ T , there exists η ∈ H η such that

Lemma 9. For a sequence of { a t } T t =1 and b , where a t , b > 0 for any t ∈ [ T ] , denoting by a max ≜ max t a t and A ≜ ⌈ b ∑ T t =1 a t ⌉ , we have

Proof of Lemma 6. The term of ∥ x -y ∥ 2 can be decomposed as follows:

where the first inequality is due to ( a + b ) 2 ≤ 2 a 2 +2 b 2 for any a, b ∈ R , and the last step is due to Cauchy-Schwarz inequality, Assumption 1 and the definition of ℓ 1 -norm, finishing the proof.

Proof of Lemma 7. Denoting the optimal step size by η ⋆ ≜ √ ln(4 XY ) /X , if the optimal step size satisfies η ≤ η ⋆ ≤ 2 η , where η ≤ η ⋆ can be guaranteed if C 0 ≥ √ X 2 T , then it holds that

Otherwise, if the optimal step size is greater than the maximum step size in the parameter pool, i.e., η ⋆ ≥ ( η = η 1 = 1 2 C 0 ) , then we have

Overall, it holds that

which completes the proof.

Proof of Lemma 8. The proof follows the same flow as Lemma 7.

Proof of Lemma 9. This result is inspired by Lemma 5 of Chen et al. [2023b], and we generalize it to arbitrary variables for our purpose. Specifically, we consider two cases: A < T and A ≥ T . For the first case, if A < T , it holds that

where the last step is due to ∑ T A +1 a t ≤ ∑ T t =1 a t ≤ A / b . The case of A < T can be proved similarly, which finishes the proof.

E.2 Stability Analysis of Base Algorithms

In this part, we analyze the negative stability terms in optimism OMD analysis, for convex, expconcave and strongly convex functions. For simplicity, we define the empirical gradient variation :

In the following, we provide two kinds of decompositions that are used in the analysis of gradientvariation and small-loss guarantees, respectively. First, using smoothness (Assumption 2), we have

Second, for an L -smooth and non-negative function f : X ↦→ R + , ∥∇ f ( x ) ∥ 2 2 ≤ 4 Lf ( x ) holds for any x ∈ X [Srebro et al., 2010, Lemma 3.1]. It holds that

Next we provide the regret analysis in terms of the empirical gradient-variation ¯ V T , for convex (Lemma 10), exp-concave (Lemma 11), and strongly convex (Lemma 12) functions.

Lemma 10. Under Assumptions 1 and 2, if the loss functions are convex, optimistic OGD with the following update rule:

x t = Π X [ ̂ x t -η t m t ] , ̂ x t +1 = Π X [ ̂ x t -η t ∇ f t ( x t )] , (E.4) where Π X [ x ] ≜ arg min y ∈X ∥ x -y ∥ , m t = ∇ f t -1 ( x t -1 ) and η t = min { D/ √ 1 + ¯ V t -1 , 1 / γ } , enjoys the following empirical gradient-variation bound:

Lemma 11. Under Assumptions 1 and 2, if the loss functions are α -exp-concave, optimistic OMD with the following update rule:

where ψ t ( · ) = 1 2 ∥· ∥ 2 U t , 7 U t = γI + αG 2 2 I + α 2 ∑ t -1 s =1 ∇ f s ( x s ) ∇ f s ( x s ) ⊤ and m t = ∇ f t -1 ( x t -1 ) , enjoys the following empirical gradient-variation bound:

Lemma 12. Under Assumptions 1 and 2, if the loss functions are λ -strongly convex, optimistic OGD (E.4) with η t = 2 / ( γ + λt ) and m t = ∇ f t -1 ( x t -1 ) , enjoys the following empirical gradientvariation bound:

Proof of Lemma 10. The proof mainly follows Theorem 11 of Chiang et al. [2012]. Following the standard analysis of optimistic OMD, e.g., Theorem 1 of Zhao et al. [2021], it holds that

where x ⋆ ∈ arg min x ∈X ∑ T t =1 f t ( x ) and ψ ( · ) ≜ 1 2 ∥ · ∥ 2 . The adaptivity term satisfies where the last step uses ∑ T t =1 a t / √ 1 + ∑ t -1 s =1 a s ≤ 4 √ 1 + ∑ T t =1 a t +max t ∈ [ T ] a t [Pogodin and Lattimore, 2019, Lemma 4.8]. Next, we move on to the optimality gap,

Finally, we analyze the stability term,

Combining the above inequalities completes the proof.

Proof of Lemma 11. The proof mainly follows Theorem 15 of Chiang et al. [2012]. Denoting by x ⋆ ∈ arg min x ∈X ∑ T t =1 f t ( x ) , it holds that

Using the stability of optimistic OMD [Chiang et al., 2012, Proposition 7], the above term can be further bounded by

By choosing the optimism as m t = ∇ f t -1 ( x t -1 ) , the above term can be consequently bounded due to Lemma 19 of Chiang et al. [2012]:

The last step is to analyze the negative stability term:

which completes the proof.

Proof of Lemma 12. The proof mainly follows Theorem 3 of Chen et al. [2023b]. Following the almost the same regret decomposition in Lemma 10, it holds that

First, we analyze the optimality gap,

We handle the last term by leveraging the negative term imported by strong convexity:

Finally, we lower-bound the stability term as

Choosing the optimism as m t = ∇ f t -1 ( x t -1 ) , we have

To analyze the first term above, we follow the similar argument of Chen et al. [2023b]. By Lemma 9 with a t = ∥∇ f t ( x t ) -∇ f t -1 ( x t -1 ) ∥ 2 , a max = 4 G 2 , A = ⌈ λ ¯ V T ⌉ , and b = λ , it holds that

Since η t = 2 / ( γ + λt ) ≤ 2 / ( λt ) , combining existing results, we have

which completes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors aim to develop an algorithm for online convex optimization that can adapt to the specific curvature of the loss function without knowing it in advance. The goal is to develop an algorithm that can adapt to strongly convex, exp-concave and convex loss functions dynamically, while also deriving some problem-dependent guarantees. They also ensure that their method is computationally efficient as it only requires one gradient query per round.\n\nTo do so, they present a three layer algorithm that estimates both the type of loss functions and its parameters. Crucially, this kind of multi-layer approach generally comes at the cost of logarithmic factors. These three algorithms follow pre existing structures (MSMWC (Chen et al. 2021) for the meta algorithms and an optimistic Online Mirror Descent for the base algorithm) but the authors rely on several tricks and refinements of the analysis to exploit negative terms in the regret analysis and obtain competitive regret bounds. \n\nFinally, they also provide a novel decomposition of the regret which allows to reduce the number of gradients queries from $O(\\log^2 T)$ to $O(1)$ at each round. This result is of independent interest, as they show that it can be used with different meta algorithms, such as Adapt-ML-Prod (van Erven and  Koolen 2016) to recover worst-case guarantees (whereas the main results of the paper are stated in terms of problem dependent terms).\n\nSTRENGTHS:\nThis paper provides significant contributions to the field of OCO. Achieving optimal results simultaneously across several environments has been an important topic in learning theory in the past few years, and, by providing universal guarantees, this paper fits in this line of work. The presented algorithm builds upon existing results and provides non-trivial improvements to their analysis. These technical contributions are interesting and likely to be build upon in the future. \n\nIt is worth noting that this paper is particularly well written. The body of the paper focuses on providing detailed explanations for the choice of methods, and the authors take a particular care in higlighting the connections and differences with the rest of the litterature. Discussionss such as in Section 3.2.1 which highlight the different approaches that could have been considered and were discarded are not that common and should be encouraged. The proofs, deferred to the appendix, are clear and detailed. Appendix A provides several applications for the results, showing that the method can be used to bridge the gap between stochastic and adversarial online convex optimization and for two-player zero-sum games. In both cases, the proposed algorithm obtains regret bounds that are at least matching up to log-factors and sometimes improving upon the state of the art while considering a broader setting.\n\nWEAKNESSES:\nWhile many comparisons with other results in the litterature have been provided, it would be interesting to get comparisons with lower bounds, in particular for the small loss bounds that depend in F and V. \nAs the results achieved by the current method are matching these of Zhang et al. (2022), though with a better runtime complexity, it could also be interesting to get experimental results to compare them.\n\nQUESTIONS:\nCould you discuss if and how you think these results could be further improved?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper put together and designed new techniques to reach a high level of adaptivity in online convex optimization. It is a timely contribution that is useful to put together various algorithms optimal for different problem setups and to better adapt to the characteristic of each setup.

STRENGTHS:
This is an excellent paper in presenting the results and the qualitative explanation of the techniques or proofs. The adaptivity contribution of the paper is also very satisfying in terms of general knowledge of machine learning.

WEAKNESSES:
I do not see any weakness in this paper. However, I have not checked the proofs in the appendix.

QUESTIONS:
No question.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes algorithms that have data-dependent regrets and adaptive to the loss function classes. The algorithm improves on Zhang et al., 2022 in the sense that 1) it also achieves a gradient variation bound for convex Lipschitz losses and 2) it requires only one gradient evaluation at each round. The first is achieved by the introduction of optimism. The second is achieved by appropriately designed surrogate functions for the base learners. \n\n---\n\nThe authors have addressed my questions. I keep the original rating.\n\nSTRENGTHS:\n1. The algorithm is new and improves on Zhang et al., 2022 (see the summary). \n2. The paper clearly addresses the ideas behind algorithm design and analysis.\n\nWEAKNESSES:\n1. **Comparison of complexity.** Please also compare the number of the instances of the base learners required for each algorithm in Table 1. This information helps evaluate the computational complexity. \n2. **Algorithm 1.** It is said that the base learners are set \"as specified in Section 2,\" but Section 2 (preliminaries) does not give any algorithm. \n1. **Presentation.**\n    - Section 4 is presented in a hasty way. The reader may not be able to understand the ideas without reading Appendix C. \n    - Table 1: The Lipschitzness and smoothness assumptions are missing. The assumptions may not hold in, e.g., online portfolio selection. \n    - Ln. 35: That \"the learner requires to know the function information (type and curvature) in advance to select suitable algorithms\" only holds for textbook OCO algorithms and not general OCO. \n    - The three aspects in Ln. 57--60 can be skipped as that will be addressed in detail in Ln. 79--89. \n    - Ln. 58: The term \"small-loss bound\" has not been explained before. It is better to explain it as the term is not common sense to general machine learning researchers. \n    - Ln. 125: That $N \\approx 1 + \\log T + \\log T$ looks weird. Perhaps there is some typo. \n2. **Typos.**\n    - Ln. 57: aspect*s*\n    - Ln. 219: *for* our purpose\n\nQUESTIONS:\nPlease address the first two weaknesses and the possible typo in Ln. 125.\n\nLIMITATIONS:\nThis is a theory paper. All assumptions are explicitly written. The potential issue regarding the computational complexity has been raised in the weakness block above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new algorithm for online learning is provided, together with improved regret bounds compared to the state-of-the-art: \n\nThe new regret guarantees are summarised in table 1 on page 2: in the case of a strongly convex function, the regret guarantee is of $O(min(\\log(V_T),\\log (F_T)))$ where $V_T$ and $F_T$ are the gradient variation and cumulative loss of the best comparator respectively (cf. equation on lines 53 to 54). In the case of exp-concave functions, the rate is the same, whereas in the case of a convex function, the rate is $\\widetilde{O}(\\min(\\sqrt{V_t},\\sqrt{F_T})$ instead. In terms of the regret bound, only the convex case is an improvement over the existing results ([3], $O(\\sqrt{F_T})$). An additional claimed advantage is that the gradient query is reduced to $O(1)$ instead of $O(\\log(T))$: this is valid for an improved algorithm (separate from the main algorithm presented in the paper) with additional surrogate losses, which is only presented in the appendix on page 18 (Algorithm 2)\n\nThe main algorithm itself is a combination of several existing techniques: at each iteration, a set of $KN$ experts $(k,i)\\in [K]\\times [N]$ each consisting of different gradient descent methods is maintained. Each expert computes the gradient update as per Optimistic OMD [3]. Then, for each fixed $k$, the experts $(k,i')$ for $i'\\leq N$ are aggregated exactly as in MsMwC (cf. [1] and equation (2.3)), then the resulting experts are aggregated again via another use of MsMwC, with different parameters. The proof is a very long list of re-expressions of the regret via various existing results and additional calculations, drawing a lot of inspiration from [1], which itself improves on  techniques in [2].\n\n\n\n\n=================Post-Rebuttal=============\n\nAfter discussing with the authors, I am convinced that the presentation issues are theoretically fixable, and I still think the amount of work is adequate. Thus, I will keep my score and hope the authors fix all the minor errors as promised and try hard to make the paper more accessible to a broader audience. \n\n==============\n\n\n\n\n**References**\n\n[1]  Sijia Chen, Wei-Wei Tu, Peng Zhao and  Lijun Zhang. Optimistic Online Mirror Descent for Bridging Stochastic and Adversarial Online Convex Optimization. ICML 2023.\n\n[2] Pierre Gaillard,  Gilles Stoltz and Tim Van Erven. A Second-order Bound with Excess Losses. COLT 2014. \n\n[3]  Lijun Zhang, Guanghui Wang, Jinfeng Yi and Tianbao Yang.  A Simple yet Universal Strategy for Online Convex Optimization. ICML 2022.\n\nSTRENGTHS:\nThis is a highly technical and long paper and it is clear the authors know what they are doing. The parts of the proofs I managed to check are generally correct with only minor typos (though they are not reader friendly).\n\nWEAKNESSES:\nIn my opinion, the main paper is poorly written. I understand that the authors are trying to explain the intuition for why they construct their algorithm in this way, but there isn't enough context or revision of existing methods to make the paper readable. \nThe paper reads like a soliloquy of a researcher trying to solve the problem at hand: for instance, in lines 140 to 148, the authors are saying \"if we used Adapt-ML [2], bad things ill happen, what can be done about this?\" then later in Section 3, their solution is proposed, which consists in using [3] instead, with a purportedly novel choice of optimism.   \n\nSee also how the authors are using expressions like \"Another try is to\" on line 206. This is an issue not just in terms of grammar but also because it occurs in the context of a vague exposition that follows the stream of consciousness of the author rather than trying to present a cogent story.\n\n\nHere are examples of more concrete **issues with the presentation**: \n\n\n**1** There is a lot of talk in the main paper of \"how to cancel out terms\" which comes a long time before the part where the authors introduce their main algorithm. When they finally decide to do so on page 7, it is only the basic version of the algorithm (not the one with the $O(1)$ gradient calls) and it still vaguely refers to \"the base learners described in Section 2\", which I can only imagine refers to what is introduced in line 168 (section 3.1), but is still presented as new in line 177 \"to unify all kinds of functions, we propose....\"\n\n\n**2** The concept of \"optimism\" isn't properly introduced either in the main or in the supplementary; the only definition is on line 72 in the intro: \"a hyperparameter encoding historical information\", which I am given to understand means slightly different things in different algorithms. A few more words of explanation here would be nice. Definitely, the base learners should be defined properly in the main text. In the current form, not even the reference to [3] is especially clear outside of Table 1. The same goes for the section names, terms such as \"exogeneous negativity\" are not very informative. In footnote 2, the authors even speak of \"implementing $m_t$\", which is definitely very informal and should be explained more clearly. \n\n\n**3** Note also that the Bregman Divergence is note defined in lines 486 to 487 or 211-212 (page 6), despite the fact that it would make the description more complete and that it is indeed defined in the reference [1] when the same equation is introduced. \n\n**4** In the proof of Lemma 2, the first equation is hard to understand without background in the field. the only explanation is \"by standard analysis of optimistic OMD\". It would be nice to say that this refers specifically to Theorem 1 of [4], which is explained better later in the appendix when the argument arises again (line 767 page 30). Since it is natural for the reader to read the proof of Lemma 2 first, the more detailed explanation should be present there as well. \n\n**5** Similarly, the proof of Lemma 1 on page 14 uses equation E.1 from page 29, which is completely unnecessary as E.1. is derived from first principles. \n\n**6** The proof of Lemma 2 is done for \"an arbitrary comparator $u\\in\\Delta$, but the concept of comparator isn't really defined there, though it becomes clear from context after some effort. This also seems to run counter to the fact that in the main paper the authors write elsewhere in the paper that they \"focus on the proof for fixed learning rate\", which seems not to be the case in the case of the proof of Lemma 2. \n\n**7** What does the notation $\\|.\\|_{U_t}$ mean on line 761? Is it explained somewhere? \n\n\n\n\nI understand that I am not an expert in the field and my difficulties understanding the bigger picture is mostly due to this, but I do feel like in the case of well written papers, I am often able to make better progress, in less time, towards understanding more complex papers than this one in equally distant areas. However, I am not sure, it may be that the prerequisites are intrinsically larger.\n\n\n\n**References**\n\n[1]  Sijia Chen, Wei-Wei Tu, Peng Zhao and  Lijun Zhang. Optimistic Online Mirror Descent for Bridging Stochastic and Adversarial Online Convex Optimization. ICML 2023.\n\n[2] Pierre Gaillard,  Gilles Stoltz and Tim Van Erven. A Second-order Bound with Excess Losses. COLT 2014. \n\n[3]  Lijun Zhang, Guanghui Wang, Jinfeng Yi and Tianbao Yang.  A Simple yet Universal Strategy for Online Convex Optimization. ICML 2022. \n\n[4]   Zhao, Peng ; Zhang, Yu-Jie ; Zhang, Lijun ; Zhou, Zhi-Hua.  Adaptivity and Non-stationarity: Problem-dependent Dynamic Regret for Online Convex Optimization. ArXiv. May 2023.\n\nQUESTIONS:\n**1** Can you describe the base learner from [3] concisely?  Perhaps rewrite the main paper in a more careful way? \n\n**2** I am a bit confused about what you mean by the $O$ notation sometimes: which constants are considered to be \"$O(1)$ exactly? In page 31, it seems like $\\gamma$ is treated as a variable (not $O(1)$), but it is less clear in the case of $D$ and $G$: at the end of the proof of Lemma 10, **the $O(1)$ notation hides a term of $4DG^2$**. If you can absorb this into the $O(1)$ notation, why do you still need to write $\\gamma D^2$? I can sort of see that the reason is that there is a $\\gamma$ in there, but providing a bit of context might help. \n\n**3** Do you really mean an equal sign at the second equality (first line) describing the Adaptivity term in page 30? it seems like if $1/\\gamma<\\frac{D}{\\sqrt{1+\\bar{V_T}}}$, **the equality wouldn't hold**. More worrying still, the inequality would be in the wrong direction. \n\n**4** In lemma 3, you make the assumption that $(\\ell_{t,k -m_{t,k}})^2 =\\langle l_{t,k}-m_{t,k},p_{t,k}\\rangle^2$. Could you provide more context there? \n\n**5** Could you summarize the difference between your work and [1] more concisely? Why was the base learner [3] not used directly in [1]? What would happen if we used the correct base learner but did not use the hierarchical approach from your method? \n\n\n\n**Minor mathematical errors:** \n\n**1** I think the term  $-8\\sum_{t=2}^T \\|p_t-p_{t-1}\\|$ should be $-4\\sum_{t=2}^T \\|p_t-p_{t-1}\\|$ in Lemma 2. Indeed, the definition of \"term C\" excludes the factor of $1/2$ at the bottom of page 14 (as evidenced not just by the curly bracket but also by the use of Pinsker's inequality in line 498 as there would need to be a factor of 16 instead of 32 if the 1/2 were included): this factor of $1/2$ needs to be reintroduced in line 513 on page 16.  It would also be nice to quote Pinsker's inequality and perhaps add a citation at line 499 (page 15). \n\n**2** In the calculations on page 15, lines 497-498, at the third equality on the first line, the $b_i$ should be at the numerator instead of the denominator.  \n\n\n\n**3** In the second line of the series of equations for the optimality gap in the proof of Lemma 10 on page 30, I think it should be $\\leq$ instead of $=$ because there is a term of $-\\|x^*-\\hat{x}_{T+1}\\|^2\\frac{1}{2\\eta_T}$ which is dropped out. \n\n\n**4** In the statement of Lemma 5, it should be $\\|p-q\\|_1^2$ instead of  $\\|p-p\\|_1^2$ \n\n\n**5** In the proof of Lemma 1 on page 14, at the second line of equalities after line 482, I think it should be $\\leq $ and not $=$ because of a missing $\\langle g_T-g_{T+1},x_T-x_{T,i*}\\rangle$ term which is added to make the sum look simpler. \n\n**6** In the definitions of the iterations of $p_t$ and $q_t$ from [1] in line 486 page 14 and also in line 211 page 6, the last term should be $D(p,q_{t-1})$ instead of $D(p,q_{t})$, consistent with equation 6 in [1].  **NB**: I have used q instead of \\hat{p} due to issues with markdown.\n\n\n\n\n**Typos**\n\n\n\n\nline 7: \"the O notation\" (missing \"the\")\n\nline 146: constants=> constant\n\nline 161, same thing\n\nline 205 \"it is still open that whether..>\" ==> \"it is still an open problem to determine whether\" \n\nline 217 \"same of\" ==> \"same as\"\n\nline 252 \"same of\" ==> \"same as\"\n\nline 488: \"follows the similar logic of\" ==> \"follows a similar logic as\" \n\nline 506 \"which equals to\" ==> \"which is equivalent to\" (this somewhat changes the meaning and makes the line more understandable. Also, one might want to consider turning some of these inline equations into {align} blocks).\n\nline 739 \"it the optimal step size\" ==> \"if the optimal step size\"\n\nLIMITATIONS:\nThe improvement compared to [3] is somewhat disappointing compared to the much larger complexity of the algorithm: it seems that eh main improvement is to use a single algorithm for all types of functions and to reach similar rates, with a better rate in the case of convex functions. This is interesting theoretically but the algorithm seems too unwieldy to be used in either practice or future work. \n\nI cannot fully vouch for the correctness as I often had to read the supplementary line by line without fully grasping the bigger picture. It is also hard to vouch for the originality compared to [3] as I am not familiar enough with it. Nevertheless, this seems like solid work, assuming the presentation was improved.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}

comments: SUMMARY:
This paper proposes an online convex optimization algorithm that is adaptive in two ways to the environment.
Specifically, it can achieve logarithmic regrets for problems with good properties such as strong convexity and exp-concavity,
as well as $\sqrt{T}$ regret in the worst-case.
Further,
it achieves improved regret bounds for problems with small variation of the function gradients.
In addition, it works with only one gradient call per round.

STRENGTHS:
- The paper is well structured and easy to follow.
- Existing studies are adequately reviewed and the paper is informative for the reader.
- The paper clearly explains the structure of the proposed method, the idea of the proof, and its implication in great detail.

WEAKNESSES:
Compared to the study by Zhang et al. [2022], the magnitude of the contribution of this paper appears somewhat limited.
Although the design of the algorithm is quite sophisticated, the main component is a combination of already known techniques.
It would be better to have a more detailed description of the challenges in algorithm design and analysis, as well as the new techniques proposed in this paper.

QUESTIONS:
Can you address the concerns described in Weaknesses above?

LIMITATIONS:
I have no concerns about the limitations and potential negative societal impact are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-25

comments: SUMMARY:
Addressing to the drawback of Zhang et al. [2022], which does not enjoy gradient-variation bounds for convex functions, using an optimistic version of ADAPT-ML-PROD with a second-order bound of  [Wei et al.,  2016],three-layer online ensemble structure with two-layer meta learner running  MSMWC [Chen et al., 2021] with different parameter configurations, and each MSMWC-MID is further connected with N base learners to explore the unknown function information, is developed.$\mathcal{O}(\ln V_T)$, $\mathcal{O}(d \ln V_T)$ and $\hat{\mathcal{O}}(\sqrt{V_T})$ regret bounds for strongly convex, exp-concave and convex loss functions is obtained. two different levels of adaptivity with problem-dependent gradient variations, e.g., adaptive to unknown function information and benign Environments are fulfilled.

STRENGTHS:
In Zhang et al. [2022], the convex case has not been addressed. a two-layer framework In Zhang et al. [2022] is generalized to three-layer online ensemble structure. a single algorithm with gradient-variation bounds for all kinds of functions is designed.

WEAKNESSES:
1 the derived results substantially depend on Zhang et al. [2022], [Chen et al., 2021], Zhao et al. [2021] and [Wei et al., 2016].
2 The results of the algorithm still belong to the class of MSMWC [Chen et al., 2021]. the decomposed regret and injecting cascaded correction terms to both the top and middle layer make the work replicate published findings without adding substantial knowledge, leading to lack of novelty.

QUESTIONS:
I believe that exists other gradient-variation bounds, which is similar to [Wei et al., 2016], Can better results be obtained by means of it?

LIMITATIONS:
the authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-26