Title: A Simple and Optimal Approach for Universal Online Learning with Gradient Variations
Venue: NeurIPS 2024
Accepted: yes
Keywords: universal online learning

gradient-variation regret
Abstract: We investigate the problem of universal online learning with gradient-variation regret. Universal online learning aims to achieve regret guarantees without prior knowledge of the curvature of the online functions. Moreover, we study the problem-dependent gradient-variation regret as it plays a crucial role in bridging stochastic and adversarial optimization as well as game theory. In this work, we design a universal approach with the *optimal* gradient-variation regret simultaneously for strongly convex, exp-concave, and convex functions, thus addressing an open problem highlighted by [Yan et al. [2023]](https://openreview.net/forum?id=AA1xrgAP5z). Our approach is *simple* since it is algorithmically efficient-to-implement with a two-layer online ensemble structure and only $1$ gradient query per round, and theoretically easy-to-analyze with a novel and alternative analysis to the gradient-variation regret. Concretely, previous works on gradient variations require controlling the algorithmic stability, which is challenging and leads to sub-optimal regret and less efficient algorithm design. Our analysis overcomes this issue by using a Bregman divergence negative term from linearization and a useful smoothness property.

A Simple and Optimal Approach for Universal Online Learning with Gradient Variations

Abstract

We investigate the problem of universal online learning with gradient-variation regret. Universal online learning aims to achieve regret guarantees without the prior knowledge of the curvature of the online functions. Moreover, we study the problem-dependent gradient-variation regret as it plays a crucial role in bridging stochastic and adversarial optimization as well as game theory. In this work, we design a universal approach with the optimal gradient-variation regret simultaneously for strongly convex, exp-concave, and convex functions, thus addressing an open problem highlighted by Yan et al. [2023]. Our approach is simple since it is algorithmically efficient-to-implement with a two-layer online ensemble structure and only 1 gradient query per round, and theoretically easy-to-analyze with a novel and alternative analysis to the gradient-variation regret. Concretely, previous works on gradient variations require controlling the algorithmic stability, which is challenging and leads to sub-optimal regret and less efficient algorithm design. Our analysis overcomes this issue by using a Bregman divergence negative term from linearization and a useful smoothness property.

1 Introduction

Online convex optimization (OCO) models a sequential T -round game between an online learner and the environments [Hazan, 2016, Orabona, 2019]. In each round t ∈ [ T ] , the learner selects a decision x t from a convex compact set X ⊆ R d . Simultaneously, the environments adversarially choose a convex loss function f t : X ↦→ R . Subsequently, the learner suffers a loss of f t ( x t ) , receives feedback on the function f t ( · ) , and updates her decision to x t +1 . In OCO, the learner aims to optimize the game-theoretical performance measure known as regret [Cesa-Bianchi and Lugosi, 2006], which is formally defined as

It represents the learner's excess cumulative loss compared with the best fixed comparator in hindsight.

In OCO, function curvatures play an important role in the best attainable regret bounds. Traditional studies examine three types of curvatures: convexity, exp-concavity, and strong convexity. Specifically, for convex functions, online gradient descent (OGD) achieves O ( √ T ) regret [Zinkevich, 2003]; for α -exp-concave functions, online Newton step assuming α is known obtains O ( d α log T ) regret [Hazan et al., 2007]; and for λ -strongly convex functions, OGD with known λ attains O ( 1 λ log T ) [Hazan et al., 2007]. These results are shown to be minimax optimal [Ordentlich and Cover, 1998, Abernethy et al., 2008]. Recent studies further strengthen them by introducing two levels of adaptivity .

High-level Adaptivity to Unknown Curvatures. Traditional studies require function curvature information in advance to select suitable algorithms for provable bounds. However, the information could be hard to access in real-world applications. To this end, a line of research aims to design a single universal algorithm that does not require the curvature information while achieving the same regret guarantees as if knowing it, achieving the adaptivity to unknown curvatures. The pioneering work of MetaGrad [van Erven and Koolen, 2016] proposed carefully designed surrogate functions and achieved O ( d α log T ) for α -exp-concave functions and O ( √ T ) for convex functions. Subsequently, Wang et al. [2019] obtained the optimal O ( 1 λ log T ) regret for λ -strongly convex functions, while maintaining the optimal rates in the other cases. Another remarkable progress of Zhang et al. [2022a] proposed a flexible framework with simplified analyses and further enhanced the minimax results using smoothness. We provide a detailed introduction of this work in Section 2.2.

Low-level Adaptivity to Gradient Variation. Although the regret guarantees based on the time horizon T are optimal in the minimax sense, in this work we are interested in achieving the gradientvariation regret [Chiang et al., 2012, Yang et al., 2014], which replaces the dependence of the time horizon T by the gradient variation quantity defined in the following:

Under smoothness assumptions, the minimax regret can be improved to O ( 1 λ log V T ) , O ( d α log V T ) , and O ( √ V T ) for λ -strongly convex, α -exp-concave, and convex functions, respectively. In this work, continuing previous gradient-variation online learning results [Zhao et al., 2020, 2024, Zhang et al., 2022b, Chen et al., 2023], we focus on the gradient-variation regret for the following reasons: (i) gradient-variation bounds safeguard the minimax guarantees. Besides, as demonstrated by Zhao et al. [2024], the gradient-variation regret is more fundamental than another well-known problemdependent quantity known as the small loss F T ≜ min x ∈X ∑ t ≤ T f t ( x ) [Srebro et al., 2010, Orabona et al., 2012] since gradient-variation regret can imply small-loss bounds directly in analysis; (ii) the gradient variation plays a crucial role in bridging adversarial and stochastic optimization [Sachs et al., 2022]; and (iii) the gradient-variation regret can be used to achieve fast rates in multi-player games [Syrgkanis et al., 2015, Zhang et al., 2022b]. More detailed explanations of the importance of achieving such adaptivity are provided at the end of this section.

Motivated by the aforementioned two levels of adaptivity, we focus on the problem of achieving universal gradient-variation regret, i.e., designing a single universal approach with gradient-variation regret across different curvature types without the prior knowledge of them. For this problem, Zhang et al. [2022a] achieved partial results of O ( 1 λ log V T ) , O ( d α log V T ) , O ( √ T ) for λ -strongly convex, α -exp-concave, and convex functions, respectively. Subsequently, Yan et al. [2023] proposed a carefully designed three-layer online ensemble approach to stabilize the algorithm and improved the convex result to O ( √ V T log V T ) , achieving the first universal gradient-variation guarantee. Although optimal for strongly convex and exp-concave functions, their results still exhibit a gap with the optimal O ( √ V T ) regret in the convex case. Here 'optimal' refers to matching the best known results with curvature information since problem-dependent lower bound cannot be easily obtained. The only lower bound we are aware of is Ω( √ V T ) for convex functions [Yang et al., 2014, Remark 5].

To handle the uncertainty, online ensemble is commonly employed and proven effective in enhancing the robustness [Zhou, 2012, Zhao, 2021], such as adaptive regret minimization [Hazan and Seshadhri, 2007, Daniely et al., 2015, Zhang et al., 2019], dynamic regret minimization [Zhang et al., 2018, Zhao et al., 2020, 2024], and universal online learning [van Erven and Koolen, 2016, Zhang et al., 2022a, Yan et al., 2023]. Concretely, an online ensemble algorithm contains multiple base learners for exploring the environments and a meta learner for ensemble. In universal online learning, base learners make guesses on the curvature information, and the meta learner tracks the best base learner (i.e., with the most accurate guess) on the fly.

Due to the deployment of the online ensemble framework, computational efficiency has become a point of concern, with two essential factors. The first is the number of base learners , because each independently runs an online learning algorithm that involves time-consuming gradient computations and projections. The second factor is the number of gradient queries , especially when the gradient evaluation is costly, e.g., in nuclear norm optimization [Ji and Ye, 2009] and mini-batch optimization [Li et al., 2014]. In universal online learning, an 'efficient' algorithm is expected to adopt only O (log T ) base learners, which is inherent to the online ensemble design, and only 1 gradient query per round, matching the gradient query complexity of standard OGD. In terms of this metric, Zhang et al. [2022a] employed O (log T ) base learners, but required O (log T ) gradient queries per round. Yan et al. [2023] used only 1 gradient query per round but required O ((log T ) 2 ) base learners (caused by their three-layer algorithm design), resulting in reduced efficiency.

Results. Motivated by the above considerations of optimality and efficiency, in this work, we propose a simple universal approach that achieves the optimal O ( 1 λ log V T ) , O ( d α log V T ) , and O ( √ V T ) regret simultaneously for λ -strongly convex, α -exp-concave, and convex functions, and is efficient with one gradient query per round and O (log T ) base learners, resolving a major open problem highlighted by Yan et al. [2023]. We summarize our theoretical results in Theorem 1, and compare our results with existing ones in Table 1. Furthermore, we validate the effectiveness of our approach by: (i) showing that our universal gradient-variation regret directly implies the optimal universal small-loss regret in analysis without any algorithm modifications; and (ii) applying them to the stochastically extended adversarial (SEA) model [Sachs et al., 2022], an intermediate framework between stochastic and adversarial optimization. We achieve the same state-of-the-art guarantees as Chen et al. [2024], but without curvature information. The details are provided in Section 4.

Techniques. Our technical contributions include two simple and novel analyses. First, the key to gradient-variation regret is to analyze its empirical version, formally, ∥∇ f t ( x t ) -∇ f t -1 ( x t -1 ) ∥ 2 2 . The previous approach to addressing this term involves controlling the algorithmic stability ∥ x t -x t -1 ∥ 2 2 , which is highly challenging in universal online learning, leading to sub-optimal results and less efficient algorithm design [Yan et al., 2023]. In this work, we overcome this issue via a novel analysis by a useful smoothness property and a Bregman divergence negative term from linearization, where the latter is inspired by the recent advance in stochastic optimization [Joulani et al., 2020]. Second, we adopt the surrogate functions proposed by Yan et al. [2023] to reduce the gradient query complexity, and provide a novel analysis for the empirical gradient variation based on the surrogates. A technical comparison with previous works is provided in Section 4.

Organization. The rest of the paper is structured as follows. Section 2 introduces preliminaries and a general framework for universal online learning. Section 3 presents our efficient approach with the optimal universal gradient-variation regret. Section 4 presents the implication and application of our results. Finally Section 5 concludes the paper. All the proofs can be found in the appendices.

2 Preliminaries

In this section, we introduce some preliminary knowledge, including notations, assumptions, definitions, and a general framework of universal online learning.

2.1 Notations, Assumptions, and Definitions

For simplicity, we use ∥ · ∥ for ∥ · ∥ 2 by default and use a ≲ b or a = O ( b ) if there exists a constant C < ∞ such that a/b ≤ C . In the following, we introduce the assumptions used in this work.

Assumption 1 (Boundedness) . For any x , y ∈ X ⊆ R d , the domain diameter satisfies ∥ x -y ∥ ≤ D , and for t ∈ [ T ] , the gradient norm of the online functions is bounded as ∥∇ f t ( x ) ∥ ≤ G .

Assumption 2 (Smoothness) . For each t ∈ [ T ] , the online function f t ( · ) is L -smooth, i.e., ∥∇ f t ( x ) -∇ f t ( y ) ∥ ≤ L ∥ x -y ∥ holds for any x , y ∈ R d .

Both assumptions are common in the literature. Specifically, the boundedness assumption is widely used in OCO [Hazan, 2016]. And the smoothness assumption is essential for first-order algorithms in achieving the gradient-variation regret [Chiang et al., 2012]. Note that here Assumption 2 requires smoothness on the whole R d space and can be relaxed to a slightly larger domain than X , formally, X + ≜ { x + b | x ∈ X , b ∈ G / L · B } , where B ≜ { x | ∥ x ∥ ≤ 1 } is a unit ball. We defer the details of this relaxed smoothness requirement and its derivation to Appendix A. In the following, we provide formal definitions of strong convexity and exp-concavity.

Definition 1. For any x , y ∈ X , a function f ( · ) is λ -strongly convex if f ( x ) -f ( y ) ≤ ⟨∇ f ( x ) , x -y ⟩ -λ 2 · ∥ x -y ∥ 2 ; f ( · ) is α -exp-concave if f ( x ) -f ( y ) ≤ ⟨∇ f ( x ) , x -y ⟩ -α 2 · ⟨∇ f ( x ) , x -y ⟩ 2 .

Note that the formal definition of β -exp-concavity is that exp( -βf ( · )) is concave. Under Assumption 1, β -exp-concavity implies Definition 1 with α = 1 2 · min { 1 / (4 GD ) , β } [Hazan, 2016, Lemma 4.3]. Therefore, we adopt Definition 1 as an alternative definition of exp-concavity for clarity.

2.2 A General Framework for Universal Online Learning

In this part, we present a general framework of universal online learning [Zhang et al., 2022a, Yan et al., 2023]. Formally, we study the problem where the learner lacks the prior knowledge of curvature information, including (i) curvature type: convexity, exp-concavity, or strong convexity; and (ii) curvature coefficient: exp-concavity α or strong convexity λ . Without loss of generality, we focus on the case of α, λ ∈ [1 /T, 1] . If α, λ < 1 /T , even the optimal minimax results O ( d α log T ) for exp-concave functions and O ( 1 λ log T ) for strongly convex functions [Hazan et al., 2007] - become linear in T , rendering the regret bounds vacuous. On the other hand, if α, λ > 1 , we can simply treat them as α, λ = 1 , only making the regret worsen by an ignorable constant factor. This simplification is also adopted by Zhang et al. [2022a], Yan et al. [2023].

To handle the uncertainty of curvatures, an online ensemble structure is usually employed, with multiple base learners exploring the environments and a meta learner tracking the best base learner. More specifically, to deal with the unknown curvature coefficients α and λ , we discretize them into the following candidate pool [Zhang et al., 2022a]:

whose size is n = ⌈ log 2 T ⌉ +1 = O (log T ) . It can be proved that the discretized candidate pool H can approximate the continuous value of α or λ with negligible constant errors. By doing this, it is natural to design three distinct groups of base learners:

(i) strongly convex base learners: n in total, each of which implements the algorithm for strongly convex functions with a guess λ i ∈ H of the strong convexity coefficient λ ;

(ii) exp-concave base learners: n in total, each of which implements the algorithm for expconcave functions with a guess α i ∈ H of the exp-concavity coefficient α ;

(iii) convex base learner: only one, it runs the algorithm for convex functions.

In total, there are N ≜ (2 n +1) = O (log T ) base learners with a two-layer structure, which is for now necessary in this problem. The best base learner is the one with the right guess of the curvature type and the closest guess of the curvature coefficient. Taking λ -strongly convex functions as an example, the guessed coefficient of the best base learner (indexed by i ⋆ ) satisfies λ i ⋆ ≤ λ ≤ 2 λ i ⋆ .

Denoting by x t,i the decision generated by the i -th base learner at the t -th round, p t,i the weight of the meta learner on the i -th base learner, an online ensemble method outputs the final decision as x t = ∑ i ∈ [ N ] p t,i x t,i . This forms a general framework for universal online learning and it remains to select suitable algorithms and loss functions for the meta and base learners. We will illuminate our concrete algorithm design in Section 3.1 and present a more detailed description of our meta/base learners configurations in Appendix B.

Algorithm 1 A Simple Approach for Universal Online Learning with Gradient Variations

Input: Meta learner A , base learners {B i } i ∈ [ N

]

1: Initialize : p 1 ,i = 1 / N and x 1 ,i to be an arbitrary decision inside X for all i ∈ [ N ]

2: for t = 1 to T do

3: Submit x t = ∑ i ∈ [ N ] p t,i x t,i , suffer f t ( x t ) , and observe the gradient ∇ f t ( x t ) # Meta Update:

4: A updates to p t +1 ∈ ∆ N using the rule (B.5) with learning rate (B.6) # Base Update:

5: Construct surrogates h exp t,i ( · ) h sc t,i ( · ) , h c t,i ( · ) defined in (3.1) using only ∇ f t ( x t )

6: B i updates to x t +1 ,i using surrogates functions (3.1) and update rules (B.1)-(B.3) for i ∈ [ N ] 7: end for

3 Our Approach

In this section, we present our approach for universal online learning with gradient-variation regret. Section 3.1 presents the overall procedure of our proposed algorithm. Subsequently, we outline our two key technical components: Section 3.2 presents a novel analysis to handle the empirical gradient variation, and Section 3.3 introduces surrogate functions to improve efficiency and provides a corresponding analysis for the empirical gradient variation defined on surrogates. We finally provide the optimal universal gradient-variation regret guarantees in Section 3.4.

3.1 Overall Algorithm

In this part, we present our simple approach for universal online learning with gradient variations, summarized in Algorithm 1. Basically, it is a two-layer online ensemble. Base learners are implemented using the preliminary configurations given in Section 2.2 and on carefully designed surrogate functions. The meta learner runs OPTIMISTIC-ADAPT-ML-PROD [Wei et al., 2016] on linearized losses. We specify the algorithmic details below, and a more detailed procedure in Appendix B.

In Line 3, the learner makes a weighted combination of the base learners' decisions { x t,i } i ∈ [ N ] using the meta learner's weights p t = ( p t, 1 , . . . , p t,N ) , submits the final decision x t , suffers a loss f t ( x t ) , and receives a single ∇ f t ( x t ) as the gradient feedback, using only 1 gradient query per round.

Meta Algorithm. In Line 4, the meta learner uses OPTIMISTIC-ADAPT-ML-PROD [Wei et al., 2016] to update the weights by the following rule:

Specifically, denoting by ℓ t,i ≜ ⟨∇ f t ( x t ) , x t,i ⟩ the loss of the i -th dimension, the meta algorithm inputs: r t,i = ⟨ ℓ t , p t ⟩ -ℓ t,i , the instantaneous regret; ε t,i , a time-varying learning rate; and m t,i , an estimation of the true loss of the t -th round (the choice of optimisms will be shown later). The meta algorithm then outputs the weights p t +1 = ( p t +1 , 1 , . . . , p t +1 ,N ) of the next round.

With appropriate learning rates (B.6), OPTIMISTIC-ADAPT-ML-PROD achieves an optimistic secondorder bound of ∑ t ≤ T r t,i ≤ O ( √ log N ∑ t ( r t,i -m t,i ) 2 + log N ) , where the log N factor is negligible since the base learner number N equals O (log T ) and we can treat O (log log T ) as a constant [Luo and Schapire, 2015]. The formal guarantee of OPTIMISTIC-ADAPT-ML-PROD is deferred to Lemma 2 in the appendix. In our problem, the instantaneous regret r t,i = ⟨ ℓ t , p t ⟩-ℓ t,i = ⟨∇ f t ( x t ) , x t -x t,i ⟩ . Thus we choose m t,i = ⟨∇ f t -1 ( x t -1 ) , x t -x t,i ⟩ for the convex base learner and m t,i = 0 otherwise (i.e., for exp-concave and strongly convex base learners). 2 By doing this, we can upper-bound ∑ t ⟨∇ f t ( x t ) , x t -x t,i ⟩ by O ( √∑ t ⟨∇ f t ( x t ) -∇ f t -1 ( x t -1 ) , x t -x t,i ⟩ 2 ) for the convex base learner and by O ( √∑ t ⟨∇ f t ( x t ) , x t -x t,i ⟩ 2 ) otherwise. Later in Section 3.3, we illuminate how such results could benefit the final regret guarantees.

Base Algorithm. In Line 5, we adopt carefully designed surrogate functions for different types of base learners to reduce the gradient query complexity [Yan et al., 2023]. Specifically, strongly convex, exp-concave, and the convex base learners run on the surrogate functions below respectively:

and h c t,i ( x ) ≜ ⟨∇ f t ( x t ) , x ⟩ , where λ i , α i are selected from the candidate coefficient pool H in (2.1). We emphasize that the surrogate functions require only 1 gradient query ∇ f t ( x t ) per round. Finally, in Line 6, the i -th base learner B i updates the decision to x t +1 ,i using optimistic online mirror descent (OOMD) [Rakhlin and Sridharan, 2013], which is general and covers many algorithms of interest, such as OGD and online Newton step. For each curvature type (convexity, exp-concavity, or strong convexity), we adopt a correspondingly configured OOMD as the base learner. For detailed update rules of differently configured OOMD, we refer readers to (B.1)-(B.3) in Appendix B.

As for previous works, Zhang et al. [2022a] adopted ADAPT-ML-PROD [Gaillard et al., 2014] as the meta learner, which does not incorporate optimisms and thus is impossible to achieve gradientvariation regret for convex functions, and operates on the original loss function f t ( · ) for base learners, which leads to a less efficient gradient query complexity of O (log T ) per round. Yan et al. [2023] used a two-layer meta algorithm MSMWC-MASTER [Chen et al., 2021] as the meta learner, resulting in a three-layer ensemble structure, which is also not efficient enough. Compared with approaches above, our Algorithm 1 is simpler and more efficient as it requires O (log T ) base learners and only 1 gradient query in each round. We emphasize that our contributions mainly lie in the technical aspects showing that although simple, our approach can achieve the optimal universal gradient-variation regret, which is accomplished via two novel analytical components.

3.2 Novel Analysis on Empirical Gradient Variations

In this part, we provide a novel analysis of the gradient-variation regret. For clarity, we illustrate from the lowest level - as we only use one gradient ∇ f t ( x t ) in the t -th round, to obtain the gradient variation V T defined in (1.2), it is necessary to first attain its empirical version ¯ V T ≜ ∑ t ≤ T ∥∇ f t ( x t ) -∇ f t -1 ( x t -1 ) ∥ 2 . Previous studies decompose this term into two parts:

using smoothness (i.e., Assumption 2). Aggregating the first term over T rounds leads to the desired V T quantity and the remaining challenge is to control the algorithmic stability ∥ x t -x t -1 ∥ 2 . Consequently, since each decision is a weighted combination of base learners' decisions (i.e., x t = ∑ i ≤ N p t,i x t,i ), the algorithmic stability is difficult to control. To this end, Yan et al. [2023] decomposed the stability term in the following way:

Consequently, for the first term, the authors injected correction terms to the meta learner following Zhao et al. [2024]. To cancel the second term, the meta algorithm must include a corresponding negative stability term in its analysis, while achieving an optimistic second-order bound simultaneously. To the best of our knowledge, the only feasible algorithm satisfying both requirements is the two-layer meta algorithm MSMWC-MASTER [Chen et al., 2021], which leads to a three-layer online ensemble structure and therefore affects the efficiency. Besides, it attains a second-order bound of the form O ( √ Q T,i log Q T,i ) , where Q T,i ≜ ∑ t ( ℓ t,i -m t,i ) 2 , which causes the sub-optimality of the regret guarantees with an additional logarithmic factor in the results of Yan et al. [2023].

In this work, we handle the empirical gradient variation alternatively via a novel and simple analysis with two key parts: (i) a negative term arising from linearization; and (ii) a useful smoothness property. First, we observe that the instantaneous regret can be transformed as:

where x ⋆ ∈ arg min x ∈X ∑ t f t ( x ) and D f ( x , y ) ≜ f ( x ) -f ( y ) -⟨∇ f ( y ) , x -y ⟩ is the Bregman divergence associated with function f ( · ) . The last term is a negative term from linearization , which

can be seen as the compensation by treating a convex function as a linear one. Previous studies on the gradient-variation regret usually omit this term, while we show below that this negative term helps to achieve a much simpler analysis of the empirical gradient variation. Second, we introduce a useful property of smoothness , formally introduced below.

Proposition 1 (Theorem 2.1.5 of Nesterov [2018]) . f ( · ) is L -smooth over R d if and only if

Compared with the commonly used ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ , Proposition 1 gives a tighter bound for the squared gradient changes since ∥∇ f ( x ) -∇ f ( y ) ∥ 2 ≤ 2 L D f ( y , x ) ≤ L 2 ∥ x -y ∥ 2 , where the second step is due to D f ( y , x ) ≤ L 2 ∥ x -y ∥ 2 , for any x , y ∈ R d [Nesterov, 2018, Theorem 2.1.5], intuitively making the analysis easier. Combining the Bregman divergence negative term (3.3) and this useful property (3.4), we address the empirical gradient variation effectively as

where the first step introduces intermediate terms ∇ f t ( x ⋆ ) and ∇ f t -1 ( x ⋆ ) , the second step uses Proposition 1, and the last step combines two summations into one by shifting the indexes of t . The Bregman divergence negative term in (3.3) can cancel the positive term in (3.5), leaving only the gradient variation quantity V T as desired.

Here we only require ∥∇ f ( x ) -∇ f ( y ) ∥ 2 ≤ 2 L D f ( y , x ) on X rather than R d , which can be satisfied by requiring L -smoothness only on a slightly larger domain than X (a relaxation of Assumption 2). Interested readers can refer to Appendix A for details. Finally we end this part with several remarks.

Remark 1. We emphasize that the Bregman divergence negative term comes from the linearization of convex functions, and is thus algorithm-independent . Therefore, we can eliminate the need to control the algorithmic stability, in contrast to previous works for gradient-variation regret [Chiang et al., 2012, Yan et al., 2023]. To the best of our knowledge, this is the first alternative analysis of the gradient-variation regret since first proposed by Chiang et al. [2012]. ◁

Remark 2. Our idea of the negative term from linearization is inspired by the recent advance in stochastic optimization [Joulani et al., 2020]. Note that they focus on a different problem of achieving the O (1 /T 2 ) rate as Nesterov's accelerated gradient [Nesterov, 2018], while we investigate the gradient-variation regret in the universal online (adversarial) convex optimization setup. ◁

Remark 3. Our analysis does not strictly outperform those that directly handle the stability term as in (3.2), e.g., Yan et al. [2023], since the latter can be used for fast rates in the multi-player game setup [Syrgkanis et al., 2015, Zhang et al., 2022b]. A more detailed discussion of the advantages and disadvantages over previous approaches is provided in Section 4.3. ◁

3.3 Novel Analysis on Empirical Gradient Variations of Surrogates

Section 3.2 already suffices to achieve the optimal universal gradient-variation regret if multiple gradient queries are permitted. In this part, we consider further improving the computational efficiency by using only 1 gradient query per round, achieving the same gradient query complexity as OGD.

As stated in Section 3.1, we implement base algorithms on carefully designed surrogate functions following Yan et al. [2023]. In this part, we show that additional novel analysis is required to handle the empirical gradient variation defined on surrogates . To see this, we first provide the entire regret decomposition to give readers a clear roadmap. Specifically, taking λ -strong convexity as an example, the regret can be decomposed as follows:

where the first step uses (3.3) and the fact that D f t ( x ⋆ , x t ) ≥ λ 2 ∥ x t -x ⋆ ∥ 2 since f t ( · ) is λ -strongly convex. The second step uses the definition of the best base learner (indexed by i ⋆ ): λ i ⋆ ≤ λ ≤ 2 λ i ⋆ and the definition of surrogate functions defined in (3.1). The first term above ( meta regret ) assesses how well the algorithm tracks the best base learner, and the second term ( base regret ) measures the best base learner's performance. The meta regret contains a linearized regret with a negative term from curvatures. This negative term is useful for exp-concave and strongly convex functions if the linearized regret enjoys a second-order bound:

where the 1 /λ factor can be absorbed by the final regret O ( 1 λ log V T ) . The base regret is defined on the surrogates, which preserves the curvature properties, but using only 1 gradient ∇ f t ( x t ) .

Below we explain the reason for handling the empirical gradient variation defined on surrogates. Taking the i -th strongly convex base learner as an example, it updates as x t,i = Π X [ ̂ x t,i -η t m t ] and ̂ x t +1 ,i = Π X [ ̂ x t,i -η t ∇ h sc t,i ( x t,i )] , an initialization of the OOMD algorithm, where η t represents the step size, Π X [ x ] = arg min y ∈X ∥ x -y ∥ denotes the Euclidean projection onto X , and ̂ x t,i is an intermediate variable. With appropriately chosen step sizes, the base learner achieves an optimistic bound of O (log D T ) , where D T = ∑ t ≤ T ∥∇ h sc t,i ( x t,i ) -m t ∥ 2 (e.g., please refer to Theorem 15 of Chiang et al. [2012]). Therefore, choosing the optimism as m t = ∇ h sc t -1 ,i ( x t -1 ,i ) leads to an empirical gradient-variation bound O (log D T ) defined on surrogates , 3 where

The empirical gradient variation defined on the original functions, i.e., ∥∇ f t ( x t ) -∇ f t -1 ( x t -1 ) ∥ 2 , can be handled via the analysis in Section 3.2. The main challenge is to deal with the rest terms caused by the surrogate functions. Yan et al. [2023] overcame this issue by controlling ( x t -x t -1 ) and ( x t,i -x t -1 ,i ) separately. Again, as we have explained in Section 3.2, since the decision x t is a weighted combination of base learners' decisions (i.e., x t = ∑ i ≤ N p t,i x t,i ), handling the algorithmic stability term ∥ x t -x t -1 ∥ 2 directly using (3.2) would results in a less efficient three-layer online ensemble structure and the sub-optimality of the regret guarantees, as Yan et al. [2023] did.

In this work, we propose a novel analysis - while controlling ( x t,i -x t ) -( x t -1 ,i -x t -1 ) in each individual round is hard, it can be bounded when aggregated over the time horizon . Specifically, we bound it by combining two summations into one:

The same idea is also used in the derivation of (3.5). Consequently, this term can be canceled out by the negative term from curvatures in the meta regret. For this cancellation to occur, appropriate coefficients are chosen, which are provided in the detailed proofs (e.g., the 'Regret Analysis' part in the proof of Theorem 1) and are omitted here for clarity.

This simple and novel analysis eliminates the need to control the overall algorithmic stability term of ∥ x t -x t -1 ∥ 2 required by previous works, and is essential for achieving the improved computational efficiency and the optimal regret guarantees, as shown in the next part.

3.4 Optimal Universal Gradient-Variation Regret Guarantees

In this part, we present our main theoretical result - our simple and efficient Algorithm 1 (in Section 3.1) which adopts two novel analyses (in Section 3.2 and Section 3.3) achieves the optimal

gradient-variation regret without requiring the curvature information in advance for universal online learning. The corresponding proof is provided in Appendix C.

Theorem 1. Under Assumptions 1 and 2 (or the relaxed Assumption 3), Algorithm 1 achieves O (log V T ) , O ( d log V T ) , and O ( √ V T ) for strongly convex, exp-concave, and convex functions.

Theorem 1 improves the O ( √ V T log V T ) bound of Yan et al. [2023] and is optimal by matching the best known results when the curvature information is known. It performs well when the gradient variation is small, such as f 1 = f 2 = · · · = f T (where V T = 0 ). Note that for α -exp-concave or λ -strongly convex functions, our guarantee is actually O (min { d α log V T , √ V T } ) or O (min { 1 λ log V T , √ V T } ) , thus ensuring O ( √ V T ) even when α = O ( 1 / T ) or λ = O ( 1 / T ) . This is because exp-concave and strongly convex functions are also convex and thus our convex bound is still applicable.

4 Implication, Application, and Discussion

In this section, we validate the effectiveness of our results by the implication of small-loss regret and the application in the SEA model. We also discuss the technical comparison with the previous correction-based approach [Yan et al., 2023] at the end of this section.

4.1 Implication to Universal Small-Loss Regret

In this part, we illustrate that our universal gradient-variation regret in Theorem 1 implies the universal small-loss regret measured by F T ≜ min x ∈X ∑ t ≤ T f t ( x ) directly in analysis , i.e., without any algorithmic modification, and thus safeguards the case of F T ≤ V T , such as min x ∈X f t ( x ) = 0 for any t ∈ [ T ] (where F T = 0 ). The corresponding proof is provided in Appendix D.1.

Theorem 2. Under Assumptions 1 and 2 (or the relaxed Assumption 3), if the online functions are non-negative, Algorithm 1 achieves O (log F T ) , O ( d log F T ) , and O ( √ F T ) for strongly convex, exp-concave, and convex functions, respectively.

Theorem 2 achieves the same optimal small-loss bounds as Zhang et al. [2022a]. Combined with Theorem 1, our approach achieves the best known problem-dependent regret guarantees in the universal online learning problem. In the end, we emphasize again that our approach is efficient as it requires O (log T ) base learners and only 1 gradient query in each round.

4.2 Application to Stochastically Extended Adversarial (SEA) Model

Stochastically extended adversarial (SEA) model [Sachs et al., 2022] interpolates between stochastic and adversarial online convex optimization. Formally, it assumes that the online function f t ( · ) is sampled stochastically from an adversarially chosen distribution D t . Denoting by F t ( · ) ≜ E f t ∼ D t [ f t ( · )] the expected function, two terms capture the essential characteristics of SEA model:

where σ 2 1: T is the variance in sampling f t ( · ) from D t ( · ) and Σ 2 1: T is the variation of { F t ( · ) } t ∈ [ T ] . Sachs et al. [2022] initiated the study of the SEA model. For smooth expected functions { F t ( · ) } T t =1 , they achieved the optimal O ( √ σ 2 1: T +Σ 2 1: T ) regret for convex expected functions, and O (( σ 2 max + Σ 2 max ) log T ) in the strongly convex case, where σ 2 max ≜ max t ∈ [ T ] max x ∈X E f t ∼ D t [ ∥∇ f t ( x ) -∇ F t ( x ) ∥ 2 ] and Σ 2 max ≜ max t ∈ [ T ] sup x ∈X ∥∇ F t ( x ) - ∇ F t -1 ( x ) ∥ 2 . Subsequently, Chen et al. [2024] improved the strongly convex regret to O (( σ 2 max +Σ 2 max ) log(( σ 2 1: T +Σ 2 1: T ) / ( σ 2 max +Σ 2 max ))) and obtained O ( d log( σ 2 1: T +Σ 2 1: T )) regret for exp-concave individual functions { f t ( · ) } T t =1 .

The gradient variation is essential in connecting the stochastic and adversarial optimization [Chen et al., 2023, Lemma 4], which is also restated in (D.2). Therefore, universal gradient-variation regret can be applied to this problem, achieving the same best known bounds as Chen et al. [2024], with a single algorithm. Table 2 compares our results with existing ones. The following theorem presents our results formally, with the corresponding proof provided in Appendix D.2.

Theorem 3. Under Assumption 1 and smoothness of F t ( · ) for any t ∈ [ T ] : if F t ( · ) is convex, Algorithm 1 achieves O ( √ σ 2 1: T +Σ 2 1: T ) ; if f t ( · ) is exp-concave, it achieves O ( d log( σ 2 1: T +Σ 2 1: T )) ; and if F t ( · ) is strongly convex, it achieves O (( σ 2 max +Σ 2 max ) log(( σ 2 1: T +Σ 2 1: T ) / ( σ 2 max +Σ 2 max ))) .

Theorem 3 requires exp-concavity of the individual function f t ( · ) rather than the expected function F t ( · ) . This assumption is also used by Chen et al. [2023] and common in the studies of stochastic exp-concave optimization [Mahdavi et al., 2015, Koren and Levy, 2015].

4.3 Discussion on Comparison with Correction-based Approach

In this part, we discuss the technical comparison with the previous correction-based approach [Yan et al., 2023]. Compared with their approach, ours is simpler and achieves the optimal universal problem-dependent regret (Theorem 1 and Theorem 2) and the best known guarantees in the SEA model (Theorem 3). Although not providing guarantees as favorable as ours, Yan et al. [2023] can control the overall algorithmic stability (i.e., ∥ x t -x t -1 ∥ 2 ) using collaborative online ensemble [Zhao et al., 2024], which is necessary in achieving fast rates in multi-player games [Syrgkanis et al., 2015]. For example, in a min-max game min x ∈X max y ∈Y x ⊤ A y , where A is a game matrix, since A is unknown, the Nash equilibrium is typically computed through repeated play, i.e., playerx and playery select { x t } T t =1 and { y t } T t =1 sequentially to approach the Nash equilibrium. For playerx , in the t -th round, it suffers a loss x ⊤ t A y t and receives the gradient A y t . Similarly, playery suffers -x ⊤ t A y t and receives -A x t . For playerx , if it updates via OOMD, its gradient-variation regret contains ∥ A y t -A y t -1 ∥ 2 , which includes the stability of playery . In this case, to achieve fast rates, we indeed need to control the algorithm stability like ∥ x t -x t -1 ∥ 2 and ∥ y t -y t -1 ∥ 2 . This can be done by Yan et al. [2023] while this work cannot since we do not directly control the algorithmic stability. Interested readers can refer to Appendix A.2 in Yan et al. [2023] for more details.

5 Conclusion

In this work, we investigate universal online learning with gradient-variation regret. We propose a simple two-layer online ensemble approach that not only achieves the optimal O ( 1 λ log V T ) , O ( d α log V T ) , and O ( √ V T ) regret simultaneously for λ -strongly convex, α -exp-concave, and convex functions and is efficient with O (log T ) base learners and only 1 gradient query per round. This is done via the negative Bregman divergence term from linearization and the useful smoothness property of ∥∇ f ( x ) -∇ f ( y ) ∥ 2 ≤ 2 L D f ( y , x ) . We further validate the effectiveness of our approach and results by implying the optimal universal small-loss regret directly in analysis and achieving the best known results in the stochastically extended adversarial model.

Two future directions are worth investigating. The first is to reduce the number of projections to only 1 in each round [Mhammedi et al., 2019, Zhao et al., 2022, Yang et al., 2024], thereby further improving the computational efficiency. The second direction involves extending our algorithm and results to the unconstrained domain using recent advances in parameter-free online learning [Orabona and P´ al, 2016, Cutkosky and Orabona, 2018, Jacobsen and Cutkosky, 2022], to broaden its applicability across a wider range of scenarios.

References

Abernethy, P. L. Bartlett, A. Rakhlin, and A. Tewari. Optimal strategies and minimax lower bounds for online convex games. In Proceedings of the 21st Annual Conference on Learning Theory (COLT) , pages 414-424, 2008.

Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

Chen, H. Luo, and C. Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Proceedings of the 34th Annual Conference on Learning Theory (COLT) , pages 1216-1259, 2021.

Chen, W.-W. Tu, P. Zhao, and L. Zhang. Optimistic online mirror descent for bridging stochastic and adversarial online convex optimization. In Proceedings of the 40th International Conference on Machine Learning (ICML) , pages 5002-5035, 2023.

Chen, Y.-J. Zhang, W.-W. Tu, P. Zhao, and L. Zhang. Optimistic online mirror descent for bridging stochastic and adversarial online convex optimization. Journal of Machine Learning Research , 25 (178):1 - 62, 2024.

Chiang, T. Yang, C. Lee, M. Mahdavi, C. Lu, R. Jin, and S. Zhu. Online optimization with gradual variations. In Proceedings of the 25th Annual Conference on Learning Theory (COLT) , pages 6.1-6.20, 2012.

Cutkosky and F. Orabona. Black-box reductions for parameter-free online learning in Banach spaces. In Proceedings of the 31st Annual Conference on Learning Theory (COLT) , pages 14931529, 2018.

Daniely, A. Gonen, and S. Shalev-Shwartz. Strongly adaptive online learning. In Proceedings of the 32nd International Conference on Machine Learning (ICML) , pages 1405-1411, 2015.

Gaillard, G. Stoltz, and T. van Erven. A second-order bound with excess losses. In Proceedings of the 27th Annual Conference on Learning Theory (COLT) , pages 176-196, 2014.

Hazan. Introduction to Online Convex Optimization. Foundations and Trends in Optimization , 2 (3-4):157-325, 2016.

Hazan and C. Seshadhri. Adaptive algorithms for online decision problems. Electronic Colloquium on Computational Complexity (ECCC) , 14(088), 2007.

Hazan, A. Agarwal, and S. Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2-3):169-192, 2007.

Jacobsen and A. Cutkosky. Parameter-free mirror descent. In Proceedings of the 35th Annual Conference on Learning Theory (COLT) , pages 4160-4211, 2022.

Ji and J. Ye. An accelerated gradient method for trace norm minimization. In Proceedings of the 26th International Conference on Machine Learning (ICML) , pages 457-464, 2009.

Joulani, A. Raj, A. Gyorgy, and C. Szepesv´ ari. A simpler approach to accelerated optimization: Iterative averaging meets optimism. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 4984-4993, 2020.

Koren and K. Y. Levy. Fast rates for exp-concave empirical risk minimization. In Advances in Neural Information Processing Systems 28 (NIPS) , pages 1477-1485, 2015.

Li, T. Zhang, Y. Chen, and A. J. Smola. Efficient mini-batch training for stochastic optimization. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) , pages 661-670, 2014.

Luo and R. E. Schapire. Achieving all with no parameters: AdaNormalHedge. In Proceedings of the 28th Annual Conference on Learning Theory (COLT) , pages 1286-1304, 2015.

Mahdavi, L. Zhang, and R. Jin. Lower and upper bounds on the generalization of stochastic exponentially concave optimization. In Proceedings of the 28th Annual Conference on Learning Theory (COLT) , pages 1305-1320, 2015.

Mhammedi, W. M. Koolen, and T. van Erven. Lipschitz adaptivity with multiple learning rates in online learning. In Proceedings of the 32nd Annual Conference on Learning Theory (COLT) , pages 2490-2511, 2019.

Nesterov. Lectures on Convex Optimization , volume 137. Springer, 2018.

Orabona. A modern introduction to online learning. ArXiv preprint , arXiv:1912.13213, 2019.

Orabona and D. P´ al. Coin betting and parameter-free online learning. In Advances in Neural Information Processing Systems 29 (NIPS) , pages 577-585, 2016.

Orabona, N. Cesa-Bianchi, and C. Gentile. Beyond logarithmic bounds in online learning. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 823-831, 2012.

Ordentlich and T. M. Cover. The cost of achieving the best portfolio in hindsight. Mathematics of Operations Research , 23(4):960-982, 1998.

Rakhlin and K. Sridharan. Online learning with predictable sequences. In Proceedings of the 26th Annual Conference on Learning Theory (COLT) , pages 993-1019, 2013.

Sachs, H. Hadiji, T. van Erven, and C. Guzm´ an. Between stochastic and adversarial online convex optimization: Improved regret bounds via smoothness. In Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 691-702, 2022.

Sachs, H. Hadiji, T. van Erven, and C. Guzm´ an. Accelerated rates between stochastic and adversarial online convex optimization. ArXiv preprint , arXiv:2303.03272, 2023.

Srebro, K. Sridharan, and A. Tewari. Smoothness, low noise and fast rates. In Advances in Neural Information Processing Systems 23 (NIPS) , pages 2199-2207, 2010.

Syrgkanis, A. Agarwal, H. Luo, and R. E. Schapire. Fast convergence of regularized learning in games. In Advances in Neural Information Processing Systems 28 (NIPS) , pages 2989-2997, 2015.

van Erven and W. M. Koolen. MetaGrad: Multiple learning rates in online learning. In Advances in Neural Information Processing Systems 29 (NIPS) , pages 3666-3674, 2016.

Wang, S. Lu, and L. Zhang. Adaptivity and optimality: A universal algorithm for online convex optimization. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence (UAI) , pages 659-668, 2019.

Wei, Y. Hong, and C. Lu. Tracking the best expert in non-stationary stochastic environments. In Advances in Neural Information Processing Systems 29 (NIPS) , pages 3972-3980, 2016.

Y.-H. Yan, P. Zhao, and Z.-H. Zhou. Universal online learning with gradient variations: A multi-layer online ensemble approach. In Advances in Neural Information Processing Systems 36 (NeurIPS) , pages 37682-37715, 2023.

Yang, M. Mahdavi, R. Jin, and S. Zhu. Regret bounded by gradual variation for online convex optimization. Machine Learning , 95(2):183-223, 2014.

Yang, Y. Wang, P. Zhao, and L. Zhang. Universal online convex optimization with 1 projection per round. ArXiv preprint , arXiv:2405.19705, 2024.

Zhang, S. Lu, and Z.-H. Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems 31 (NeurIPS) , pages 1330-1340, 2018.

Zhang, T.-Y. Liu, and Z.-H. Zhou. Adaptive regret of convex and smooth functions. In Proceedings of the 36th International Conference on Machine Learning (ICML) , pages 7414-7423, 2019.

Zhang, G. Wang, J. Yi, and T. Yang. A simple yet universal strategy for online convex optimization. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 2660526623, 2022a.

Zhang, P. Zhao, H. Luo, and Z.-H. Zhou. No-regret learning in time-varying zero-sum games. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 2677226808, 2022b.

Zhao. Online Ensemble Theories and Methods for Robust Online Learning . PhD thesis, Nanjing University, Nanjing, China, 2021. Advisor: Zhi-Hua Zhou.

Zhao, Y.-J. Zhang, L. Zhang, and Z.-H. Zhou. Dynamic regret of convex and smooth functions. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 12510-12520, 2020.

Zhao, Y.-F. Xie, L. Zhang, and Z.-H. Zhou. Efficient methods for non-stationary online learning. In Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 11573-11585, 2022.

Zhao, Y.-J. Zhang, L. Zhang, and Z.-H. Zhou. Adaptivity and non-stationarity: Problem-dependent dynamic regret for online convex optimization. Journal of Machine Learning Research , 25(98):1 52, 2024.

Z.-H. Zhou. Ensemble Methods: Foundations and Algorithms . Chapman & Hall/CRC Press, 2012.

Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML) , pages 928-936, 2003.

A On Smoothness Assumption

In this section, we propose a relaxation of the smoothness requirement to a slightly larger domain than the feasible domain X , in contrast to the whole R d space as in Assumption 2. The relaxed smoothness assumption is detailed in the following.

Assumption 3 (Relaxed Smoothness) . Under the condition of ∥∇ f t ( x ) ∥ ≤ G for any x ∈ X and t ∈ [ T ] , all online functions are L -smooth: ∥∇ f t ( x ) -∇ f t ( y ) ∥ ≤ L ∥ x -y ∥ for any t ∈ [ T ] and x , y ∈ X + , where X + ≜ { x + b | x ∈ X , b ∈ G / L · B } and B ≜ { x | ∥ x ∥ ≤ 1 } is a unit ball.

The domain of Assumption 3 is slightly larger than X -for any x + ∈ X + , we can always find an x ∈ X such that ∥ x + -x ∥ ≤ G / L . In this work, one of the key technical contributions is to handle the empirical gradient variation via a useful smoothness property (Proposition 1). We show in the following that this condition can be satisfied by requiring only Assumption 3.

Lemma 1. Under Assumption 1, for any online function f ( · ) satisfying Assumption 3, it holds that ∥∇ f ( x ) -∇ f ( y ) ∥ 2 ≤ 2 L D f ( y , x ) for any x , y ∈ X .

Proof. To begin with, we present the self-bounding property [Srebro et al., 2010], which is useful in proving our result - if a function f : R d ↦→ R is L -smooth and bounded from below, then for any x ∈ R d , it holds that

Next, we aim to prove that if we only need (A.1) on a bounded domain X , we require smoothness only on a slightly larger domain than X . To see this, we delve into the proof of the self-bounding property. Specifically, for any x , v ∈ R d , it holds that

where the first step requires smoothness on x and x + v . Consequently, by taking maximization over v , it holds that

which leads to the self-bounding property (A.1) by taking v = -1 L ∇ f ( x ) . The above proof is from Theorem 4.23 of Orabona [2019]. This means that for the self-bounding property, we only require the smoothness to hold for any x ∈ X and x -1 L ∇ f ( x ) . Under Assumption 1, this can be satisfied by requiring smoothness on a slightly larger domain than X , namely, X + ≜ { x + b | x ∈ X , b ∈ G / L · B } .

Now we are ready to prove the final result. To begin with, we define a surrogate function of g ( x ) ≜ f ( x ) -⟨∇ f ( x 0 ) , x ⟩ for any x ∈ X , where x 0 ∈ X . Due to the above property we have just proved, by requiring smoothness on X + , we have

Denoting by y ⋆ ∈ arg min y ∈ R d g ( y ) , the above inequality equals to

due to the definition of g ( · ) . The proof using the self-bounding property is from Theorem 2.1.5 of Nesterov [2018]. Finally, we note that g ( · ) is minimized at y ⋆ = x 0 , leading to ∥∇ f ( x ) -∇ f ( x 0 ) ∥ 2 ≤ 2 L D f ( x 0 , x ) for any x , x 0 ∈ X , which finishes the proof.

B Omitted Details of Algorithm 1

In this section, we provide some omitted details of our Algorithm 1, including the losses and update rules of the base and meta learners.

Base Learners. To begin with, we duplicate the candidate coefficient pool (2.1) for both the expconcave coefficient α and the strongly convex coefficient λ , denoted by H exp ≜ H and H sc ≜ H . Consequently, denoting by N exp = N sc ≜ |H| the size of candidate pool, for each α i ∈ H exp and λ j ∈ H sc , where i ∈ [ N exp ] and j ∈ [ N sc ] , we define corresponding groups of base learners for optimizing exp-concave and strongly convex functions. Specifically, for α -exp-concave functions, we define a group of base learners {B exp i } i ∈ [ N exp ] , where the i -th base learner runs the algorithm below:

where ψ t,i ( x ) ≜ 1 2 x ⊤ U t,i x , U t,i = (1 + α i G 2 2 ) I + α i 2 ∑ t -1 s =1 ∇ h exp s,i ( x s,i ) h exp s,i ( x s,i ) ⊤ , α i is the i -th element in H exp , and h exp t,i ( · ) is a surrogate loss function for B exp i , defined as

Similarly, for λ -strongly convex functions, we define a group of base learners {B sc i } i ∈ [ N sc ] , where the i -th base learner runs the algorithm below:

where η t,i = 2 / (1 + λ i t ) , λ i is the i -th element in H sc , and h sc t,i ( · ) is a surrogate loss function for B sc i , defined as

For convex functions, we only have to define one base learner B c , which updates as

where η t,i = min { D/ √ 1 + ∑ t -1 s =2 ∥∇ f t ( x t ) -∇ f t -1 ( x t -1 ) ∥ 2 , 1 } . Finally, we conclude the configurations of base learners. Specifically, we deploy

as the total set of base learners.

Meta Learner. The meta learner simply runs OPTIMISTIC-ADAPT-ML-PROD [Wei et al., 2016], which updates as follows:

where ℓ t,i ≜ ⟨∇ f t ( x t ) , x t,i ⟩ is the loss of the i -th dimension, r t,i = ⟨ ℓ t , p t ⟩ -ℓ t,i represents the instantaneous regret, m t,i = ⟨∇ f t -1 ( x t -1 ) , x t -x t,i ⟩ for the index i indicating B c and m t,i = 0 for indexes indicating B exp and B sc . The learning rate ε t,i is chosen as

C Proof for Section 3

In this section, we provide the proof of Theorem 1, our main theoretical result for the optimal universal gradient-variation regret.

Proof. We first give different decompositions of the regret for different curvature types, then analyze the meta and base regret, and finally combine them to achieve the regret bound. For simplicity, we define g t ≜ ∇ f t ( x t ) and

for short. Using the analysis in Section 3.2, the empirical gradient variation can bounded as

Regret Decomposition. Denoting by x ⋆ ∈ arg min x ∈X ∑ t ∈ [ T ] f t ( x ) , for convex functions, we decompose the regret as

where h c t,i ( x ) ≜ ⟨ g t , x ⟩ .

For α -exp-concave functions, we decompose the regret as

where the second step is due to the definitions of exp-concavity and Bregman divergence and the last step is due to the definition of the surrogate function h exp t,i ( x ) ≜ ⟨ g t , x ⟩ + α i 4 ⟨∇ f t ( x t ) , x -x t ⟩ 2 , where α i ∈ H , defined in (2.1).

For λ -strongly convex functions, following the similar decomposition in the exp-concavity case,

due to the definition of the surrogate h sc t,i ( x ) ≜ ⟨ g t , x ⟩ + λ i 4 ∥ x -x t ∥ 2 , where λ i ∈ H in (2.1).

Meta Regret Analysis. We adopt OPTIMISTIC-ADAPT-ML-PROD [Wei et al., 2016] as the meta learner, and present its regret analysis below for self-containedness.

Lemma 2 (Theorem 3.4 of Wei et al. [2016]) . Denoting by p t the weights of the algorithm, ℓ t the loss vector, and m t,i the optimism, by choosing the learning rate optimally as (B.6) , the regret of OPTIMISTIC-ADAPT-ML-PROD (B.5) with respect to any expert i ∈ [ N ] satisfies

where r t,i = ⟨ ℓ t , p t -e i ⟩ , e i denotes the i -th standard basis vector, C 0 = √ ln N +ln(1 + N e (1 + ln( T +1))) / √ ln N , and C 1 = 1 4 (ln N +ln(1 + N e (1 + ln( T +1)))) + 2 √ ln N +16ln N .

Here we adopt ℓ t,i = ⟨ g t , x t,i ⟩ such that ⟨ ℓ t , p t -e i ⟩ = ⟨ g t , x t -x t,i ⟩ . Besides, since the number of base learners N = O (log T ) as explained in Section 2, the constants C 0 and C 1 are in the order of O (log log T ) and can be treated as ignorable constants, following previous convention [Luo and Schapire, 2015, Gaillard et al., 2014].

For convex functions, we choose the optimism as m t,i = ⟨ g t -1 , x t -x t,i ⟩ for the index i indicating the convex base learner. As explained in Section 3.1, although x t is unknown for now, we only require the scalar value of ⟨ g t -1 , x t ⟩ . Denoting by z = ⟨ g t -1 , x t ⟩ , it actually forms a fixed-point problem of z = ⟨ g t -1 , x t ( z ) ⟩ , where x t is a function of z since x t depends on p t,i , p t,i relies on m t,i , and m t,i depends on z . Such a one-dimensional fixed-point problem can be solved with an O (1 /T ) approximation error through O (log T ) binary searches, and aggregating the approximate error over the whole time horizon will only incur an additive constant to the final regret. As a result, such an optimism setup is valid. Consequently, the meta regret in (C.2) can be bounded as

where the second step adopts Assumption 1, the fourth step uses √ a + b ≤ √ a + √ b for any a, b ≥ 0 , the last step uses AM-GM inequality: √ ab ≤ ax 2 + b 2 x for any a, b, x > 0 . Note that C 2 is used to ensure the positive Bregman divergence term to be canceled and will be specified in the end.

For exp-concave functions, we choose the optimism as m t,i = 0 for indexes i indicating exp-concave base learners. By Lemma 2, the meta regret in (C.3) can be bounded as

where the last step omits the ignorable additive C 0 or C 1 terms and is due to AM-GM inequality. C 2 is a constant to be specified.

For strongly convex functions, we choose the optimism m t,i = 0 for indexes i indicating strongly convex base learners. By Lemma 2, the meta regret in (C.4) can be bounded as

where the last step omits the ignorable additive C 0 or C 1 terms and is due to AM-GM inequality. C 4 is a constant to be specified.

Base Regret Analysis. For convex functions, when using the update rule (B.3), due to the standard analysis of OOMD for convex functions (e.g., Lemma 10 of Yan et al. [2023]), it holds that

where the second step is due to the property of the surrogate function: ∇ h c t,i ( x t,i ) = g t , and the last step uses AM-GM inequality. C 5 is a constant to be specified.

For exp-concave functions, when using the update rule (B.1), due to the standard analysis of OOMD for exp-concave functions (e.g., Lemma 11 of Yan et al. [2023]), the base regret can be bounded as

Next, we analyze the empirical gradient variation defined on the surrogate function h exp t,i ( · ) :

where the first step is due to the property of the surrogate function: ∇ h exp t,i ( x t,i ) = g t + α i 2 g t ⟨ g t , x t -x t,i ⟩ , the second step is by the Cauchy-Schwarz inequality: ( a + b + c ) 2 ≤ 3( a 2 + b 2 + c 2 ) for any a, b, c ∈ R . Plugging the surrogate's empirical gradient variation back to the base regret, we obtain

The second step requires C 6 ≥ 1 by Lemma 5 and uses the property of the best base learner, i.e., α i ⋆ ≤ α ≤ 2 α i ⋆ . The last step is because of α i ≤ 1 .

For strongly convex functions, when using the update rule (B.2), due to the analysis of OOMD for strongly convex functions (e.g., Lemma 12 of Yan et al. [2023]), the base regret can be bounded as

Next, we analyze the empirical gradient variation defined on the surrogate function h sc t,i ( · ) :

where the first step is due to the property of the surrogate: ∇ h sc t,i ( x t,i ) = g t + λ i 2 ( x t,i -x t ) , and the second step is due to the Cauchy-Schwarz inequality. Plugging the surrogate's empirical gradient variation back to the base regret, we obtain

where the second step requires C 7 ≥ 1 by Lemma 5 and uses the property of the best base learner, i.e., λ i ⋆ ≤ λ ≤ 2 λ i ⋆ . The last step is due to λ i ≤ 1 .

Regret Analysis. For convex functions, by combining the meta and base regret, it holds that

by choosing C 2 = C 0 and C 5 = 5 D .

For exp-concave functions, by combining the meta and base regret, it holds that

by choosing C 6 = max { 1 , 144 L, 32 G 2 α i ⋆ } and C 3 = 4 C 0 α i ⋆ . Note that such a parameter configuration will only add an O (1 /α ) factor to the final regret bound, which can be absorbed.

For strongly convex functions, by combining the meta and base regret, it holds that

by choosing C 7 = max { 1 , 1152 G 2 L, 256 G 2 λ i ⋆ } and C 4 = 4 C 0 D 2 λ i ⋆ . Note that such a parameter configuration will only add an O (1 /λ ) factor to the final regret bound, which can be absorbed.

Note that the constants C 2 , C 3 , C 4 , C 5 , C 6 , C 7 only exist in analysis and thus can be chosen arbitrarily, finishing the proof.

D Proofs for Section 4

In this section, we provide proofs for Section 4, including Theorem 2 and Theorem 3.

D.1 Proof of Theorem 2

Proof. To begin with, we give a different decomposition for the empirical gradient variation ¯ V T :

where the last step is by the self-bounding property (A.1) for non-negative functions. For simplicity, we denote by ¯ F T ≜ ∑ T t =1 f t ( x t ) .

The regret decomposition is the same as that in the proof of Theorem 1 (i.e., (C.2), (C.3), and (C.4)), and thus omitted here. In the following, we analyze the meta and base regret, and combine them for the final regret bounds.

Meta Regret Analysis. Our Algorithm 1 achieves the small-loss bounds without modifying the algorithm. As a result, the meta algorithm and the corresponding step size and optimism configurations are the same as that in the proof of Theorem 1.

For convex

functions, by choosing the optimism as

m

t,i

=

⟨

g

t

-

1

,

x

t

-

x

t,i

⟩

for the index

i

indicating the base learner for the convex case, the meta regret in (C.2) can be bounded as

For exp-concave functions, we choose the optimism as m t,i = 0 for indexes i indicating the expconcave base learners. The meta regret is bounded in the same way as (C.5).

For strongly convex functions, we choose the optimism as m t,i = 0 for indexes i indicating the strongly convex base learners. The meta regret is bounded in the same way as (C.6).

Base Regret Analysis. For convex functions, using the same base algorithms as in the proof of Theorem 1, the base regret can be bounded as

For exp-concave functions, using the same base algorithms as in the proof of Theorem 1, the base regret can be bounded by (C.7). Following (C.8), the empirical gradient variation defined on the surrogate function h exp t,i ( · ) can be bounded as

Plugging the surrogate's empirical gradient variation back to the base regret, we obtain

where the second step requires C 8 ≥ 1 by Lemma 5 and uses the property of the best base learner, i.e., α i ⋆ ≤ α ≤ 2 α i ⋆ . The last step is due to α i ≤ 1 .

For strongly convex functions, using the same base algorithms as in the proof of Theorem 1, the base regret can be bounded by (C.9). Following (C.10), the empirical gradient variation defined on the surrogate function h sc t,i ( · ) can be bounded as

Plugging the surrogate's empirical gradient variation back to the base regret, we obtain

where the second step requires C 9 ≥ 1 by Lemma 5 and uses the property of the best base learner, i.e., λ i ⋆ ≤ λ ≤ 2 λ i ⋆ . The last step is due to λ i ≤ 1 .

Regret Analysis. For convex functions, by combining the meta and base regret, it holds that

where the last step is due to Lemma 9 of Zhao et al. [2024], restated below for self-containedness.

Lemma 3 (Lemma 9 of Zhao et al. [2024]) . For any x, y, a, b > 0 satisfying x -y ≤ √ ax + b , it holds that x -y ≤ √ ay + ab + a + b .

For exp-concave functions, by combining the meta and base regret, it holds that

where the second step chooses C 3 = 4 C 0 α i ⋆ and C 8 = max { 1 , 32 G 2 α i ⋆ } . Note that such a parameter configuration will only add an O (1 /α ) factor to the final regret bound, which can be absorbed.

For strongly convex functions, by combining the meta and base regret, it holds that

where the second step is by choosing C 4 = 4 C 0 D 2 λ i ⋆ and C 9 = max { 1 , 256 G 2 λ i ⋆ } . Note that such a parameter configuration will only add an O (1 /λ ) factor to the final regret bound, which can be absorbed. Also note that the constants C 3 , C 4 , C 8 , C 9 only exist in analysis and thus can be chosen arbitrarily, finishing the proof.

D.2 Proof of Theorem 3

Proof. To begin with, we give a different analysis of the empirical gradient variation:

where the first step is due to Cauchy-Schwarz inequality and the last step is because of the definitions of σ 2 1: T and Σ 2 1: T (given in Section 4) and the analysis proposed in Section 3.2.

In the following, we first give regret decompositions for different curvature types, then we analyze the meta and base regret, and combine them for the final regret guarantees.

Regret Decomposition. Denoting by x ⋆ ∈ arg min x ∈X ∑ t ∈ [ T ] f t ( x ) , for convex functions, we decompose the regret as

where the first and third step use F t ( x ) = E [ f t ( x )] , the second step uses the definition of Bregman divergence, and the fourth step is due to h c t,i ( x ) ≜ ⟨ g t , x ⟩ .

For exp-concave functions, following the similar decomposition as in the proof of Theorem 1 in Appendix C, we decompose the regret as

where the second step uses the definition of the expected function F t ( · ) , the third step requires the exp-concavity of f t ( · ) , and the fourth step is due to h exp t,i ( x ) ≜ ⟨ g t , x ⟩ + α i 4 ⟨∇ f t ( x t ) , x -x t ⟩ 2 , where α i ∈ H , defined in (2.1).

For strongly convex functions, following the similar decomposition as in Appendix C, we have

where the second step, different from the exp-concave case, only requires the strong convexity of F t ( · ) , and the third step is due to h sc t,i ( x ) ≜ ⟨ g t , x ⟩ + λ i 4 ∥ x -x t ∥ 2 , where λ i ∈ H , defined in (2.1).

Meta Regret Analysis. Our Algorithm 1 can be applied to the SEA model without any algorithm modifications. As a result, we directly use the same parameter configurations as in the proof of Theorem 1 (i.e., in Appendix C).

For convex functions, the meta regret can be bounded as

where the second step is by Jensen's inequality and the last step is due to AM-GM inequality. C 10 is a constant to be specified.

For exp-concave and strongly convex functions, the meta regret is bounded in the same way as (C.5) and (C.6), and thus omitted here.

Base Regret Analysis. For convex functions, the base regret can be bounded as

where the first step is by Jensen's inequality, the second step is due to (D.2), and the last step is because of AM-GM inequality. C 11 is a constant to be specified.

For exp-concave functions, the base regret is bounded by (C.7). Following (D.2), we control the empirical gradient variation defined on surrogates as

Plugging the surrogate's empirical gradient variation back to the base regret, we obtain

where the second step requires C 12 ≥ 1 by Lemma 5.

For strongly convex functions, we need to delve into the proof details of the base algorithm, i.e., OOMD (B.2) for strongly convex functions with step size η t = 2 / (1 + λ i t ) . For example, from Lemma 12 of Yan et al. [2023], the base regret can be bounded as

Subsequently, we analyze the empirical gradient variation defined on surrogates in each round, i.e., ∥∇ h sc t,i ⋆ ( x t,i ⋆ ) -∇ h sc t -1 ,i ⋆ ( x t -1 ,i ⋆ ) ∥ 2 . Denoting by σ 2 t ≜ max x ∈X E f t ∼ D t [ ∥∇ f t ( x ) -∇ F t ( x ) ∥ 2 ] and Σ 2 t ≜ E [sup x ∈X ∥∇ F t ( x ) -∇ F t -1 ( x ) ∥ 2 ] for simplicity,

where the first step is due to the property of the surrogate: ∇ h sc t,i ( x t,i ) = g t + λ i 2 ( x t,i -x t ) , and the second step is due to the Cauchy-Schwarz inequality. Plugging the above term back into the base regret and omitting the ignorable O (1) term, we achieve

To handle the above term of ∑ T t =1 a t /t for some variable sequence { a t } T t =1 , we import a useful lemma from Yan et al. [2023].

Lemma 4 (Lemma 9 of Yan et al. [2023]) . For a sequence of { a t } T t =1 and b , where a t , b > 0 for any t ∈ [ T ] , denoting by a max ≜ max t a t and A ≜ ⌈ b ∑ T t =1 a t ⌉ , we have

Using Lemma 4, we control the base regret as

where the first term initializes Lemma 4 as a t = σ 2 t + σ 2 t -1 +Σ 2 t (i.e., a max = O ( σ 2 max +Σ 2 max ) ) and b = 1 / ( σ 2 max +Σ 2 max ) , the second term initializes Lemma 4 as a t = D F t ( x ⋆ , x t ) + D F t -1 ( x ⋆ , x t -1 ) (i.e., a max = 4 GD due to Assumption 1) and b = λ i ⋆ , the third term initializes Lemma 4 as a t = λ 2 i ⋆ ∥ x t,i ⋆ -x t ∥ 2 + λ 2 i ⋆ ∥ x t -1 ,i ⋆ -x t -1 ∥ 2 (i.e., a max = 2 D 2 due to λ i ≤ 1 and Assumption 1) and b = λ i ⋆ . The O (1) term contains ignorable terms like O (1 /λ ) . The second step requires C 13 , C 14 ≥ 1 by Lemma 5.

Regret Analysis. For convex functions, by combining the meta and base regret, it holds that

by choosing 10 0 11

For exp-concave functions, by combining the meta and base regret, it holds that

by choosing C 12 = max { 1 , 240 L, 32 G 2 α i ⋆ } and C 3 = 4 C 0 α i ⋆ . Note that such a parameter configuration will only add an O (1 /α ) factor to the final regret bound, which can be absorbed.

For strongly convex functions, by combining the meta and base regret, it holds that

by choosing C 13 = max { 1 , 1920 LGD } , C 14 = max { 1 , 128 D 2 λ i ⋆ } and C 4 = 4 C 0 D 2 λ i ⋆ . Note that such a parameter configuration will only add an O (1 /λ ) factor to the final bound, which can be absorbed.

Note that the constants C 3 , C 4 , C 10 , C 11 , C 12 , C 13 , C 14 only exist in analysis and thus can be chosen arbitrarily, finishing the proof.

E Technical Lemmas

Lemma 5. For any a > 1 , b > 0 , it holds that ln( a + b ) ≤ ln( Ca ) + b C for some C ≥ 1 .

Proof. The one-line proof is presented below:

where the first step is due to C ≥ 1 , and the last step adopts ln(1 + x ) ≤ x for any x ≥ 0 .

Lemma 6 (Corollary 5 of Orabona et al. [2012]) . If a, b, c, d, x > 0 satisfy x -d ≤ a ln( bx + c ) , then it holds that

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have claimed the paper's contribution in both the abstract and the introduction part.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: One limitation we could discover is that our approach is not applicable in multi-player games since it does not control the algorithmic stability, which is essential and necessary in achieving fast rates in games.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are provided and justified in Section 2. We provide theoretical guarantees in Section 3 and Section 4, and all the corresponding proofs can be found in Appendix C and Appendix D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is a purely theoretical work, and we do not find specific societal impacts that should be highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not include experiments (data or models), and thus poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies universal Online Convex Optimization (OCO) with gradient-variation-dependent regret bounds. That is, to design one single algorithm that is unaware of but is able to adapt to both two following groundtruth: 1) the type of curvatures: the loss functions could be convex, strongly convex, or exp-concave; 2) the curvature coefficient: exp-concavity $\\alpha$ or strong convexity $\\lambda$. As a result, the regret guarantee achieved by the algorithm scales with the cumulative gradient variation $V_T$ (which depends on the loss function sequence), rather than the time horizon $T$, as well as the corresponding curvature type of the underlying loss functions.\n\nThis paper proposes a new simple algorithm, that for the first time achieves optimal gradient-variation bounds for all three curvature types. Note that the gradient-variation bounds immediately imply small-loss (aka first-order) regret bounds as well as worst-case bounds. The #base learners is also improved from $(\\log T)^2$ to $\\log T$ due to the two-layer structure. The main result also finds broad applications including the SEA model and dynamic regret bounds.\n\nTechnique-side, the improvement comes from an alternative way to analyze the the empirical gradient variation w.r.t. surrogate losses and utilizing a negative Bregmen divergence term (due to linearization) to cancel other positive terms, which is often omitted in the analysis.\n\nSTRENGTHS:\nI overall like such results. The authors present their observations and insights (from the regret analysis) in detail, leading to improved (and indeed optimal) regret bounds and even (conceptually) simpler algorithm design.\n\nWEAKNESSES:\nI didn’t spot any significant technical issues, and I'm just suggesting some minor “weakness”.\n\n1. When the authors introduce the notion of $F_T$ and small-loss bound for the first time (around Eq. (1.2)), they may want to add that now the loss functions are non-negative (which I think should be necessary for all small-loss/first-order bounds?). Obviously, one can’t take squared root or logarithmic to a negative number.\n\n2. In the application to dynamic regret, the problem setup is not clearly defined. What is the type of loss function? Is the strong-convexity/log-concavity known? It is particularly confusing since it’s right after the universal OCO setup.\n\nQUESTIONS:\n1. The idea of utilizing negative Bregmen divergence terms also appeared in other problems, such as high-probability regrets in adversarial bandits [1]. Could the authors comment on the connection (if any) between the use therein and this work?\n\n2. Under the universal OCO setup, is it possible to handle time-varying curvatures, just like in [2]?\n\n3. Seems that a universal OCO algorithm cannot be anytime? The reason is that the number of base learner (for discretization) depends on $T$.\n\nReferences\n\n[1] Lee, Chung-Wei, Haipeng Luo, Chen-Yu Wei, and Mengxiao Zhang. \"Bias no more: high-probability data-dependent regret bounds for adversarial bandits and mdps.\" Advances in neural information processing systems 33 (2020): 15522-15533. https://arxiv.org/abs/2006.08040\n\n[2] Luo, Haipeng, Mengxiao Zhang, and Peng Zhao. \"Adaptive bandit convex optimization with heterogeneous curvature.\" In Conference on Learning Theory, pp. 1576-1612. PMLR, 2022. https://proceedings.mlr.press/v178/luo22a/luo22a.pdf\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studied the problem of regret minimization of a set of functions $\\{f_t\\}_{t=1}^{T}$ over a compact and convex constraint set $\\mathcal{X}$, i.e.,\n$\\sum{t=1}^{T}f_{t}(x_t) - \\text{min}{x\\in\\mathcal{X}}\\sum{t=1}^{T}f_{t}(x),$\nwhere $x_t$ is the output of the proposed algorithm at round $1\\leq t\\leq T$.\nThe set of functions ${f_t}_{t=1}^{T}$ potentially satisfy certain curvature assumptions, e.g., strong convexity, convexity, or exp-concavity. In the paper, it is unknown which curvature assumption the function satisfies. The main goals of the paper are the following:\n\n1. To construct a universal algorithm that adaptively acts on the curvature property of the function and achieves a proper regret bound.\n2. For the case where the function is $\\lambda$-strongly convex or $\\alpha$-exp-concave, the algorithm should be adapted with respect to the curvature parameter, $\\lambda$ to $\\alpha$.\n3. The algorithm should achieve a good problem-dependent regret bound: The goal of the paper is to attain a regret bound that depends on the following quantities:\n$V_T = \\sum_{t=1}^{T}\\text{sup}_{x\\in \\mathcal{X}} \\| \\nabla f_{t}(x) - \\nabla_f{t-1} (x) \\|^2, \\quad \\text{and}\\quad F_T = \\text{min}{x\\in\\mathcal{X}}\\sum{t=1}^{T}f_t(x).$\n\nThe proposed algorithm of the paper is a modification of the algorithm proposed by [1]. Similar to the approach introduced by [1], in Algorithm 1 (page 5) of the paper, the authors proposed $\\emph{base learners}$ that are aggregated by a meta-algorithm, which outputs the final output of the algorithm at round $t$, $x_t$. The contribution of the paper mainly concerns the technical aspects that outperform the performance of [1] from the following points of view:\n\n1.The paper improves the number of required base learners from $\\log(T)^2$ (in [1]) to $\\log(T)$.\n\n2.This improvement of the algorithm outperforms the algorithm proposed by [1] up to a logarithmic factor for the situation where the loss functions $f_t$ are convex.\n\n[1] Y.-H. Yan, P. Zhao, and Z.-H. Zhou. Universal online learning with gradual variations: A multi-layer online ensemble approach. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.\n\nSTRENGTHS:\nThe paper uses simple but interesting technique that contributes in tigher bounds for the case of convex losses. Inspired by [2] the authors used that exploit the imposed smoothness assumption of the loss function and Bregman divergence negative term from linearization of loss function, explained in Section 3.2.\n\n\n\n[2] P. Joulani, A. Raj, A. Gyorgy, and C. Szepesvari. A simpler approach to accelerated optimization: iterative averaging meets optimism. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.\n\nWEAKNESSES:\nThe main weakness of the paper lies in its presentation. The content is too dense, and the last section on dynamic regret could be moved to the appendix. Some key parts of the paper are not well explained. For instance, it is unclear how the authors managed to outperform the number of required base learners in [1] by a logarithmic factor. Was this achieved through the application of a Bregman divergence negative term?\n\nThe contribution of the paper is limited to a simple technical improvement that enhances the achieved regret up to a logarithmic factor for convex functions.\n\nThe optimality of the result with respect to $V_T$ and $F_T$ has not been discussed by the authors.\n\n\n[1] Y.-H. Yan, P. Zhao, and Z.-H. Zhou. Universal online learning with gradual variations: A multi-layer online ensemble approach. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.\n\nQUESTIONS:\n1. I do not understand the comment on the small $\\alpha$ and $\\lambda$ in lines 146-147. Can these cases be considered convex? For the convex case, the rate is of the order of $\\sqrt{T}$. How do the optimal minimax results hold for this regime, which indicates that the regret is linear?\n\n2. I would appreciate it if the authors could explain the question I raised in the weakness section and outline the main difference that helps them improve the number of base learners.\n\n3. I do not understand the comment in line 305: \"which can be easily canceled by the negative term from curvatures in the meta regret.\" For this cancellation to occur, the coefficient in the equation after line 304 really matters. Could the authors explain this further?\n\n4. Could the authors explain if the final result is optimal with respect to $V_T$ and $F_T$?\n\nMinor Points:\n\nThe terms $\\sigma_{\\text{\\max}}$ and $\\Sigma_{\\text{\\max}}$ are not defined in Theorem 3.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper investigates the problem of universal online convex optimization to achieve problem dependent regret guarantees for different classes of convex functions (strongly convex, exp-concave, and convex) simultaneously. Problem/function/data dependent regret guarantees have become popular in literature to bridge stochastic and adversarial guarantees.

STRENGTHS:
S1) The paper is well written and easy to understand.

S2) The literature review is comprehensive and up to date.

S3) Simplicity of the incorporation of Bregman divergence is a plus.

WEAKNESSES:
W1) The contribution seems limited in that the improvement is only logarithmic for both efficiency and regret results.

W2) While the regret analysis is novel, algorithmic contribution is very limited, which leads me to believe this paper is more suitable to be a technical note.

QUESTIONS:
Q1) Why is $\log^2 T$ computational complexity claimed to be inefficient throughtout the paper? In Table 1, the number of gradient queries and base learners are given as part of efficiency, however, a decrease on the number of queries seems much more significant to me.

Q2) Improvement over the results of Yan et al. [2023] seems incremental. Are there scenarios where this improvement becomes significant?

Q3) Is your approach the same as Zhang et al. [2022]  but using Optimistic ADAPT-ML-PROD [Wei et al., 2016] instead of  ADAPT-ML-PROD [Gaillard et al., 2014]?

LIMITATIONS:
No significant limitations. Necessary assumptions about the problem setting are properly addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the regret minimization problem in online convex optimization without access to curvature information. They tackle the task of achieving problem-dependent optimal regret while requiring no prior knowledge of the function class (convex, exp-concave, or strongly convex). They propose an efficient-to-implement two-layer online ensemble structure that requires only one gradient query within each round. Their main technical novelty lies in providing a novel approach for gradient-variation bounds.\n\nSTRENGTHS:\nThe authors tackle a very interesting problem in online convex optimization. The paper is well-written and the presentation makes it easy to follow. The main novelty lies in Sections 3.2 and 3.3, where they provide a new way of tackling gradient variations by utilizing the Bregman divergence term. They also make clever use of Proposition 1 in their analysis. The overall method utilizes techniques from several existing works and cleverly combines them to achieve an impressive bound on the regret.\n\nWEAKNESSES:\nThe proposed approach seems reasonable to me. While I have not gone through the technical details very carefully, I seek one clarification on the proof of Theorem 1. In my opinion, the bottleneck of the proof is in showing the existence of an appropriate choice of $C_3$ and $C_4$ (page 17, line 594, 596). Can the authors comment if such a setting always exists? I would at least expect certain conditions like $\\alpha_i^* > G^2/9L $ or $\\lambda_i^* > 1/9L$ for results to hold.\n\nAnother small thing: I understand the authors ignore very small terms like $\\log \\log T$ from the order notation. It might be good to put a note in the introduction about it while presenting the result. I understand that it is there in Section 3.1 -- it might be good to move it earlier.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}