Title: Gradient-Variation Online Learning under Generalized Smoothness
Venue: NeurIPS 2024
Accepted: yes
Keywords: online learning

generalized smoothness

gradient-variation regret bound

Lipschitz-adaptive algorithm
Abstract: Gradient-variation online learning aims to achieve regret guarantees that scale with variations in the gradients of online functions, which is crucial for attaining fast convergence in games and robustness in stochastic optimization, hence receiving increased attention. Existing results often require the smoothness condition by imposing a fixed bound on gradient Lipschitzness, which may be unrealistic in practice. Recent efforts in neural network optimization suggest a generalized smoothness condition, allowing smoothness to correlate with gradient norms. In this paper, we systematically study gradient-variation online learning under generalized smoothness. We extend the classic optimistic mirror descent algorithm to derive gradient-variation regret by analyzing stability over the optimization trajectory and exploiting smoothness locally. Then, we explore universal online learning, designing a single algorithm with the optimal gradient-variation regrets for convex and strongly convex functions simultaneously, without requiring prior knowledge of curvature. This algorithm adopts a two-layer structure with a meta-algorithm running over a group of base-learners. To ensure favorable guarantees, we design a new Lipschitz-adaptive meta-algorithm, capable of handling potentially unbounded gradients while ensuring a second-order bound to effectively ensemble the base-learners. Finally, we provide the applications for fast-rate convergence in games and stochastic extended adversarial optimization.

Gradient-Variation Online Learning under Generalized Smoothness

Abstract

Gradient-variation online learning aims to achieve regret guarantees that scale with variations in the gradients of online functions, which is crucial for attaining fast convergence in games and robustness in stochastic optimization, hence receiving increased attention. Existing results often require the smoothness condition by imposing a fixed bound on gradient Lipschitzness, which may be unrealistic in practice. Recent efforts in neural network optimization suggest a generalized smoothness condition, allowing smoothness to correlate with gradient norms. In this paper, we systematically study gradient-variation online learning under generalized smoothness. We extend the classic optimistic mirror descent algorithm to derive gradient-variation regret by analyzing stability over the optimization trajectory and exploiting smoothness locally. Then, we explore universal online learning , designing a single algorithm with the optimal gradient-variation regrets for convex and strongly convex functions simultaneously, without requiring prior knowledge of curvature. This algorithm adopts a two-layer structure with a meta-algorithm running over a group of base-learners. To ensure favorable guarantees, we design a new Lipschitz-adaptive meta-algorithm, capable of handling potentially unbounded gradients while ensuring a second-order bound to effectively ensemble the base-learners. Finally, we provide the applications for fast-rate convergence in games and stochastic extended adversarial optimization.

1 Introduction

Weconsider online convex optimization (OCO) [Hazan, 2016; Orabona, 2019], a flexible framework that models the decision-making problem in an online fashion. At each round t ∈ [ T ] , an online learner is required to submit a decision x t from a convex compact set X ⊆ R d and the environments reveal a convex function f t : X ↦→ R . Then the learner suffers a loss f t ( x t ) and updates her decision. The standard performance measure is the regret [Zinkevich, 2003] that benchmarks the cumulative loss of the learner against the best decision in hindsight, formally defined as

Regret bounds of O ( √ T ) and O ( 1 λ log T ) are established for convex and λ -strongly convex functions respectively [Zinkevich, 2003; Hazan et al., 2007]. While these results are known to be minimax optimal [Abernethy et al., 2008], in this paper we are more interested in obtaining gradientvariation regret guarantees, which replace the dependence of T in the regret bounds by variations in

the gradients of online functions [Chiang et al., 2012] defined as

This quantity can be as small as a constant in stable environments where online functions remain fixed, and is at most O ( T ) in the worst case under standard OCO assumptions, safeguarding minimax results. Besides this favorable adaptivity, recent studies have shown close relationships of gradient-variation online learning to various fields, including fast convergence in games [Rakhlin and Sridharan, 2013b; Syrgkanis et al., 2015; Zhang et al., 2022b] and robust stochastic optimization [Sachs et al., 2022; Chen et al., 2024], hence receiving increased attention [Zhao et al., 2020; Yan et al., 2023; Tsai et al., 2023; Ataee Tarzanagh et al., 2024; Zhao et al., 2024].

In online learning, it is proved that the smoothness assumption is necessary for first-order algorithms to achieve gradient-variation regret bounds as discussed in Remark 1 of Yang et al. [2014], which is also restated in Proposition 2 in Appendix B. Previous works typically rely on the global L -smoothness condition, imposing a fixed upper bound on the gradient Lipschitzness, i.e., requiring ‖∇ 2 f t ( x ) ‖ 2 ≤ L for all t ∈ [ T ] and x ∈ X . However, this global assumption restricts the applicability of theories to loss functions that are quadratically bounded from above. Furthermore, recent studies in neural network optimization have observed phenomena where the global smoothness condition fails to model optimization dynamics effectively, especially for important types of neural networks like LSTM [Zhang et al., 2020b] and Transformer [Crawshaw et al., 2022]. Therefore, modern optimization has devoted efforts to generalizing the smoothness condition. For example, Zhang et al. [2020b] introduce ( L 0 , L 1 ) -smoothness, which assumes ‖∇ 2 f ( x ) ‖ 2 ≤ L 0 + L 1 ‖∇ f ( x ) ‖ 2 for an offline objective function f ( · ) . A notable generalization is the recent proposal of the /lscript -smoothness condition [Li et al., 2023], which assumes ‖∇ 2 f ( x ) ‖ 2 ≤ /lscript ( ‖∇ f ( x ) ‖ 2 ) with a link function /lscript ( · ) , significantly broadening previous assumptions through the flexibility of /lscript ( · ) . Given this, it is natural to ask how to design online algorithms to exploit generalized smoothness and obtain favorable gradient-variation regret guarantees.

In this paper, we provide a systematic study of gradient-variation online learning under generalized smoothness. We extend the classic optimistic online mirror descent (optimistic OMD) algorithm [Chiang et al., 2012; Rakhlin and Sridharan, 2013a] to derive gradient-variation regret bounds, achieving O ( √ V T ) regret and O (log V T ) regret for convex and strongly convex functions under generalized smoothness, respectively. We emphasize the importance of stability analysis across the optimization trajectory, which allows generalized smoothness to be effectively exploited locally. Specifically, optimistic OMD maintains two sequences with submitted decisions { x t } T t =1 and intermediate decisions { ̂ x t } T t =1 . We need to control algorithmic stability by appropriate step size tuning and optimism design, ensuring that x t is sufficiently close to x t to exploit local smoothness at x t .

̂ ̂ Based on this development, we investigate universal online learning [van Erven and Koolen, 2016; Wang et al., 2019; Mhammedi et al., 2019; Zhang et al., 2022a; Yan et al., 2023; Yang et al., 2024], where the learner aims to design a single algorithm that simultaneously attains the optimal regret for both convex and strongly functions without the prior knowledge of curvature information. For this scenario, a common wisdom is to adopt an online ensemble consisting of a meta-base two-layer structure to handle the environmental uncertainty [Zhao et al., 2024], i.e., the unknown curvature of loss functions, where a meta-algorithm is running over a set of base-learners with different configurations. The base-learners are basically the instantiations of the developed variants of optimistic OMD, as mentioned earlier. However, designing the meta-algorithm is non-trivial with new challenges. The first challenge is from the potentially unbounded smoothness, which might lead to unbounded Lipschitz constants as well. This challenge requires the meta-algorithm to be Lipschitz-adaptive , adapting to Lipschitzness on the fly. Furthermore, we also expect it to provide a second-order regret , technically required when analyzing the ensemble errors, and to enable predictions with optimism , thereby producing the gradient variation. The second challenge is the complexity introduced by the combination procedure inherent in the ensemble method, which further complicates the smoothness estimation, making it difficult to properly tune the meta-algorithm and exploit smoothness.

To this end, we address both challenges with the function-variation-to-gradient-variation conversion and a newly-designed Lipschitz-adaptive meta-algorithm. The conversion technique, drawing inspiration from Bai et al. [2022], decouples the design between the meta and base levels and derives the gradient variation directly from function values, allowing us to avoid the cancellation-based analysis [Yan et al., 2023] for utilizing smoothness at the meta level. Nevertheless, this conversion

requires the meta-algorithm to handle heterogeneous inputs due to certain technical considerations, and we are not aware of available algorithms satisfying all the requirements, motivating us to design a new algorithm. Based on optimistic Adapt-ML-Prod [Wei et al., 2016] and the clipping technique [Cutkosky, 2019], we present a new Lipschitz-adaptive meta-algorithm with a simpler algorithmic design, which can be of independent interest. With this algorithm, we can apply the function-variation-to-gradient-variation conversion to achieve the optimal results for both convex and strongly convex functions, up to doubly logarithmic factors of T , without knowing curvature.

Our findings for gradient-variation online learning are useful for several important applications, including fast-convergence online games [Rakhlin and Sridharan, 2013a; Syrgkanis et al., 2015] and stochastic extended adversarial online learning [Sachs et al., 2022], where we establish new results under the generalized smoothness condition.

The rest of paper is organized as follows. Section 2 provides preliminaries and key ideas for exploiting the generalized smoothness throughout the trajectory. In Section 3 we study universal online learning and present our key meta-algorithm. Section 4 discusses our applications. Related work is provided in Appendix A. All proofs can be found in the remaining appendices (Appendix B - D).

2 Gradient-Variation Online Learning under Generalized Smoothness

In this section, we first introduce the problem setup, including the formal definition of generalized smoothness and other assumptions used in the paper. We then extend the optimistic online mirror descent framework to achieve gradient-variation regret bounds under generalized smoothness.

2.1 Problem Setup: Generalized Smoothness and Assumptions

Recent studies [Zhang et al., 2020b; Chen et al., 2023b] extend the global smoothness condition by allowing the smoothness to positively correlate with the gradient norm, where a particular function is required to model this relationship. Zhang et al. [2020b] introduce the ( L 0 , L 1 ) -smoothness condition, where the smoothness is upper bounded by a linear function of the gradient norm, i.e., ‖∇ 2 f ( x ) ‖ 2 ≤ L 0 + L 1 ‖∇ f ( x ) ‖ 2 . Li et al. [2023] further generalize this by imposing a weaker assumption on the link function and propose the generalized smoothness defined as follows.

Definition 1 ( /lscript -smoothness) . A twice-differentiable function f : X ↦→ R is called /lscript -smooth for some non-decreasing continuous link function /lscript : [0 , + ∞ ) ↦→ (0 , + ∞ ) if it satisfies that ‖∇ 2 f ( x ) ‖ 2 ≤ /lscript ( ‖∇ f ( x ) ‖ 2 ) for any x ∈ X .

The mild requirement on the link function /lscript ( · ) allows for considerable generality. By selecting a linear link function, /lscript -smoothness immediately recovers ( L 0 , L 1 ) -smoothness [Zhang et al., 2020b]. Furthermore, it has been shown that /lscript -smoothness can imply a wide class of functions including rational, logarithmic, and self-concordant functions [Li et al., 2023]. Based on this generalized smoothness notion, we now provide the formal assumption on the smoothness of online functions.

Assumption 1 (generalized smoothness) . The online function f t : X ↦→ R is /lscript t -smooth in an open set containing X ⊆ R d for t ∈ [ T ] , and the learner can query /lscript t ( x ) provided any point x ∈ X .

We also require a standard bounded domain assumption in the OCO literature [Hazan, 2016].

Assumption 2 (bounded domain) . The feasible domain X ⊆ R d , which contains the origin 0 , is non-empty and closed with the diameter bounded by D , i.e., ‖ x -y ‖ 2 ≤ D for any x , y ∈ X .

We do not assume the prior knowledge of the Lipschitz constant of online functions. In fact, the unboundedness of smoothness may result in unbounded Lipschitz constants. If a Lipschitz upper bound were known, the generalized smoothness condition would be trivialized, as it would allow us to directly compute the upper bound of the smoothness constant. Furthermore, following the discussion in Jacobsen and Cutkosky [2023, Page 2, second paragraph on the right], we assume that there exist finite but unknown upper bounds G and L for Lipschitzness and smoothness to ensure the theoretical results are valid. Note that these quantities will only appear in the final regret bounds, and our algorithms does not use them as the inputs. Throughout the paper, we use the O ( · ) -notation to hide the constants and use the ˜ O ( · ) -notation to omit the poly-logarithmic factors in T .

2.2 Algorithmic Framework

Wechoose optimistic online mirror descent (optimistic OMD) [Rakhlin and Sridharan, 2013a] as the algorithmic framework, which provides a unified view to design and analyze many online algorithms. Compared to classic OMD [Nemirovskij and Yudin, 1985; Beck and Teboulle, 2003], optimistic OMD predicts with side information, an optimistic vector M t ∈ R d . This optimistic vector, also known as optimism, serves as a prediction of the incoming function f t +1 ( · ) , leading to tighter regret bounds when accurate. Optimistic OMD updates the decisions in two steps:

where D ψ t ( x , y ) = ψ t ( x ) -ψ t ( y ) -〈∇ ψ t ( y ) , x -y 〉 is the Bregman divergence associated with the regularizer ψ t : X ↦→ R . Optimistic OMD maintains two sequences: the sequence of submitted decisions { x t } T t =1 , and the one of intermediate decisions { ̂ x t } T t =1 . Although a simplified optimistic OMDwith one-step update per round exists [Joulani et al., 2020], we will demonstrate later that tuning the step size based on intermediate decisions is crucial for adapting to generalized smoothness.

2.3 Gradient-Variation Regret for Convex and Strongly Convex Functions

When minimizing the convex or strongly convex functions, we set the regularizer as ψ t ( x ) = 1 2 η t ‖ x ‖ 2 2 and optimistic OMD updates with the following steps:

x t = Π X [ ̂ x t -η t M t ] , ̂ x t +1 = Π X [ ̂ x t -η t ∇ f t ( x t )] , (4) where Π X [ y ] = arg min x ∈X ‖ x -y ‖ 2 denotes the Euclidean projection operator. Next, we briefly review approaches for obtaining the gradient-variation bound under global smoothness . This bound typically follows from the regret analysis for optimistic OMD:

On the right-hand side, the second term is known as the stability term, while the third one is the negative terms that can be further bounded by O ( -∑ T t =1 1 η t ‖ x t -x t -1 ‖ 2 2 ) . Previous studies [Chiang et al., 2012; Zhao et al., 2024] for gradient-variation regret under global smoothness often set optimism as M t = ∇ f t -1 ( x t -1 ) , such that, the positive stability term can be upper bounded by η t ‖∇ f t ( x t ) -∇ f t -1 ( x t ) ‖ 2 2 + η t ‖∇ f t -1 ( x t ) -∇ f t -1 ( x t -1 ) ‖ 2 2 , where the first part can be directly converted to the desired gradient variation and the second part will be at most η t L 2 ‖ x t -x t -1 ‖ 2 2 under global smoothness. Given the smoothness constant L , tuning the step size as η t ≤ 1 / (4 L ) ensures O ( η t L 2 ‖ x t -x t -1 ‖ 2 2 -1 η t ‖ x t -x t -1 ‖ 2 2 ) ≤ 0 , thus obtaining the gradient-variation bound.

However, under generalized smoothness, we do not have a global parameter L for setting the step sizes, and the smoothness constants are related to the decisions. To follow the previous approach, optimistic OMD would require the smoothness constant before generating x t to tune the step size, ensuring that the negative terms are large enough to cancel η t ‖∇ f t -1 ( x t ) -∇ f t -1 ( x t -1 ) ‖ 2 2 . Nevertheless, the smoothness constant between x t and x t -1 can only be evaluated after updating to x t , resulting in a contradiction. Unlike offline optimization, where the function is fixed and smoothness constants can be shown to decrease along the trajectory [Li et al., 2023], in online optimization, the online functions change at each round, preventing the reuse of previous smoothness estimations.

To address this challenge, our key idea is to perform a trajectory-wise analysis and configure the algorithm using estimated smoothness so far. The key technical lemma is the local smoothness property of /lscript t -smooth functions [Li et al., 2023], which allows the smoothness constant between two points to be estimated in advance, provided that the two points are close enough.

Lemma 1 (local smoothness [Li et al., 2023, Lemma 3.3]) . Suppose f : X ↦→ R is /lscript -smooth. For ∀ x , y ∈ X such that ‖ x -y ‖ 2 ≤ ‖∇ f ( x ) ‖ 2 /lscript t (2 ‖∇ f ( x ) ‖ 2 ) , ‖∇ f ( x ) -∇ f ( y ) ‖ 2 ≤ /lscript (2 ‖∇ f ( x ) ‖ 2 ) · ‖ x -y ‖ 2 .

Recall that in the update procedures (3) of optimistic OMD, the submitted decision x t is updated based on the intermediate decision ̂ x t . Therefore, it is convenient to control their distance and then exploit the local smoothness at point ̂ x t . Specifically, we set optimism M t = ∇ f t -1 ( ̂ x t ) and the step size η t ≤ 1 / (4 ̂ L t -1 ) , where ̂ L t -1 = /lscript t -1 (2 ‖∇ f t -1 ( ̂ x t ) ‖ 2 ) denotes the locally estimated

smoothness and is used to tune the step size. This configuration leads to η t ‖∇ f t ( x t ) -∇ f t -1 ( ̂ x t ) ‖ 2 2 for the second term in Eq. (5), which can be further upper bounded as

Theorem 1. Under Assumptions 1 - 2 and assuming online functions are convex, we set the optimism as M t = ∇ f t -1 ( x t ) and f 0 ( · ) = 0 , with step sizes as η 1 = D and, for t ≥ 2 ,

‖∇ f t ( x t ) -∇ f t -1 ( ̂ x t ) ‖ 2 2 ≤ 2 ‖∇ f t ( x t ) -∇ f t -1 ( x t ) ‖ 2 2 +2 ‖∇ f t -1 ( x t ) -∇ f t -1 ( ̂ x t ) ‖ 2 2 . (6) The first part is basically the favorable gradient variation, so it suffices to handle the second part. Performing the stability analysis for OMD and noticing the step size setting, it can be verified that ‖ x t -̂ x t ‖ 2 ≤ η t ‖∇ f t -1 ( ̂ x t ) ‖ 2 ≤ ‖∇ f t -1 ( ̂ x t ) ‖ 2 / (4 ̂ L t -1 ) . This satisfies the criteria for applying Lemma 1 to the /lscript t -1 -smooth function f t -1 ( · ) , allowing us to upper bound the second term in (6) by O ( ̂ L 2 t -1 · ‖ x t -̂ x t ‖ 2 2 ) . We have clipped η t by 1 / (4 ̂ L t -1 ) , thereby ensuring the negative term is sufficient to cancel out the positive term. Below, we summarize the result for convex functions.

where V T = ∑ T t =2 sup x ∈X ‖∇ f t ( x ) -∇ f t -1 ( x ) ‖ 2 2 measures the gradient variations and ̂ L max = max t ∈ [ T ] L t is the maximum smoothness constant over the optimization trajectory.

̂ This result implies a tighter bound in scenarios where the environments change slowly, i.e., V T = O (1) . Meanwhile, it safeguards the worst-case optimal result since V T ≤ O ( T ) holds in all cases. When assuming /lscript t ( · ) ≤ L for t ∈ [ T ] , the /lscript t -smoothness condition degenerates to the classic global L -smoothness condition, and our result implies an O ( √ V T + LD 2 ) bound, which matches the best-known gradient-variation regret bounds with the first-order oracle [Chiang et al., 2012; Yan et al., 2023; Zhao et al., 2024] even in terms of the dependence on D and L . Compared to offline optimization, our result depends on ̂ L max , the maximum smoothness constant along the trajectory. This dependence arises from the adversarial nature of online learning, where the loss functions chosen in consecutive rounds may differ significantly, making it hopeless to leverage the previous estimates of smoothness to improve the dependence.

We further provide an improved gradient-variation regret bound for strongly convex functions, with step size tuning based on recent result under global smoothness [Chen et al., 2024, § 3.4].

Theorem 2. Under Assumptions 1 - 2 and assuming online functions are λ -strongly convex, we set the optimism as M t = ∇ f t -1 ( ̂ x t ) , f 0 ( · ) = 0 , and step sizes as η 1 = 2 /λ and, for t ≥ 2 , η t = 2 / ( λt +16max s ∈ [ t ] /lscript s -1 (2 ‖∇ f s -1 ( ̂ x s ) ‖ 2 )) , optimistic OMD in (4) ensures the regret bound REG T ≤ O ( 1 λ log V T + L max · D 2 ) , where L max = max t ∈ [ T ] L t .

̂ ̂ ̂ The above theorem requires the knowledge of curvature information λ . In Section 3, we design a universal method to remove this requirement and achieve the optimal guarantees for convex and strongly convex functions simultaneously without knowing λ . In Appendix B.2, we discuss the challenge to obtain a gradient-variation bound for exp-concave functions under Assumption 1.

3 Universal Online Learning under Generalized Smoothness

Classic online learning algorithms require the curvature information of online functions as algorithmic parameters to achieve favorable regret guarantees. However, obtaining these curvature parameters can be difficult in practice. This challenge motivates the recent study of universal online learning [van Erven and Koolen, 2016; Cutkosky and Boahen, 2017; Wang et al., 2019; Mhammedi et al., 2019; Zhang et al., 2022a; Yan et al., 2023; Yang et al., 2024], which aims to design a single algorithm that can achieve optimal regrets without knowing the curvature information. In this section, we study universal online learning with gradient-variation regret under generalized smoothness.

3.1 Reviewing Related Work and Techniques

We review related work on gradient-variation universal online learning under global smoothness [Zhang et al., 2022a; Yan et al., 2023]. To handle the unknown curvature, universal online learning algorithms utilize a two-layer structure, consisting of a meta-algorithm that ensembles a group of base-learners. Each base-learner optimizes functions with a specific convex curvature, while the meta-algorithm is designed to ensure that ensemble errors do not ruin base-learners' guarantees. Denoted by N the number of base-learners, the decision x t = ∑ i ∈ [ N ] p t,i x t,i submitted by a twolayer structure algorithm comprises two key components: p t ∈ ∆ N , the weights provided by the meta-algorithm, and x t,i , the decision of the i -th base-learner. The analysis of a universal algorithm begins by decomposing the regret into two parts against any base-learner. In particular, we choose the base-learner with the best performance (the index i /star is unknown) as the benchmark:

where the first part is the meta-regret, evaluating the meta-algorithm's performance against the best base-learner, and the second part, known as the base-regret, measures the best learner's performance.

Zhang et al. [2022a] advocate for a simple approach by employing the meta-algorithms with secondorder regret guarantees, which facilitates the analysis at the meta level. In specific, they use Adapt-ML-Prod [Gaillard et al., 2014] as the meta-algorithm, showing that the meta-regret for strongly convex and exp-concave functions are constants by exploiting the negative terms from convexity. Consider λ -strongly convex functions as an example and assume the i /star -th base-learner ensures the optimal O ( 1 λ log V T ) base-regret. At the meta level, Zhang et al. [2022a] pass the linearized regret r t,i = 〈∇ f t ( x t ) , x t -x t,i 〉 to the meta-algorithm for each base-learner. By strong convexity and the guarantees of Adapt-ML-Prod, the meta-regret can be bounded by a constant:

where the last inequality follows from √∑ t 〈∇ f t ( x t ) , x t -x t,i /star 〉 2 ≤ ̂ G max √∑ t ‖ x t -x t,i /star ‖ 2 2 and is then canceled by the negative terms via the AM-GM inequality. By leveraging the negative terms from strong convexity, the meta-regret can be well-bounded, allowing the overall regret to be dominated by the base-regret, which is then controlled by selecting appropriate base-algorithms.

However, this method is unsuitable for producing the gradient-variation bound for convex functions. To address it, Yan et al. [2023] propose to use a meta-algorithm which ensures an optimistic and second-order regret bound while provides additional negative terms -∑ t ‖ p t -p t -1 ‖ 2 1 . Besides showing that the meta-regret is a constant for strongly convex and exp-concave functions following the previous approach, with newly designed optimism, Yan et al. [2023] prove that the meta-regret for convex functions can be roughly bounded by:

The first term is the gradient variation, matching the optimal order of convex functions. Yan et al. [2023] demonstrate that the remaining stability terms can be canceled through the collaboration between the base and meta levels [Zhao et al., 2024] with the prior knowledge of the global smoothness constant L , thus obtaining the near-optimal gradient-variation bounds for convex functions as well. Nevertheless, the employed meta-algorithm already has a two-layer structure, resulting in a three-layer structure for the overall algorithm, which is relatively complicated.

3.2 Key Challenges and Main Ideas

We aim to design a universal algorithm with the optimal gradient-variation bounds under generalized smoothness, which exhibits two challenges. First, the Lipschitz condition of online functions is unknown to the meta-algorithm, which requires the meta-algorithm to be Lipschitz-adaptive , provide a second-order regret , and enable predictions with optimism . Second, the combination of the

ensemble method further complicates the estimation of smoothness constants, making it challenging to tune algorithms properly and to cancel stability terms as Yan et al. [2023] did.

We tackle the second challenge by utilizing a function-variation-to-gradient-variation conversion to derive the gradient-variation bounds at the meta level, drawing inspiration from the development of dynamic regret minimization [Bai et al., 2022]. This conversion technique decouples the meta and base levels, allowing us to avoid cancellation-based analysis. To illustrate, suppose a meta-algorithm ensuring O ( √∑ t ( /lscript t,i /star -m t,i /star ) 2 ) provided optimism m t = ( m t, 1 , . . . , m t,N ) . By setting /lscript t,i /star = f t ( x t,i /star ) -f t ( x ref ) and m t,i /star = f t -1 ( x t,i /star ) -f t -1 ( x ref ) , where x ref is a fixed reference point, the meta regret bound becomes O ( √∑ t [( f t ( x t,i ) -f t -1 ( x t,i )) -( f t ( x ref ) -f t -1 ( x ref ))] 2 ) . By the mean value theorem, [( f t ( x t,i ) -f t -1 ( x t,i )) -( f t ( x ref ) -f t -1 ( x ref ))] 2 equals the first term below, which can be further upper bounded by the gradient variation:

This technique brings hope for minimizing the convex functions. To develop a universal method, our first attempt is to utilize MsMwC-Master [Chen et al., 2021] as the meta-algorithm, which satisfies all the three requirements imposed by the first challenge. Nevertheless, the heterogeneous inputs at the meta level present another challenge to this approach. The heterogeneity arises as we pass r t,i = f t ( x t ) -f t ( x t,i ) to the meta-algorithm for base-learner responsible for convex functions, leveraging the conversion technique, while r t,i = 〈∇ f t ( x t ) , x t -x t,i 〉 for base-learners minimizing strongly convex functions. It remains unclear how to adapt MsMwC-Master [Chen et al., 2021] to our heterogeneous inputs, as MsMwC-Master requires an /lscript t as inputs, and the guarantee is for the regret in the form of r t = 〈 p t , /lscript t 〉 -/lscript t . However, such an /lscript t cannot be retrieved from our above design. Fortunately, we observe that the Prod algorithms [Cesa-Bianchi et al., 2007; Gaillard et al., 2014; Wei et al., 2016] are friendly to heterogeneous inputs. Technically, the Prod algorithms provide the same guarantees as long as ∑ i ∈ [ N ] p t,i r t,i ≤ 0 , thanks to the potential-based analysis. Therefore, aside from the requirements of the first challenge, we expect that the meta-algorithm can be analyzed similarly to the Prod algorithms, motivating us to design a new meta-algorithm. As a byproduct, we present a universal algorithm with a two-layer structure under global smoothness with the developed techniques. It is more efficient than that by Yan et al. [2023] and attains the optimal gradient-variation guarantees for convex, strongly convex, and exp-concave functions, at a cost of additional function value queries. We defer algorithms and regret bounds to Appendix C.5.

3.3 A New Lipschitz-Adaptive Meta-Algorithm

In Algorithm 1, we present our meta-algorithm, which builds on optimistic Adapt-ML-Prod [Wei et al., 2016] and incorporates the clipping technique introduced by Cutkosky [2019] and further refined by Chen et al. [2021]. This algorithm, described in the language of Prediction with Experts' Advice (PEA), may be of independent interest beyond adapting to the generalized smoothness. Apart for satisfying all expected requirements, this algorithm offers a simpler design, which does not require a forced restart as opposed to MsMwC-Master [Chen et al., 2021].

This efficiency improvement is achieved through a simple self-confident learning rate in Line 6 of Algorithm 1, unlike that uses fixed learning rates and thus require restarts. In essence, our approach incorporates the clipping mechanism by adding B 2 t to the denominator and removing the threshold on learning rates commonly applied in prior Prod algorithms [Gaillard et al., 2014; Wei et al., 2016]. This term B 2 t acts as a threshold, ensuring that η t,i | ¯ r t,i -m t,i | ≤ 1 / 2 , a critical condition in the analysis (typically satisfied when the Lipschitz constant is provided for prior Prod algorithms). Lipschitz-adaptive algorithms may be sensitive to the choice of B 0 , while in our case, B 0 = Θ(1 / (log T )) is sufficient and does not ruin the guarantees as the factor O (log log) is often treated as a constant [Gaillard et al., 2014; Luo and Schapire, 2015]. Theorem 3 summarizes the guarantee of Algorithm 1 and the proof is provided in Appendix C.3.

Theorem 3. Setting m t,i = 〈 p t , /lscript t -1 〉 -/lscript t -1 ,i in Algorithm 1 ensures that, for any i /star ∈ [ N ] , ∑ T t =1 〈 p t , /lscript t 〉 -∑ T t =1 /lscript t,i /star is bounded as follows, where B T = max { B 0 , max t ∈ [ T ] ‖ r t -m t ‖ ∞ } :

Algorithm 1 Lipschitz Adaptive Optimistic Adapt-ML-Prod

Input: prior information of the scale B 0 , the number of experts N

.

3: Update the weight for i ∈ [ N ] ˜ w t,i = w t,i exp( η t,i m t,i ) ; 4: Calculate decision p t ∈ ∆ N with p t,i = η t,i ˜ w t,i ∑ j ∈ [ N ] η t,j ˜ w t,j and submit it;

1: Initialization: set w 1 ,i = 1 , m 1 ,i = 0 and η 1 ,i = 1 / √ 1 + 4 B 2 0 for all i ∈ [ N ] . 2: for t = 1 to T do

5: Receive r t , update B t = max { B t -1 , ‖ r t -m t ‖ ∞ } , and build ¯ r t,i = m t,i + B t -1 B t ( r t,i -m t,i ) ;

7: Update the weight for i ∈ [ N ] : w t +1 ,i = ( w t,i exp ( η t,i ¯ r t,i -η 2 t,i (¯ r t,i -m t,i ) 2 )) η t +1 ,i η t,i . 8: end for

6: Update the learning rate for i ∈ [ N ] : η t +1 ,i = √ 1 1+ ∑ t s =1 (¯ r s,i -m s,i ) 2 +4 B 2 t ;

Input: curvature coefficient pool H , number of base-learners N , prior information of the scale B 0 .

1: Initialization : Send N and B 0 to the meta-algorithm, for λ ∈ H , initialize an algorithm in Theorem 2 with it; initialize an algorithm in Theorem 1.

2: for t = 1 to T do

3: Obtain p t from meta-algorithm, x t,i from each base-learner i ∈ [ N ] ;

6: For strongly convex functions learners : set r t,i = 〈∇ f t ( x t ) , x t -x t,i 〉 ;

4: Submit x t = ∑ i ∈ [ N ] p t,i x t,i ; 5: Receive f t ( · ) and send it to each base-learner for update;

7: For convex functions learner : set r t,i = f t ( x t ) -f t ( x t,i ) ;

9: end for

8: Send m t +1 ,i = f t ( x t ) -f t ( x t,i ) to the meta-algorithm for i ∈ [ N ] .

Our algorithm improves efficiency at the cost of an additional factor O ( √ log N ) . This factor is ignorable for universal online learning since we set N = O (log T ) , and the factor O (log log T ) is negligible. Considering other related Lipschitz-adaptive algorithms, Mhammedi et al. [2019] obtain a regret bound of O ( √∑ t ( r t,i /star ) 2 · (log( N ) + log log( B T T )) + B T log( N )) , which offers better dependence on the dominant term √∑ t ( r t,i /star ) 2 but it is unclear how to include optimism. Chen et al. [2021] achieve a bound of O ( √∑ t ( /lscript t,i /star -m t,i /star ) 2 · log( NT ) + B T log( NT )) with a twolayer algorithm; however, the O ( √ log T ) term would ruin the desired O (log V T ) bound for strongly convex functions. We remark that the compared methods enjoy other strengths not discussed here, such as the ability to compete with an arbitrary competitor x ∈ ∆ N and the versatility to handle various learning scenarios, while our method is sufficient for our purpose and the only option to tackle all the challenges as we mention in Section 3.2. Lastly, notice that the optimism m t involves the decision p t , which might be improper since p t depends on m t as well. We refer readers to Appendix C.1 for efficiently setting m t through a univariate binary search.

We emphasize that optimism m t in our algorithm is not chosen arbitrarily. In Line 5, we clip the regret with optimism to keep them on the same scale. The performance is then evaluated based on the clipped regret ¯ r t,i . For the analytical purpose, it is essential that 〈 p t , ¯ r t 〉 ≤ 0 , and a sufficient condition for this is ensuring 〈 p t , m t 〉 ≤ 0 , which imposes an additional requirement on m t . In Appendix C.2, we discuss how this requirement introduces challenges for exp-concave functions minimization in universal online learning.

3.4 Overall Algorithm and Regret Guarantees

The function-variation-to-gradient-variation technique decouples the design of universal methods into the base and meta levels, and we are ready to combine the proposed components together. We employ algorithms in Section 2.3 as the base-learners and use Algorithm 1 as the meta-learner, concluding in Algorithm 2. Theorem 4 presents its guarantee with the proof in Appendix C.4.

Theorem 4. Under Assumptions 1 - 2 and assuming a global lower bound such that ¯ f ≤ f t ( x ) for any x ∈ X , t ∈ [ T ] , setting N = /ceilingleft log 2 T /ceilingright + 1 , defining the curvature coefficient pool H =

{ 2 i -1 /T : i ∈ [ N -1] } , and specifying B 0 , Algorithm 2 simultaneously ensures:

where λ ∈ [1 /T, 1] , B T = O (max { B 0 , D max t ∈ [ T ] sup x ‖∇ f t ( x ) - ∇ f t -1 ( x ) ‖ 2 } ) and ̂ G max is the Lipschitz constant on the optimization trajectory.

Without loss of generality, we assume λ ∈ [1 /T, 1] for strongly convex functions. If λ < 1 /T , the optimal O ((log V T ) /λ ) bound would imply linear regret, in which case we would treat them as general convex functions. If λ > 1 , our result is slightly worse than the optimal one by a negligible constant factor. This simplification is also employed by Zhang et al. [2022a]; Yan et al. [2023].

Remark 1. This theorem additionally requires the lower bound of loss functions, which is used to perform the binary search when setting the optimism. We defer the details of the binary search to Appendix C.1. This assumption is also employed recently in parameter-free optimizations [Hazan and Kakade, 2019; Attia and Koren, 2024; Khaled and Jin, 2024], and we can simply choose ¯ f = 0 in empirical risk minimization settings [Hazan and Kakade, 2019].

4 Applications

In this section, we demonstrate the importance of our results by providing two applications (SEA model and online games), where new results can be directly implied from our findings.

4.1 Stochastically Extended Adversarial (SEA) Model

The stochastically extended adversarial (SEA) model [Sachs et al., 2022] interpolates adversarial and stochastic online optimization. It assumes that the environments select the loss function f t ( · ) from a distribution D t . The adversarial nature is characterized by shifts in distribution D t , and when D t remains constant, the model captures the environments' stochastic behavior. The following quantities are introduced to measure the levels of adversarial and stochastic behaviors in environments:

where we denote by F t ( x ) = E f t ∼ D t [ f t ( x )] . In above, Σ 2 1: T represents the adversarial shift of the distribution, and σ 2 1: T denotes the stochastic variance.

Sachs et al. [2022] prove an O ( √ σ 2 1: T + √ Σ 2 1: T ) regret for convex functions, and a refined regret bound of O (( σ 2 max +Σ 2 max ) log( σ 2 1: T +Σ 2 1: T )) for strongly convex functions is obtained by Chen et al. [2023a]; Sachs et al. [2023], where σ 2 max = max t ∈ [ T ] sup x ∈X E [ ‖∇ f t ( x ) -∇ F t ( x ) ‖ 2 2 ] and Σ 2 max = max T t =1 sup x ∈X ‖∇ F t ( x ) - ∇ F t -1 ( x ) ‖ 2 2 . Yan et al. [2023] present a universal method which can obtain ˜ O ( √ σ 2 1: T + √ Σ 2 1: T ) and O (( σ 2 max +Σ 2 max ) log( σ 2 1: T +Σ 2 1: T )) bounds for convex and strongly convex functions. However, these results require the global smoothness assumption.

Our result in Section 3 implies a new finding for the SEA model, relaxing the assumption from the global smoothness to the generalized smoothness, while adapting to unknown curvature, summarized in Corollary 1. The proof can be found in Appendix D.1.

Corollary 1. Under Assumptions 1 - 2 and assuming a global lower bound for the loss functions such that ¯ f ≤ f t ( x ) for any x ∈ X , t ∈ [ T ] , setting N = /ceilingleft log 2 T /ceilingright + 1 , defining the curvature coefficient pool H = { 2 i -1 /T : i ∈ [ N -1] } , and specifying B 0 with a specific value, then, under the SEA model, Algorithm 2 simultaneously ensures:

˜ ˜ where ̂ G max is the maximum empirical Lipschitz constant, ˜ σ 2 1: T = ∑ T t =1 E [sup x ∈X ‖∇ f t ( x ) -∇ F t ( x ) ‖ 2 2 ] , and ˜ σ 2 max = E [max t ∈ [ T ] sup x ∈X ‖∇ f t ( x ) -∇ F t ( x ) ‖ 2 2 ] .

Note that, in real-world streaming learning applications, this corollary can offer a more generalized depiction of data throughput with limited computing resources [Zhou, 2024; Wang et al., 2024], given the connection between these scenarios and the SEA model [Chen et al., 2024, § 5.6]. Our result depends on ˜ σ 2 1: T , a larger quantity than σ 2 1: T but still can track the stochastic variance. This is because our algorithm utilizes the information afterward x t -1 to generate x t . Werefer the interested readers for this subtle issue to the discussion by Chen et al. [2024]. This dependence currently is unknown how to improve even under the global smoothness condition since we need to leverage the function variation to produce gradient variation, inevitably involving the afterward information.

4.2 Fast Rates in Games

Our second application explores the min-max game, aiming to achieve an ε -approximate solution to the problem min x ∈X max y ∈Y f ( x , y ) within an O (1 /T ) fast convergence rate. Here, we assume that f ( · , y ) is convex for any y ∈ Y , and correspondingly, f ( x , · ) is concave given any x ∈ X . Additionally, we assume that both X ⊂ R n and Y ⊂ R m are bounded convex sets. Pioneering research [Syrgkanis et al., 2015] demonstrates that optimistic algorithms can reach a convergence rate of O (1 /T ) by leveraging gradient variation. However, these results are limited to the global smoothness condition. In this part, we demonstrate that our findings in Section 2 directly imply a new algorithm that can exploit generalized smoothness.

Following the notations of Nemirovski [2004], we define Z = X×Y , z = ( x , y ) ∈ Z and introduce an operator F : Z → R n × R m with F ( z ) = ( ∇ x f ( x , y ) , -∇ y f ( x , y )) . We extend the concept of /lscript -smoothness to the min-max optimization setting as follows.

Definition 2 ( /lscript -smoothness for min-max game) . A differentiable convex-concave function f : X × Y ↦→ R is called /lscript -smooth with a non-decreasing link function /lscript : [0 , + ∞ ) ↦→ (0 , + ∞ ) if it satisfies: for any z 1 , z 2 ∈ Z , if z 1 , z 2 ∈ B ( z , ‖ F ( z ) ‖ 2 /lscript (2 ‖∇ F ( z ) ‖ 2 ) ) , then ‖ F ( z 1 ) -F ( z 2 ) ‖ 2 ≤ /lscript (2 ‖ F ( z ) ‖ ) · ‖ z 1 -z 2 ‖ 2 , where B ( z , r ) denotes the Euclidean ball centered at point z with radius r .

This definition is a counterpart to Definition 1 in the min-max game, but weaker as it does not require the twice-differentiability requirement. The ε -approximate solution ( x /star , y /star ) to the min-max game is formally defined by f ( x /star , y ) -ε ≤ f ( x /star , y /star ) ≤ f ( x , y /star ) + ε for any x ∈ X , y ∈ Y . To achieve the fast convergence rate to the solution, indeed our result in Section 2 can be directly applied to the min-max game and obtains the following tailored algorithm for min-max optimization:

z t = Π Z [ ̂ z t -η t F ( ̂ z t )] , ̂ z t +1 = Π Z [ ̂ z t -η t F ( z t )] . (10) In above we set the optimism at the point ̂ z t in order to exploit the smoothness locally. We conclude our result in Corollary 2, with the proof available in Appendix D.2.

Corollary 2. Assume that the convex-concave function f ( x , y ) is /lscript -smooth, and the domain Z is bounded with diameter D . By applying the tuning strategy described in Theorem 1, and defining the final approximated solution as ¯ z T = 1 T ∑ T t =1 z t , where z t is generated by Eq. (10) , we achieve an ε -approximate solution with a convergence rate of O (1 /T ) .

5 Conclusion

In this paper, we provide a systematic study of gradient-variation online learning under the generalized smoothness condition. We exploit trajectory-wise smoothness to achieve the optimal regret bounds: O ( √ V T ) for convex functions and O (log V T ) for strongly convex functions, respectively. We further consider more complicated online learning scenarios, motivating us to design a new Lipschitz-adaptive meta-algorithm, which can be of independent interest. Hinging on this algorithm with the function-variation-to-gradient-variation technique, we design a universal algorithm which guarantees the optimal results for convex functions and strongly convex functions simultaneously without knowing the curvature. In addition, our findings directly imply new results in stochastic extended adversarial online learning and fast-rate games under generalized smoothness.

An important future direction for future research is to explore whether our method can be further extended to accommodate the one-gradient feedback model, where the learner receives only the gradient information of the decision submitted in each round. Another interesting problem is to exploit the exp-concavity in gradient-variation online learning under generalized smoothness.

References

D. Abernethy, P. L. Bartlett, A. Rakhlin, and A. Tewari. Optimal stragies and minimax lower bounds for online convex games. In Proceedings of the 21st Annual Conference on Learning Theory (COLT) , pages 415-424, 2008.

Ataee Tarzanagh, P. Nazari, B. Hou, L. Shen, and L. Balzano. Online bilevel optimization: Regret analysis of online alternating gradient methods. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 2854-2862, 2024.

Attia and T. Koren. How free is parameter-free stochastic optimization? In Proceedings of the 41st International Conference on Machine Learning (ICML) , pages 2009-2034, 2024.

Auer, N. Cesa-Bianchi, and C. Gentile. Adaptive and self-confident on-line learning algorithms. Journal of Computer and System Sciences , 64(1):48-75, 2002.

Bai, Y.-J. Zhang, P. Zhao, M. Sugiyama, and Z.-H. Zhou. Adapting to online label shift with provable guarantees. In Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 29960-29974, 2022.

Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operation Research Letter , 31(3):167-175, 2003.

Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

Cesa-Bianchi, Y. Mansour, and G. Stoltz. Improved second-order bounds for prediction with expert advice. Machine Learning , 66(2-3):321-352, 2007.

Chen, H. Luo, and C. Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Proceedings of the 34th Conference on Learning Theory (COLT) , pages 12161259, 2021.

Chen, W.-W. Tu, P. Zhao, and L. Zhang. Optimistic online mirror descent for bridging stochastic and adversarial online convex optimization. In Proceedings of the 40th International Conference on Machine Learning (ICML) , pages 5002-5035, 2023a.

Chen, Y.-J. Zhang, W.-W. Tu, P. Zhao, and L. Zhang. Optimistic online mirror descent for bridging stochastic and adversarial online convex optimization. Journal of Machine Learning Research , 25 (178):1 - 62, 2024.

Chen, Y. Zhou, Y. Liang, and Z. Lu. Generalized-smooth nonconvex optimization is as efficient as smooth nonconvex optimization. In Proceedings of the 40th International Conference on Machine Learning (ICML) , pages 5396-5427, 2023b.

C.-K. Chiang, T. Yang, C.-J. Lee, M. Mahdavi, C.-J. Lu, R. Jin, and S. Zhu. Online optimization with gradual variations. In Proceedings of the 25th Conference On Learning Theory (COLT) , pages 6.1-6.20, 2012.

Crawshaw, M. Liu, F. Orabona, W. Zhang, and Z. Zhuang. Robustness to unbounded smoothness of generalized signsgd. In Advances in Neural Information Processing Systems 34 (NeurIPS) , volume 35, pages 9955-9968, 2022.

Cutkosky. Artificial constraints and hints for unbounded online learning. In Proceedings of the 32nd Annual Conference on Computational Learning Theory (COLT) , pages 874-894, 2019.

Cutkosky and K. Boahen. Stochastic and adversarial online learning without hyperparameters. In Advances in Neural Information Processing Systems 30 (NIPS) , pages 5059-5067, 2017.

Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12(7), 2011.

Faw, L. Rout, C. Caramanis, and S. Shakkottai. Beyond uniform smoothness: A stopped analysis of adaptive SGD. In Proceedings of the 36th Conference on Learning Theory (COLT) , pages 89160, 2023.

Gaillard, G. Stoltz, and T. van Erven. A second-order bound with excess losses. In Proceedings of the 27th Annual Conference on Learning Theory (COLT) , pages 176-196, 2014.

Hazan. Introduction to Online Convex Optimization. Foundations and Trends in Optimization , 2 (3-4):157-325, 2016.

Hazan and S. Kakade. Revisiting the polyak step size. ArXiv preprint , arXiv:1905.00313, 2019.

Hazan, A. Agarwal, and S. Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2-3):169-192, 2007.

Jacobsen and A. Cutkosky. Parameter-free mirror descent. In Proceedings of the 35th Annual Conference on Learning Theory (COLT) , pages 4160-4211, 2022.

Jacobsen and A. Cutkosky. Unconstrained online learning with unbounded losses. In Proceedings of the 40th International Conference on Machine Learning (ICML) , pages 14590-14630, 2023.

Joulani, A. György, and C. Szepesvári. A modular analysis of adaptive (non-)convex optimization: Optimism, composite objectives, variance reduction, and variational bounds. Theoretical Computer Science , 808:108-138, 2020.

Khaled and C. Jin. Tuning-free stochastic optimization. In Proceedings of the 41st International Conference on Machine Learning (ICML) , pages 23622-23661, 2024.

Kulis and P. L. Bartlett. Implicit online learning. In Proceedings of the 27th International Conference on Machine Learning (ICML) , pages 575-582, 2010.

Li, J. Qian, Y. Tian, A. Rakhlin, and A. Jadbabaie. Convex and non-convex optimization under generalized smoothness. In Advances in Neural Information Processing Systems 35 (NeurIPS) , 2023.

Lu, R. M. Freund, and Y. Nesterov. Relatively smooth convex optimization by first-order methods, and applications. SIAM Journal on Optimization , 28(1):333-354, 2018.

Luo and R. E. Schapire. Achieving all with no parameters: AdaNormalHedge. In Proceedings of the 28th Annual Conference on Computational Learning Theory (COLT) , pages 1286-1304, 2015.

Mhammedi, W. M. Koolen, and T. van Erven. Lipschitz adaptivity with multiple learning rates in online learning. In Proceedings of the 32nd Annual Conference on Computational Learning Theory (COLT) , pages 2490-2511, 2019.

Nemirovski. Prox-method with rate of convergence O (1 /t ) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1):229-251, 2004.

Nemirovskij and D. B. Yudin. Problem complexity and method efficiency in optimization. SIAM Review , 27(2):264-265, 1985.

Nesterov. Lectures on Convex Optimization , volume 137. Springer, 2018.

Nicolò and O. Francesco. Temporal variability in implicit online learning. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 12377-12387, 2020.

Orabona. A Modern Introduction to Online Learning. ArXiv preprint , arXiv:1912.13213, 2019.

Orabona and D. Pál. Scale-free online learning. Theoretical Computer Science , 716:50-69, 2018.

Rakhlin and K. Sridharan. Online learning with predictable sequences. In Proceedings of the 26th Conference On Learning Theory (COLT) , pages 993-1019, 2013a.

Rakhlin and K. Sridharan. Optimization, learning, and games with predictable sequences. In Advances in Neural Information Processing Systems 26 (NIPS) , pages 3066-3074, 2013b.

Reisizadeh, H. Li, S. Das, and A. Jadbabaie. Variance-reduced clipping for non-convex optimization. ArXiv preprint , arXiv:2303.00883, 2023.

Sachs, H. Hadiji, T. van Erven, and C. Guzmán. Between stochastic and adversarial online convex optimization: Improved regret bounds via smoothness. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 691-702, 2022.

Sachs, H. Hadiji, T. van Erven, and C. Guzmán. Accelerated rates between stochastic and adversarial online convex optimization. ArXiv preprint , arXiv:2303.03272, 2023.

Syrgkanis, A. Agarwal, H. Luo, and R. E. Schapire. Fast convergence of regularized learning in games. In Advances in Neural Information Processing Systems 28 (NIPS) , pages 2989-2997, 2015.

Telgarsky. Stochastic linear optimization never overfits with quadratically-bounded losses on general data. In Proceedings of the 35th Annual Conference on Learning Theory (COLT) , pages 5453-5488, 2022.

Tsai, Y. Lin, and Y . Li. Data-dependent bounds for online portfolio selection without lipschitzness and smoothness. In Advances in Neural Information Processing Systems 36 (NeurIPS) , pages 62764-62791, 2023.

van Erven and W. M. Koolen. MetaGrad: Multiple learning rates in online learning. In Advances in Neural Information Processing Systems 29 (NIPS) , pages 3666-3674, 2016.

Wang, H. Zhang, Z. Ma, and W. Chen. Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions. In Proceedings of the 36th Annual Conference on Learning Theory (COLT) , pages 161-190, 2023.

Wang, S. Lu, and L. Zhang. Adaptivity and optimality: A universal algorithm for online convex optimization. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence (UAI) , pages 659-668, 2019.

Wang, M. Yu, P. Zhao, and Z.-H. Zhou. Learning with adaptive resource allocation. In Proceedings of the 41st International Conference on Machine Learning (ICML) , pages 52099-52116, 2024.

C.-Y. Wei, Y.-T. Hong, and C.-J. Lu. Tracking the best expert in non-stationary stochastic environments. In Advances in Neural Information Processing Systems 29 (NIPS) , pages 3972-3980, 2016.

Y.-H. Yan, P. Zhao, and Z.-H. Zhou. Universal online learning with gradient variations: A multilayer online ensemble approach. In Advances in Neural Information Processing Systems 36 (NeurIPS) , pages 37682-37715, 2023.

Yang, M. Mahdavi, R. Jin, and S. Zhu. Regret bounded by gradual variation for online convex optimization. Machine Learning , 95(2):183-223, 2014.

Yang, Y. Wang, P. Zhao, and L. Zhang. Universal online convex optimization with 1 projection per round. ArXiv preprint , arXiv:2405.19705, 2024.

Zhang, J. Jin, C. Fang, and L. Wang. Improved analysis of clipping algorithms for non-convex optimization. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 1551115521, 2020a.

Zhang, T. He, S. Sra, and A. Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In Proceedings of the 8th International Conference on Learning Representations (ICLR) , 2020b.

Zhang, G. Wang, J. Yi, and T. Yang. A simple yet universal strategy for online convex optimization. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 26605-26623, 2022a.

Zhang, P. Zhao, H. Luo, and Z.-H. Zhou. No-regret learning in time-varying zero-sum games. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 2677226808, 2022b.

Zhao, Y.-J. Zhang, L. Zhang, and Z.-H. Zhou. Dynamic regret of convex and smooth functions. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 12510-12520, 2020.

Zhao, Y.-J. Zhang, L. Zhang, and Z.-H. Zhou. Adaptivity and non-stationarity: Problemdependent dynamic regret for online convex optimization. Journal of Machine Learning Research , 25(98):1 - 52, 2024.

Z.-H. Zhou. Learnability with time-sharing computational resource concerns. National Science Review , 11(10):nwae204, 2024.

Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML) , pages 928-936, 2003.

A Related Work

In this section, we review the related work in gradient-variation online learning, the generalized smoothness conditions, and the Lipschitz-adaptive algorithms.

A.1 Gradient-Variation Online Learning

The gradient-variation quantity defined in Eq. (2) is firstly introduced by Chiang et al. [2012] for global smooth functions. Chiang et al. [2012] obtain O ( √ V T ) and O ( d α log V T ) regret bounds respectively for general convex and α -exp-concave functions. Zhang et al. [2022a] later achieve O ( 1 λ log V T ) result for λ -strongly convex functions. Considering the non-stationary environments, Zhao et al. [2020] study gradient variation under the dynamic regret, a strengthened measure that requires the learner to compete with a sequence of time-varying comparators. Their work revives the study of gradient-variation online learning, especially revealing the importance of the stability analysis in the two-layer online ensemble. Their results are further improved in Zhao et al. [2024], where they only require one gradient per round with the same optimal guarantees through the collaboration between the meta and the base. For universal online learning [van Erven and Koolen, 2016], there are several recent researches trying to derive the gradient-variation bounds in this context [Zhang et al., 2022a; Sachs et al., 2023; Yan et al., 2023].

Gradient variation demonstrates its importance due to its close connection with many online learning problems. Pioneering works [Rakhlin and Sridharan, 2013a; Syrgkanis et al., 2015; Zhang et al., 2022b] demonstrate the importance of gradient variation via a useful property (Regret bounded by Variation in Utilities, RVU) to achieve fast-rate convergences in multi-player games. Recently, Sachs et al. [2022] and Chen et al. [2024] reveal that the gradient variation is also essential in bridging stochastic and adversarial convex optimization.

However, all the mentioned works require the global smoothness assumption. As highlighted in Proposition 2 in Appendix B, the smoothness condition is necessary to obtain the gradient-variation bounds for algorithms with first-order oracle assumption [Nesterov, 2018]. Exceptions are that Jacobsen and Cutkosky [2022]; Bai et al. [2022] obtain the gradient-variation bounds without the smoothness assumption, but they require implicit updates [Kulis and Bartlett, 2010; Nicolò and Francesco, 2020] per round, which may be inefficient under online settings. Our work considers generalized smoothness, and we develop first-order methods to achieve gradient-variation bounds.

A.2 Generalized Smoothness

Generalized smoothness has received increasing attention in recent years since the analysis under the standard global smoothness is insufficient to depict the dynamics of neural network optimization. Based on the empirical observation for the relationship between the smoothness and the gradients of LSTMs, Zhang et al. [2020b] relax the global smoothness assumption by ( L 0 , L 1 ) -smoothness condition, which assumes ‖∇ 2 f ( x ) ‖ 2 ≤ L 0 + L 1 ‖∇ f ( x ) ‖ 2 for offline objective function f : X ↦→ R . With this new smoothness assumption, Zhang et al. [2020b] explain the importance of gradient clipping in neural network training. There are a variety of subsequent works developed for different methods and settings [Zhang et al., 2020a; Crawshaw et al., 2022; Reisizadeh et al., 2023; Faw et al., 2023; Wang et al., 2023]. There are studies further generalizing the ( L 0 , L 1 ) -smoothness condition. Chen et al. [2023b] introduce the α -symmetric smoothness condition, which symmetrizes the ( L 0 , L 1 ) -smoothness condition and allows the smoothness to depend polynomially on the gradient. Notably, Li et al. [2023] propose the /lscript -smoothness, defined as ‖∇ 2 f ( x ) ‖ 2 ≤ /lscript ( ‖∇ f ( x ) ‖ 2 ) . This condition does not specify a particular form for the function /lscript : R → R , other than some mild assumptions about its properties, thereby allowing great generality of this notion.

Telgarsky [2022] introduces the concept of ( G,L ) -quadratically bounded functions, which aims to generalize the Lipschitz condition as ‖∇ f ( x ) ‖ 2 ≤ G + L ‖ x -x 0 ‖ 2 with a reference point x 0 ∈ X . Though this notion covers the standard global smoothness condition, it lacks a detailed depiction of the relationship between the smoothness and the gradients, hindering further research into understanding the optimization dynamics such as the role of gradient clipping [Zhang et al., 2020b]. Jacobsen and Cutkosky [2023] investigate online convex optimization under this constraint such that the online functions may be unbounded. However, it is unclear how to obtain the gradientvariation regret in their context and using their methods.

We also mention another generalization of the standard smoothness called the relative smoothness [Lu et al., 2018]. A function f : X ↦→ R is L -smooth relative function if f ( x ) ≤ f ( y ) + 〈∇ f ( y ) , x -y 〉 + L D h ( x , y ) for any x , y ∈ int X , where h : X ↦→ R is a reference function with D h ( x , y ) denoting the Bregman divergence. However, the global constant L is still required to tune the algorithm, and studying this notion is beyond the scope of our paper.

A.3 Lipschitz-Adaptive Algorithms

The upper bound of gradients G and the diameter of the bounded feasible domain D are often required to build up online algorithms. An algorithm that requires the diameter D of the bounded feasible domain D but not the upper bound of gradients is known to be Lipschitz-adaptive [Duchi et al., 2011; Orabona and Pál, 2018; Cutkosky, 2019; Mhammedi et al., 2019]. For the general OCO setting, to the best of our knowledge, there are no Lipschitz-adaptive algorithm that ensures a gradient-variation bound. When specialized to the Prediction with Experts' Advice (PEA) setting [Cesa-Bianchi and Lugosi, 2006], Chen et al. [2021] design a two-layer algorithm with a restarting mechanism that can obtain a gradient-variation bound in PEA setting. In this paper, we develop a new Lipschitz-adaptive algorithm with a lightweight design, which guarantees the gradient-variation bound as well and is the only option for our purposes, to the best of our knowledge.

B Omitted Details for Section 2

In this section, we first provide a judgement of the necessity of smoothness for first-order online algorithms in Appendix B.1. Next, we will provide proofs for the theorems in Section 2.3. In Appendix B.2, we present the omitted discussion for the challenge in designing the algorithm for exp-concave functions. We introduce a key lemma in Appendix B.3, which abstracts the key idea of exploiting the generalized smoothness. Later, the proofs for Theorem 1 and Theorem 2 are provided in Appendix B.4 and Appendix B.5, respectively.

B.1 Necessity of Smoothness

Wefirst provide a lower bound on the convergence rate for first-order methods with convex functions.

Proposition 1 (Theorem 3.2.1 of Nesterov [2018]) . There exists a G -Lipschitz and convex function f : R d ↦→ R with ‖ x 1 -x /star ‖ ≤ D where x /star ∈ arg min x ∈ R d f ( x ) such that

for any optimization scheme that generates a sequence { x t } satisfying that

where Lin { a 1 , . . . , a t } denotes the linear span of vectors a 1 , . . . , a t .

Yang et al. [2014] prove the necessity of the smoothness to obtain the gradient-variation bounds with convex functions using the first-order online algorithms. We formally present this idea in below.

Proposition 2 (Remark 1 of Yang et al. [2014]) . The smoothness assumption for G -Lipschitz and convex online functions f t : X ↦→ R is necessary for any online algorithms, whose decisions are linear combinations of the queried gradients when no projections are made, and ensure:

with only c · T times gradient queries, with c ∈ N being a constant independent of T and environmental parameters, such as the Lipschitz constant and the smoothness constant.

Proof. We prove this proposition by contradiction. Assume that f t is non-smooth. Then consider a special case where the algorithm projects the decisions onto X = R d , i.e., no projections are made,

and all the online functions are equal f 1 = · · · = f T = f . Notice that, under such case, the gradient variation is zero and therefore,

which implies ¯ x T = 1 T ∑ T t =1 x t approaches an O (1 /T ) convergence rate. Now it remains to check whether this convergence rate contradicts with Proposition 1. Denoted by x ′ 1 , . . . , x ′ cT the points that the algorithm queries gradients on, because the decisions are linear combinations of the gradients when no projections are made, then,

which indicates that, there exists a method which promises O ( c/T ) = O (1 /T ) convergence rate under the protocol considered in Proposition 1, contradicting the lower bound.

B.2 Challenge for Exp-Concave Functions in Regret Minimization

In Section 2, we design algorithms for convex and strongly convex functions respectively. However, the gradient-variation regret for exp-concave functions [Hazan et al., 2007] under generalized smoothness has not yet been addressed. For online learning with global smooth functions, it has been demonstrated that an O ( d log V T ) regret is attainable for exp-concave functions [Chiang et al., 2012], which is realized by an optimistic variant of the online Newton step algorithm [Hazan et al., 2007] using the last-round gradient as optimism, i.e., M t = ∇ f t -1 ( x t -1 ) . However, it remains unclear how to obtain a general optimistic bound of order O ( d log D T ) with D T = ∑ T t =1 ‖∇ f t ( x t ) -M t ‖ 2 2 for arbitrary optimism { M t } T t =1 . Our technique for achieving gradient-variation regret under generalized smoothness relies on a flexible setting for optimism (which may not be the last-round gradient) and step size tuning (which requires a trajectory-wise stability analysis), making it challenging for extension to exp-concave functions. We leave this as an open question for future research.

B.3 Lemma for Regret Minimization

Below, we present a lemma that leverages the local smoothness of the optimization trajectory to derive gradient variation within the OMD framework. This lemma can be applied in the analysis of both convex and strongly convex functions.

Lemma 2. Under Assumptions 1 and 2, by selecting regularizer as ψ t ( x ) = 1 2 η t ‖ x ‖ 2 2 , setting step sizes satisfying that η t +1 ≤ η t and η t ≤ 1 / (4 ̂ L t -1 ) , where ̂ L t -1 = /lscript t -1 (2 ‖∇ f t -1 ( ̂ x t ) ‖ 2 ) , and by choosing optimism as M t = ∇ f t -1 ( x t ) , the OMD in Eq. (4) ensures the regret bound:

where x /star ∈ arg min x ∈X ∑ T t =1 f t ( x ) denotes the best decision in hindsight.

Proof. Following the standard analysis of OMD (Lemma 3), OMD with the optimism chosen as M t = ∇ f t -1 ( x t ) ensures the following regret bound:

The analysis for TERM-A is straightforward under Assumption 2:

Next, we analyze the TERM-B under the generalized smoothness condition. By the basic calculation, we can decompose the TERM-B into following two terms:

where the first term is the desirable gradient variation and the second term should be further analyzed under the generalized smoothness. Given the optimism setting and the step size tuning, we demonstrate that x t and x t are sufficiently close to apply local smoothness:

̂ In above, the first inequality is by the the Pythagorean theorem and the second inequality is by the step size tuning. Therefore, we can apply Lemma 1 to bound the gradient deviation in Eq. (12) by

Finally, combining TERM-A, TERM-B and TERM-C together, we obtain:

where the second inequality is by setting η t ≤ 1 / (4 ̂ L t -1 ) . Hence, we finish the proof. B.4 Proof of Theorem 1

Proof. The step size tuning in Eq. (7) in Theorem 1 satisfies the criterions for applying Lemma 2, specifically that η t +1 ≤ η t and η t ≤ 1 / (4 ̂ Lt -1) , where ̂ L t -1 = /lscript t -1 (2 ‖∇ f t -1 ( ̂ x t ) ‖ 2 ) . Therefore, by Lemma 2 and the convexity of loss functions, we obtain:

where x /star ∈ arg min x ∈X ∑ T t =1 f t ( x ) . The first term can be bounded as follows,

where we denote by ̂ L max = max t ∈ [ T ] ̂ L t . For the second term, we apply the self-confident tuning lemma (Lemma 5) by choosing f ( x ) = 1 / √ x in the lemma statement:

where we use ̂ G max to denote the empirically maximum Lipschitz constant. The additional term O ( ̂ G 2 max D ) results from the lack of knowledge about ̂ G max . However, we can improve this term to O ( ̂ G max D ) by incorporating the clipping technique [Cutkosky, 2019; Chen et al., 2021] into OMD framework. In the main text, we avoid introducing this method to prevent complicating the approach further. Our goal in the OMD introduction is to illustrate how to adapt to generalized smoothness at the base level. The details of this refined approach are provided in Appendix B.6.

B.5 Proof of Theorem 2

Proof. For λ -strongly convex functions, we tune the step size as η t = 2 / ( λt +16max s ∈ [ t -1] ̂ L s ) , where we denote by ̂ L t = /lscript t (2 ‖∇ f t ( ̂ x t +1 ) ‖ ) the locally estimated smoothness constant. By the property of strongly convex functions and Eq. (11) in Lemma 2, we have:

Unlike in the case of convex functions, TERM-A involves negative terms derived from the strong convexity of the loss functions, which require a slightly different analysis:

where the last term can be cancelled by TERM-C, and the third term is in the same order of TERM-B. Therefore, we only need to focus on TERM-B. For TERM-B, we follow the analysis of Chen et al. [2024], who applies a simpler analysis for the self-confident tuning. First, we define:

Then by dividing the time horizon into [1 , α ] and [ α +1 , T ] , we can upper bound TERM-B as:

where we use ̂ G max to denote the maximum Lipschitz constant estimated empirically. Therefore:

which finishes the proof.

B.6 OMDIncorporating Clipping Technique

In Appendix B.4, an additional term O ( ̂ G 2 max D ) shows up in the final regret bound, which results from the lack of knowledge about ̂ G max . In this subsection, we improve the term O ( ̂ G 2 max D ) to O ( ̂ G max D ) by incorporating the clipping technique [Cutkosky, 2019; Chen et al., 2021] into the OMDframework. This modified OMD algorithm is defined as follows,

x t = Π X [ ̂ x t -η t ∇ f t -1 ( ̂ x t )] , ̂ x t +1 = Π X [ ̂ x t -η t ˜ g t ] , (14) where ˜ g t = ∇ f t -1 ( ̂ x t ) + B t -1 B t ( ∇ f t ( x t ) - ∇ f t -1 ( ̂ x t )) is a clipped gradient with the maintained threshold B t = max { B 0 , max s ∈ [ t ] ‖∇ f s ( x s ) - ∇ f s -1 ( ̂ x s ) ‖ 2 } . Notice that, x t still updates from ̂ x t in the same manner as illustrated in Theorem 1, therefore, we can apply the similar analysis to it when exploiting the smoothness locally. Correspondingly, we provided the following step size tuning:

˜ ̂ where the key modification is that we add B 2 t -1 in the denominator to facilitate the tuning analysis and we denote by ̂ L t = /lscript t (2 ‖∇ f t ( ̂ x t +1 ) ‖ ) . The following theorem presents the guarantee.

Theorem 5. Under Assumptions 1 - 2, assuming online functions are convex, OMD presented in Eq. (14) with the step sizes in Eq. (15) ensures that:

where V T = ∑ T t =2 sup x ∈X ‖∇ f t ( x ) -∇ f t -1 ( x ) ‖ 2 2 is the gradient variations, ̂ L max = max t ∈ [ T ] ̂ L t is the maximum smoothness constant over the optimization trajectory, and ̂ G max is the maximum empirically estimated Lipschitz constant.

Proof. First, we prove that the clipping technique incurs a constant in the regret bound:

In the following analysis, we will focus on the regret associated with the clipped gradient ˜ g t . Following the standard analysis of OMD (Lemma 3), and with simple calculations, we arrive at:

TERM-A-VAR + TERM-B-VAR -TERM-C

where in the forth line we apply Lemma 1, and ub the final line, we apply the AM-GM inequality, 2 √ ab -a ≤ b . Combing each component together, we conclude the proof as:

C Omitted Details for Section 3

In this section, we present the omitted details for Section 3. Appendix C.1 discusses how set the optimism m t efficiently via a binary search. Appendix C.2 provides the omitted discussion for the challenge in universal online learning with exp-concave functions. Proofs for the theorems in Section 3 are included in Appendix C.3 and Appendix C.4. Appendix C.5 gives a simple universal algorithm under the global smoothness condition, which improves the optimality and efficiency of the method by Yan et al. [2023], at a cost of additional function value queries.

C.1 Discussion on Optimism

In this part, we discuss how to set the optimism m t that involves the decision p t . We present the steps for incorporating the optimism and generating the decision p t in Algorithm 1 for reference:

Below, we prove the existence of a solution to g ( α ) = α . Provided the lower bound ¯ f of function f ( · ) , and by the convexity of the function f ( · ) , the searching range of α is restricted to ¯ f ≤ α ≤ max i ∈ [ N ] { f ( x i ) } , and thus α is bounded. The continuity of the function f ( · ) implies the continuity of the function g ( α ) as well. The choice of α = ¯ f results in g ( α ) -α = f ( ∑ i ∈ [ N ] p t,i ( α ) x i ) -¯ f ≥ 0 , and the choice of α = max i ∈ [ N ] { f ( x i ) } implies g ( α ) -α ≤ ∑ i ∈ [ N ] p t,i ( α ) f ( x i ) -max i ∈ [ N ] { f ( x i ) } ≤ 0 , indicating that a solution to g ( α ) = α exists. By using a binary search within [ ¯ f, max i ∈ [ N ] { f ( x i ) } ] , we can approach α within an error O (1 /T ) in O (log T ) iterations.

˜ For more general consideration, we set optimism the as m t,i = f ( ∑ i ∈ [ N ] p t,i x t,i ) -h ( x t,i ) where f : X ↦→ R is a convex and continuous function and x i ∈ X is a decision available before setting the optimism. Notice that ˜ w i requires p t to update while p t is produced based on ˜ w t,i , resulting in a circular argument. Following Wei et al. [2016], the p t can be solved via the binary search technique. We define α = f ( ∑ i ∈ [ N ] p t,i x i ) , then the weight ˜ w t,i is a function of α as ˜ w t,i ( α ) = w t,i exp( η t,i ( α -f ( x t,i ))) . Furthermore, with the update formulation in (19), the decision p t,i is a function of α as well, with the formulation p t,i ( α ) = η t,i ˜ w t,i ( α ) ∑ j ∈ [ N ] η t,j ˜ w t,j ( α ) . By introducing a function g ( α ) = f ( ∑ i ∈ [ N ] p t,i ( α ) x i ) , solving the decision p t is equal to solving g ( α ) = α .

The above argument requires the lower bound of the function f ( · ) to determine the searching range over α . When considering a simpler case where f ( x i ) = /lscript i and f ( ∑ i ∈ [ N ] p t,i x i ) = ∑ i ∈ [ N ] p t,i /lscript i , we can omit the requirement for the lower bound because α falls within [min i ∈ [ N ] { /lscript i } , max i ∈ [ N ] { /lscript i } ] , because of the simpler structure of linear functions.

C.2 Challenge for Exp-Concave Functions in Universal Online Learning

In Section 3.4, we present Algorithm 2 that achieves gradient-variation bounds for convex and strongly convex functions simultaneously. However, it does not guarantee such bounds for exp-concave functions. Besides the challenges discussed in Section B.2, a new obstacle arises in designing the meta-algorithm. We require optimism to satisfy 〈 p t , m t 〉 ≤ 0 , since we pass the meta-algorithm with heterogeneous inputs, and therefore we set m t,i = f t -1 ( x t ) -f t -1 ( x t,i ) for all the base-learners. This optimism design is suitable for strongly convex functions, as the term √∑ t ( f t -1 ( x t ) -f t -1 ( x t,i )) 2 introduced by optimism can be bounded by ̂ G max √∑ t ‖ x t -x t,i ‖ 2 2 , cancelled by the negative term -∑ t λ ‖ x t -x t,i ‖ 2 2 from strong convexity. However, for exp-concave functions, the negative term -〈∇ f t ( x t ) , x t -x t,i 〉 2 from exp-concavity may not be sufficient to cancel √∑ t ( f t -1 ( x t ) -f t -1 ( x t,i )) 2 . We leave this as an open problem for future exploration.

C.3 Proof of Theorem 3

In this subsection, we prove a slightly generalized version of Theorem 3, which does not specify optimism and instead imposes conditions only on ¯ r t . One can verify that the setting of optimism in Theorem 3 satisfies the requirement of Theorem 6.

Theorem 6. By setting ¯ r t,i such that ∑ N i =1 p t,i ¯ r t,i ≤ 0 , Algorithm 1 ensures that, for any i /star ∈ [ N ] , the regret ∑ T t =1 〈 p t , /lscript t 〉 -∑ T t =1 /lscript t,i /star can be bounded by:

where B T = max { B 0 , max t ∈ [ T ] ‖ r t -m t ‖ ∞ } .

Proof. First, we demonstrate that the clipping technique [Chen et al., 2021; Cutkosky, 2019] incurs a constant in the regret bound:

In below, we focus on the analysis associated with clipped regret ¯ r t,i /star . Following previous work [Wei et al., 2016], we define W t = ∑ N i =1 w t,i to represent the summation of weights at time t . The quantity W t can be realized as the potential to be analyzed. Next, we consider to upper bound ln W T +1 .

By the inequality x ≤ x α +( α -1) /e for x > 0 , α ≥ 0 , for any i ∈ [ N ] , we have:

Based on the updates in Line 7 of Algorithm 1, we bound the first term on the right-hand side as:

≤ ˜ w T,i (1 + η T,i (¯ r T,i -m T,i )) . (22) The last inequality is by exp( x -x 2 ) ≤ 1 + x for x ≥ -1 / 2 . This is a crucial condition needed to be verified for Lipschitz-adaptive meta-algorithm. By the tuning of learning rates, and the clipping technique, we control the range of η T,i (¯ r T,i -m T,i ) well:

which meets the criterion for applying the mentioned inequality. By plugging inequality (22) into (21), we can further analyze the weights for all experts at time T :

where we apply 1 -x ≤ exp( -x ) for any x ∈ R , and the last inequality is by the assumption in the theorem statement that ∑ N i =1 p t,i ¯ r t,i ≤ 0 for any t ∈ [ T ] . Now we are ready to upper bound W T +1 in an inductive style:

where the last inequality is by the induction. It remains to analyze the last term, the deviations of the learning rates. We present the following analysis tailored for the new learning rate setting, ∀ i ∈ [ N ] :

By Lemma 5 with the choices of f ( x ) = 1 /x, a 0 = 1 + 4 B 2 0 , a t = φ t,i in the lemma statement, summing up the preceding inequality from 1 to T results in:

Now combining (23) and (24), we can upper bound ln W T +1 as follows:

In another direction, we lower bound ln W T +1 ≥ ln w T +1 ,i /star with an inductive argument:

Rearranging the above equality with notice of Eq. (20), we have:

where the second term is in the order of

As for the first term, by applying Lemma 6, it can be bounded as follows:

Thus, the proof is complete.

C.4 Proof of Theorem 4

Proof. We first decompose the static regret based on the performance of base-learner i /star into two parts as presented in Eq. (8).

The base-regret is guaranteed by the corresponding base-learner via Theorem 1 and Theorem 2. And we mainly focus on the analysis of the meta-regret by leveraging Theorem 6. First we are required to verify the condition that ∑ i ∈ [ N ] p t,i ¯ r t,i ≤ 0 . Without loss of generality, we assume the 1 -st baselearner is for convex functions. Recall that we set r t,i = 〈∇ f t ( x t ) , x t -x t,i 〉 for strongly convex

functions learners i ∈ [2 , N ] , r t, 1 = f t ( x t ) -f t ( x t, 1 ) for the convex function learner, and optimism m t,i = f t -1 ( x t ) -f t -1 ( x t,i ) for each base-learner i ∈ [ N ] , therefore we have:

Therefore, Theorem 6 is applicable in analyzing the meta-regret for universal online learning. In follows, we prove the regret bounds for convex functions and strongly convex functions respectively.

Convex functions. By Theorem 1, the base-regret is bounded by O ( √ V T ) , as for the meta-regret, by the setting of inputs and optimism, by Theorem 6, it is bounded by where we denote by C T = O (log( B T +log( B T T ))) and in the third line we apply the mean value theorem. Combining the base-regret and meta-regret together, we concludes that the static regret bound for convex functions is bounded by O ( √ V T · log( B T +log( B T T ))) , where B T = O ( ̂ G max D ) with ̂ G max denoting the maximum Lipschitz constant.

Strongly convex functions. For λ /star -strong convex functions with λ /star ∈ [1 /T, 1] , by the construction of the curvature coefficient pool H , there exists i /star ∈ [2 , N ] such that

With this specific i /star -th base-learner, the base-regret can be upper bounded by O ((log V T ) /λ /star ) by Theorem 2, up to a multiplicative constant of 2 . The meta-regret can be bounded as follows:

Algorithm 3 Universal Gradient-Variation Online Learning under Global Smoothness

Input: curvature coefficient pools H sc , H exp, number of base-learners N , optimistic Adapt-MLProd [Wei et al., 2016] as the meta-algorithm, base-algorithms A cvx , A sc , A exp .

2: for t = 1 to T do

1: Initialization : Pass N to the meta-algorithm, initialize a base-learner with A cvx ; for λ ∈ H sc , initialize a base-learner with A sc ; for α ∈ H exp, initialize a base-learner with A exp .

3: Obtain p t from meta-algorithm, x t,i from each base-learner i ∈ [ N ] ;

6: For strongly convex and exp-concave functions learners : set r t,i = 〈∇ f t ( x t ) , x t -x t,i 〉 ;

4: Submit x t = ∑ i ∈ [ N ] p t,i x t,i ; 5: Receive ∇ f t ( x t ) and send it to each base-learner for update;

7: For convex functions learner : set r t, 1 = f t ( x t ) -f t ( x t, 1 ) ;

8: Send r t to the meta-algorithm;

9: Send m t +1 , 1 = f t ( x t ) -f t ( x t, 1 ) and m t +1 ,i = 0 , i ∈ [2 , N ] to the meta-algorithm.

10: end for

where we use ̂ G max to denote the maximum Lipschitz constant on the optimization trajectory, treat log log T as constants, and B T = O ( ̂ G max D ) . In the fourth line, we again utilize the mean value theorem. The fifth line follows from Lemma 7, which ensures that:

thus, our result depends on the Lipschitz constant on the optimization trajectory. In the last step, we apply the AM-GM inequality. The above statements show that the meta-regret is bounded by constants. With the base-regret guarantee, the proof for strongly convex functions is complete.

C.5 A Simple Universal Algorithm under Global Smoothness

As a byproduct, our techniques can be used to design a simpler two-layer universal algorithm that achieves the optimal gradient-variation regret bounds for convex, strongly convex, and exp-concave functions simultaneously, thereby improving upon the results of Yan et al. [2023]. The crux involves leveraging the function-variation-to-gradient-variation technique for convex functions, and following the strategy of Zhang et al. [2022a] to demonstrate that the meta-regret is bounded by constants for strongly convex and exp-concave functions, at a cost of O (log T ) times function value queries per round. Given the global smoothness constant and the Lipschitz constant, our algorithm does not need to be Lipschitz-adaptive, thus suitable for more general optimism settings.

In Algorithm 3, we present this idea. In contrast to Algorithm 2, which is designed under the generalized smoothness, this algorithm in addition can guarantee gradient-variation bound for expconcave functions. In below, we present the theoretical guarantees for Algorithm 3.

Corollary 3. Under Assumption 2, and assuming the loss functions are L -smooth and G -Lipschitz, we set N = 2 /ceilingleft log 2 T /ceilingright +1 . The curvature coefficient pools are defined as H sc = H exp = { 2 i -1 /T : i ∈ [( N -1) / 2] } . By selecting suitable base-algorithms, for λ, α ∈ [1 /T, 1] , Algorithm 3 ensures the following results simultaneously:

Proof. When assuming the Lipschitz constant G , optimistic Adapt-ML-Prod [Wei et al., 2016] can ensure the meta-regret bounded by O ( √ ∑ t t =1 ( r t,i -m t,i ) 2 · log log T ) , thus the dependence of logarithmic terms is improved compared to Theorem 4. The proofs for convex functions and strongly convex functions are nearly identical to the proofs for Theorem 4 in Appendix C.4; thus, we omit them here. We highlight the importance of the function-variation-to-gradient-variation technique, bounding the meta-regret of order O ( √ V T · log log T ) without the cancellation-based analysis.

Next, we show that the meta-regret for α /star -exp-concave functions is bounded by a constant. By the construction of the curvature pool H exp, there exists base-learner i /star with the input curvature α i /star satisfying that α i /star ≤ α /star ≤ 2 α i /star . Decompose the regret against this specific base-learner and by the definition of exp-concave functions, we have:

where the second-to-last line follows from the settings that r t,i /star = 〈∇ f t ( x t ) , x t -x t,i /star 〉 and m t,i /star = 0 , and we apply the AM-GM inequality in the final step. Therefore, by choosing the base-algorithm that ensures the regret bound of O ( d α /star log V T ) , we complete the proof for this theorem.

D Omitted Details for Section 4

This section provides the omitted proofs for our two applications.

D.1 Proof of Corollary 1

Proof. We prove this theorem in a black-box manner, thanks to the gradient-variation bound we derive. By Theorem 4, for convex functions, Algorithm 2 ensures:

O ( ‖∇ f t ( x ) -∇ F t ( x ) ‖ 2 2 + ‖∇ F t ( x ) -∇ F t -1 ( x ) ‖ 2 2 + ‖∇ F t -1 ( x ) -∇ f t -1 ( x ) ‖ 2 2 ) . (26) Finally, by taking expectation on both sides and leveraging the concavity of the square root, we have:

where we decompose ‖∇ f t ( x ) -∇ f t -1 ( x ) ‖ 2 2 as follows:

which is in order of O (√ Σ 2 I + √ ˜ σ 2 I ) . For strongly convex functions, as shown in Eq. (13) in the proof of Theorem 2, the multiplicative factor ̂ G max can be replaced by a more refined factor max t ∈ [ T ] ‖∇ f t ( x t ) - ∇ f t -1 ( x t ) ‖ 2 . Then Algorithm 2 can ensure the following bound for strongly convex functions:

By applying a similar argument as in Eq. (26) to decompose the gradient variation, taking the expectation on both sides, and leveraging the concavity of the logarithm, we conclude the proof.

D.2 Proof of Corollary 2

Proof. The step sizes for optimistic OMD in (10) is η t = min { D, min s ∈ [ t ] 1 4 /lscript s -1 (2 ‖ F s -1 ( ̂ z s ) ‖ 2 ) } . By the convexity and the concavity for the objective function f ( · , · ) , for any z = ( x , y ) ∈ Z , we can linearize the gap for an ε -approximate solution as f (¯ x T , y ) -f ( x , ¯ y T ) ≤ 1 T ∑ T t =1 〈 F ( z t ) , z t -z 〉 .

Following the proof of Lemma 2, we can demonstrate that optimistic OMD in (10) ensures:

The first term on the right-hand side is in the order of O ( ̂ L max D 2 ) by the step size configuration, where ̂ L max denotes the maximum smoothness constant on the optimization trajectory. By the /lscript -smoothness in Definition 2, the second term can be bounded as η t ‖ F ( z t ) -F ( ̂ z t ) ‖ 2 2 ≤ O ( η t /lscript s -1 (2 ‖ F s -1 ( ̂ z s ) ‖ 2 ) 2 ·‖ z t -̂ z t ‖ 2 2 ) , which can be further cancelled by the negative terms. Therefore, ∑ T t =1 〈 F ( z t ) , z t -z 〉 is bounded by a constant, thus, the convergence rate to an ε -approximate solution is O (1 /T ) , implying a fast convergence rate.

E Supporting Lemmas

E.1 Lemmas for Optimistic OMD

We first present the generic lemma for optimistic OMD with dynamic regret [Zhao et al., 2024], u u x T x which encompasses the standard regret by setting 1 = ... = T = /star ∈ arg min x ∈X ∑ t =1 f t ( ) . Lemma 3 (Theorem 1 of Zhao et al. [2024]) . Optimistic OMD specialized at Eq. (3) satisfies

where u 1 , . . . , u T ∈ X are arbitrary comparators in the feasible domain.

The next lemma, known as the stability lemma, establishes an upper bound on the proximity between successive decisions in terms of the gradient utilized for updates.

Lemma 4 (Proposition 7 of Chiang et al. [2012]) . Consider the following two updates: (i) x = arg min x ∈X {〈 g , x 〉 + D ψ ( x , c ) } , and (ii) x ′ = arg min x ∈X {〈 g ′ , x 〉 + D ψ ( x , c ) } . When the regularizer ψ : X ↦→ R is λ -strongly convex function with respect to norm ‖·‖ , we have λ ‖ x -x ′ ‖ ≤ ‖ g -g ′ ‖ ∗ .

E.2 Self-Confident Tuning Lemmas

In this part, we provide some useful lemmas when analyzing the self-confident tuning strategy.

Lemma 5 (Extension of Lemma 14 Gaillard et al. [2014]) . Let a 0 > 0 and a t ∈ [0 , B ] be real numbers for all t ∈ [ T ] and let f : (0 , + ∞ ) ↦→ [0 , + ∞ ) be a nonincreasing function. Then ∑ T t =1 a t f ( ∑ t -1 s =0 a s ) ≤ B · f ( a 0 ) + ∫ ∑ T t =0 a t a 0 f ( u )d u . be non-negative real numbers.

E.3 Technical Lemma

Lemma 7. Let f : X ↦→ R be a convex, twice differentiable function. Then for any x , y ∈ int X and λ ∈ [0 , 1] , we have that:

Proof. Define a function that φ ( t ) = f ( t x + (1 -t ) y ) . For this univariate convex function, we have φ ′ ( t ) = 〈∇ f ( t x + (1 -t ) y ) , x -y 〉 . By the convexity of φ ( t ) , there is | φ ′ ( t ) | ≤ max {| φ ′ (1) | , | φ ′ (0) |} , concluding the proof.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract, we present the contributions sequentially through ordinal words and we discuss the contributions further in the introduction part.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: One limitation of our work is the requirement for multiple gradient queries. Another limitation is that we cannot obtain the gradient-variation results for exp-concave functions due to the technical reasons.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are provided in Section 2. The main theoretical results are provided from Section 2 to Section 4. All proofs can be found in the Appendix and, specifically from Appendix B to Appendix D, which contain the proofs for the main results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not include experiments, and no data or code will be provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper only focuses on online learning theory, and we are not aware of any potential societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not include experiments, and there are no risks for misuse of data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not include experiments, and we do not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not include experiments, and we do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper focuses on online learning theory and does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper focuses on online learning theory and does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper contributes gradient-variation extensions of several online learning guarantees to\na generalized smoothness setting. Under a more general smoothness assumption,\nthe paper first provides an algorithm which achieves a $O(\\sqrt{V_T})$\ngurantee, where $V_T=\\sum_{t} \\sup_{x}\\\\|\\nabla f_t(x)-\\nabla f_{t-1}(x)\\\\|^2$, as well as a $O(\\log{V_T})$ guarantee for\nstrongly-convex functions.\nThe result is then extended to a universal guarantee, which holds\nwithout knowledge of whether the losses are strongly convex or just convex.\nThe approach is also extended to provide an analogous strongly-adaptive guarantee and a $O(\\sqrt{V_TP_T})$ dynamic regret guarantee.\n\nSTRENGTHS:\nThe generalized smoothness assumption is nice. The paper reads well and the technical development easy to follow. Results seem novel.\n\nWEAKNESSES:\nThe assumption that the learner has prior knowledge of and can query the smoothness function arbitrarily seems very strong\n\nThe method requires multiple-query gradient access\n\nThe approach requires prior knowledge of $\\min_x f_t(x)$, which makes the problem quite a bit easier. One of the difficult things\nthat arises in these non-globally-bounded settings is that the learner doesn't have a proper frame-of-reference for how large\nlosses and gradients really are; giving the learner access to $\\min_x f_t(x)$ provides a very strong version of such a frame-of-reference\n\nQUESTIONS:\n- \"However, to ensure the theoretical results are valid, we assume that there exist finite but unknown upper bounds G and L for the Lipschitzness and smoothness following the discussion in Jacobsen and Cutkosky [35].\"\n  - What discussion is being referred to here specifically?\n\nLIMITATIONS:\n- The limitation in terms of a potential exp-concave result is stated explicitly in Section 2.3\n- The assumption that $\\min_x f_t(x)$ is required is pointed out explicitly. Although it it's significance\n  is a bit downplayed.\n- Multiple-query gradient access is also pointed out in the conclusion", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nProblem: The paper studies the OCO problem under the generalized smoothness assumption, i.e. at each time $t$, $f_{t}$ satisfies $\\|| \\nabla ^ {2} f_{t} (x) \\|| \\le \\ell_{t}(\\|| \\nabla f_{t}(x) \\||)$ for all $x \\in \\mathcal{X}$, where $\\ell_{t}$ is a positive non-decreasing function. In addition to the information model in OCO, the algorithm can query $\\ell_{t}$ at any local point $x \\in \\mathcal{X}$. This condition subsumes global smoothness, i.e. $\\ell \\le L$ is upper bounded globally, and $(L_{0},L_{1})$ smoothness, i.e. $\\ell(x) = L_0 + L_1 x$; which have been studied in the prior works. The authors obtain gradient-based variation bounds for several settings of interest (see contributions below).\n\nContributions: \n1) First, the authors extend the classic optimistic OMD algorithm to handle generalized smoothness and obtain an optimal $\\mathcal{O}(\\log V_{T})$ rate for strongly convex and $\\mathcal{O}(\\sqrt{V_{T}})$ static regret for convex functions (ref. Theorem 1, 2). However, these rates are first derived assuming that the algorithm knows the curvature of the $f_{t}$'s, i.e. whether all the $f_{t}$'s are convex or strongly convex.\n\n2) To circumvent the curvature information, the authors devise an algorithm that is adaptive to the curvature of the functions $f_{t}$'s (referred to as universal online learning -- the goal is to obtain an algorithm with simultaneous guarantees for convex and strongly convex functions). The proposed algorithm obtains the optimal $\\mathcal{O}(\\sqrt{V_T}), \\mathcal{O}(\\log V_T)$ static regret bounds for convex, strongly convex functions respectively (ref. Theorem 4).\n\n3) Finally, the authors consider stronger regret measures, i.e. interval regret and dynamic regret (referred to as non-stationary online learning), and propose an algorithm with the optimal gradient variation-based regret guarantees (ref. Theorem 5, 6).\n\nSTRENGTHS:\nThe paper is solid and shows that gradient variation-based bounds can be derived under a more general smoothness assumption. Prior works focused on global smoothness, or $(L_{0}, L_{1})$ smoothness, and did not consider some or the other regret metrics as considered in the paper, e.g. the authors mention that an adaptive regret guarantee was not obtained even in the context of global smoothness.  \n\nWhile this might seem doable, the authors very well mention the key challenges in obtaining the desired algorithms, e.g. for contribution (2) above, section 3.2 does a great job of highlighting the key challenges. I liked this section and the authors spent considerable effort explaining why existing meta algorithms, e.g. McMwC-Master algorithm (Chen et al.) do not directly work since they cannot handle heterogeneous inputs. \n\nThe function-to-gradient variation technique is simple and seems pretty useful since it allows us to decouple the meta-algorithm and the base learners and explicitly aim toward obtaining a meta-algorithm with the desired guarantee. It helps avoid the seemingly messier cancellation-based tricks as done by Yan et al.\n\nWEAKNESSES:\n1) Section 4 is presented in a rush. Like section 3.1, it would be quite beneficial to have some more discussion on the algorithm and potential challenges in deriving non-stationary regret guarantees. The authors claim that their proposed algorithm is the first to obtain gradient variation-based non-stationary regret guarantees under the generalized smoothness assumption, whereas prior works did not obtain similar guarantees under the global smoothness assumption. The authors should have at least backed this with the potential difficulties faced by existing approaches. I recommend that the authors cut down some earlier discussions, such as the discussion about exp-concave functions, which in my opinion is not very relevant, given that the entire paper is about convex and strongly convex functions.\n\n2) While the paper shows interesting results, I feel the key ideas are borrowed from existing works. While this is fine, in some places the authors failed to mention the potential technical challenges in the analysis. This is the most applicable to section 3. The authors mention that the proposed algorithm can be analyzed similarly to the prod-style algorithms. Are there technical difficulties in directly incorporating the analysis of prod-style algorithms into Algorithm 1? \n\nSome minor typos that I found:\n\n1) Line 38: \"stochatsic -> stochastic\"\n2) Lemma 1: Remove the $\\ell_t$\n\nQUESTIONS:\n1) The discussion towards the end of page 3 is confusing to me. The authors mention that assuming the Lipschitzness of $f_{t}$ is not reasonable since the bound on $\\|| \\nabla ^ {2} f(x) \\||$ can be obtained from knowing $\\ell$ directly, and the function is globally smooth. However, the bounds presented in the Theorems are a function of the Lipschitzness parameters. I would appreciate clarification from the authors.\n\n2) In the Equation after Line 384, $\\xi_{t, i}$ is not necessarily same for $f_t$ and $f_{t - 1}$ since the Mean Value Theorem only guarantees the existence of a $\\xi$. With $\\xi_t$ and $\\xi_{t-1}$ being possibly different, I don't think the next inequality follows immediately. Is that a typo or am I missing something?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
The paper aims to provide gradient-variation regret bound when only a generalized smoothness condition holds. It considers the optimistic mirror descent algorithm. It further designed a meta-algorithm, which can be Lipscthiz adaptive. Lastly, it considers adaptive regret and dynamic regret.

STRENGTHS:
(1) Compared with existing literature, the work relaxed the smoothness assumption.

(2) The meta-algorithm to achieve universality seems novel to me.

WEAKNESSES:
The clarity and effectiveness of the paper could be improved by condensing the extensive paragraphs that discuss "challenges." Instead, it would be beneficial to expand the remarks on how the findings compare with existing results following the theorems. This would provide readers with a clearer understanding of the paper's contributions to the current body of knowledge.

QUESTIONS:
1. The paper addresses the generalized smoothness condition and frequently mentions the potential for unbounded gradients and smoothness. However, Assumption 2 posits a bounded domain. Does this imply that the gradients and smoothness are also bounded? In reference [7], such boundedness does not appear to be a requirement. 

2. On page 5, line 203, the paper asserts that the complexity \(O(D\sqrt{V_T} + LD^2)\) aligns with the optimal constant dependency. It would be helpful if the authors could specify where exactly in reference [7] this lower bound is discussed.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies adaptive online learning under the generalized smoothness assumption. The authors proposed optimistic OMD based algorithms that achieves the first gradient variation bounds under this general setting. Under this assumption, they also provide uninveral algorithms which can adapt to different function types, and also algorithms with dynamic regret.\n\nSTRENGTHS:\n1. This paper is the first to study adaptive online learning under generalized smoothness, which is a much milder condition compared to the standard global smoothness that considered by previous work. \n\n\n2. In section 2, the authors provide optimistic OMD based algorithms with adaptive step size, and prove that it enjoys O(sqrt{V_T}) and O(log V_T/\\lambda) regret bounds for convex and strongly convex functions. To my knowledge, these are the first adaptive bounds under the generalized smoothness condition, which is a novel and interesting contribution. \n\n3. The authors also provide a universal algorithm for achieving gradient variation bounds under the generalized smoothness setting, which can adapt to convex and strongly convex functions, without knowing the type of loss functions in advance. Existing universal methods cannot be directly applied due to the unknown Lipschitzness parameters, and the authors address it by using novel techniques such as function-variation-to-gradient-variation conversion.\n\n4. The authors also showed that the proposed methods can be applied to get adaptive bounds in stochastic extended adversarial online learning and the fast-rate games under more general assumptions.\n\nWEAKNESSES:\n1. The base learner proposed in Section 2 needs to know the Smooth constant of the previous observed loss functions (at round t-1), which might be difficult to compute if the loss functions are complicated.\n\n2. The proposed universal algorithm requires to query the gradient O(\\log T) times per iterations, while most of the existing universal methods only need to query the gradient once.\n\n3. Previous universal algorithms with gradient variation bound can deal with exp-concave functions, while the proposed algorithm can only deal with convex and strongly convex functions.\n\n4. Previous work can also achieve small-loss bound in this setting. It would be great to understand if small-loss bound is achievable under this more general assumption. \n\n\nMinor comments:\n\nLine 56: sqrt{V}_T: T should be inside the square root;\n\nLine 140: “a finte bound but unknown L” is not assumed in this paper\n\nLine 197: it is more common to use equality for the big-O notation.\n\nQUESTIONS:\nPlease see weakness\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}