Title: Online Composite Optimization Between Stochastic and Adversarial Environments
Venue: NeurIPS 2024
Accepted: yes
Keywords: Online Composite Optimization

Online Learning
Abstract: We study online composite optimization under the Stochastically Extended Adversarial (SEA) model. Specifically, each loss function consists of two parts: a fixed non-smooth and convex regularizer, and a time-varying function which can be chosen either stochastically, adversarially, or in a manner that interpolates between the two extremes. In this setting, we show that for smooth and convex time-varying functions, optimistic composite mirror descent (OptCMD) can obtain an $\mathcal{O}(\sqrt{\sigma_{1:T}^2} + \sqrt{\Sigma_{1:T}^2})$ regret bound, where $\sigma_{1:T}^2$ and $\Sigma_{1:T}^2$ denote the cumulative stochastic variance and the cumulative adversarial variation of time-varying functions, respectively. For smooth and strongly convex time-varying functions, we establish an $\mathcal{O}((\sigma_{\max}^2 + \Sigma_{\max}^2)\log(\sigma_{1:T}^2 + \Sigma_{1:T}^2))$ regret bound, where $\sigma_{\max}^2$ and $\Sigma_{\max}^2$ denote the maximal stochastic variance and the maximal adversarial variation, respectively. For smooth and exp-concave time-varying functions, we achieve an $\mathcal{O}(d \log (\sigma_{1:T}^2 + \Sigma_{1:T}^2))$ bound where $d$ denotes the dimensionality. Moreover, to deal with the unknown function type in practical problems, we propose a multi-level \textit{universal} algorithm that is able to achieve the desirable bounds for three types of time-varying functions simultaneously. It should be noticed that all our findings match existing bounds for the SEA model without the regularizer, which implies that there is \textit{no price} in regret bounds for the benefits gained from the regularizer.

Online Composite Optimization Between Stochastic and Adversarial Environments

Abstract

We study online composite optimization under the Stochastically Extended Adversarial (SEA) model. Specifically, each loss function consists of two parts: a fixed non-smooth and convex regularizer, and a time-varying function which can be chosen either stochastically, adversarially, or in a manner that interpolates between the two extremes. In this setting, we show that for smooth and convex time-varying functions, optimistic composite mirror descent (OptCMD) can obtain an O ( √ σ 2 1: T + √ Σ 2 1: T ) regret bound, where σ 2 1: T and Σ 2 1: T denote the cumulative stochastic variance and the cumulative adversarial variation of time-varying functions, respectively. For smooth and strongly convex time-varying functions, we establish an O (( σ 2 max +Σ 2 max ) log( σ 2 1: T +Σ 2 1: T )) regret bound, where σ 2 max and Σ 2 max denote the maximal stochastic variance and the maximal adversarial variation, respectively. For smooth and exp-concave time-varying functions, we achieve an O ( d log( σ 2 1: T +Σ 2 1: T )) bound where d denotes the dimensionality. Moreover, to deal with the unknown function type in practical problems, we propose a multilevel universal algorithm that is able to achieve the desirable bounds for three types of time-varying functions simultaneously. It should be noticed that all our findings match existing bounds for the SEA model without the regularizer, which implies that there is no price in regret bounds for the benefits gained from the regularizer.

1 Introduction

Online composite optimization has drawn considerable attention in recent years [Duchi and Singer, 2009, Ghadimi and Lan, 2012, Lei and Zhou, 2017, Scroccaro et al., 2023]. Formally, it can be viewed as an iterative game between a learner and the environment. In each round t , the learner makes a decision x t from a convex set X ⊆ R d and then suffers a loss φ t ( x t ) in the form of

where f t ( · ) : X → R denotes the time-varying function that is chosen by the environment, and r ( · ) : X → R denotes the fixed non-smooth and convex regularizer, such as the glyph[lscript] 1 -norm for sparse vectors and the trace norm for low-rank matrices [Langford et al., 2009, Flammarion and Bach, 2017, Zhang et al., 2019, Garber and Kaplan, 2019]. In the literature, online composite optimization is generally divided into two categories: stochastic composite optimization [Lan, 2012, 2016, Zhang et al., 2017, Lei and Tang, 2018, Lei et al., 2019, Kulunchakov and Mairal, 2019] and adversarial composite optimization [Xiao, 2009, Duchi et al., 2010, Mohri and Yang, 2016, Joulani et al., 2020, Yang et al., 2024c]. In the former one, f t ( · ) is assumed to be independent and identically distributed (i.i.d.) over time; in the latter one, f t ( · ) can be chosen arbitrarily or even adversarially. However,

the environment in real-world scenarios is seldom purely stochastic or adversarial, but rather falls somewhere in between [Amir et al., 2020, Garber et al., 2020, Sherman et al., 2021, Zimmert and Seldin, 2021, Ito, 2021], and our understanding for the more common intermediate scenarios remains limited in online composite optimization.

Recently, Sachs et al. [2022] introduce the Stochastically Extended Adversarial (SEA) model as an intermediate setting in online optimization without regularizer, i.e., r ( x ) = 0 . In the SEA model, loss functions are not restricted to fully i.i.d. or adversarial; instead, the environment selects them from any intermediate state between the two extreme settings. To reflect how stochastic or adversarial the environments are, they introduce the cumulative stochastic variance σ 2 1: T and the cumulative adversarial variation Σ 2 1: T , as shown below:

where σ 2 t = sup x ∈X E f t ∼D t [ ‖∇ f t ( x ) -∇ F t ( x ) ‖ 2 2 ] denotes the variance of gradients and F t ( x ) = E f t ∼D t [ f t ( x )] denotes the expected function for f t ( · ) . For the SEA model, two classical algorithms in optimistic online learning [Rakhlin and Sridharan, 2013]-optimistic Follow-The-Regularized Leader (FTRL) [Sachs et al., 2022] and optimistic Online Mirror Descent (OMD) [Chen et al., 2023]-have been proven ensuring sublinear regret bounds with respect to both σ 2 1: T and Σ 2 1: T .

In this paper, we extend the SEA model into online composite optimization, termed as composite SEA, to bridge the gap between stochastic and adversarial composite optimization. Specifically, in (1), r ( · ) remains fixed over time and f t ( · ) is selected from a distribution D t , which is chosen by the environments either stochastically, adversarially, or in a manner that interpolates between the two extremes. The goal of composite SEA is to minimize the expected regret in terms of (1):

which benchmarks the cumulative composite loss of the learner and that of the best fixed decision. To handle the composite setting, a natural impulse is to treat f t ( · ) + r ( · ) as one function and directly apply existing methods for the SEA model, i.e., optimistic FTRL and optimistic OMD. However, such a straightforward application is unsuitable because (i) these methods heavily rely on the smoothness of loss functions, but due to the non-smooth component r ( · ) , the summation function φ t ( · ) = f t ( · )+ r ( · ) loses this crucial property; (ii) directly applying these methods ignores the presence of r ( · ) and thus fails to gain the benefits from the regularizer, e.g., the sparsity induced by the glyph[lscript] 1 -norm. Therefore, a natural question arises whether it is possible to deal with both the intermediate nature of environments and the composite structure of loss functions concurrently.

We affirmatively answer the above question by revisiting a variant of optimistic OMD [Scroccaro et al., 2023], named Optimistic Composite Mirror Descent (OptCMD). This method inherits the idea of optimistic online learning [Rakhlin and Sridharan, 2013], i.e., exploiting estimates on upcoming loss functions for decision updates, and can disentangle f t ( · ) and r ( · ) during the optimization, thereby effectively leveraging distinct properties of each. However, OptCMD is originally designed for adversarial composite optimization, and a simple extension will lead to unsatisfactory bounds in composite SEA. In this paper, we reanalyze OptCMD and show that with suitable configurations, OptCMD attains O ( √ σ 2 1: T + √ Σ 2 1: T ) , O (( σ 2 max +Σ 2 max ) log( σ 2 1: T +Σ 2 1: T )) and O ( d log( σ 2 1: T + Σ 2 1: T )) bounds for smooth and general convex, smooth and strongly convex, and smooth and expconcave time-varying functions, respectively. Our findings generalize previous results in stochastic and adversarial composite optimization, and can reduce to them by specializing σ 2 1: T and Σ 2 1: T . Moreover, our results coincide with existing bounds for the SEA model [Sachs et al., 2022, Chen et al., 2023], which indicates that there is no price in regret bounds for the benefits from r ( · ) .

One concern of OptCMD is the requirement of the function type in advance, which is often impractical in real-world problems and motivates us to design a universal algorithm that is agnostic to the prior knowledge about loss functions. We note that recently, Yan et al. [2023] have introduced a universal algorithm for the SEA model without regularizer, based on the meta-expert structure in online learning [van Erven and Koolen, 2016, Mhammedi et al., 2019, Wang et al., 2019, Zhang et al., 2022]. However, their method [Yan et al., 2023] cannot naturally support the composite setting, as it highly depends on the smoothness of the (summation) loss functions, which does not necessarily hold in the composite SEA model. In this paper, we propose a novel universal algorithm for composite SEA,

based on the observations that (i) the regularizer remains fixed over time and is accessible to the meta-algorithm from the beginning round; (ii) the meta-algorithm is able to obtain expert decisions for the current round before estimating their performance. With these two facts, we can utilize the information about r ( · ) to track experts, ensuring the meta-regret will not deviate from that of the best one. Specifically, inspired by Yan et al. [2023], we employ a two-layer Multi-scale Multiplicativeweight with Correction (MsMwC) [Chen et al., 2021] as the meta-algorithm, and choose OptCMD as the expert-algorithm. To effectively estimate the expert performance, we first collect decisions of each expert for the current round, and then explicitly integrate them and expert performance on r ( · ) into the losses and optimisms used in MsMwC. Theoretical analysis demonstrates that, with the proposed integration, our universal algorithm can achieve desirable bounds for three types of loss functions simultaneously. We summarize our contributions as shown below.

We first introduce the composite SEA model, which serves as an intermediate setting between stochastic and adversarial composite optimization and can naturally adapt to two extreme settings;

For the composite SEA model, we demonstrate that OptCMD can attain the regret bounds of O ( √ σ 2 1: T + √ Σ 2 1: T ) , O (( σ 2 max +Σ 2 max ) log( σ 2 1: T +Σ 2 1: T )) and O ( d log( σ 2 1: T +Σ 2 1: T )) for smooth and general convex, smooth and strongly convex, and smooth and exp-concave time-varying functions, respectively. Owing to the versatility of σ 2 1: T and Σ 2 1: T , these bounds can recover previous results in stochastic and adversarial composite optimization;

Moreover, to handle the unknown function type, we propose a new universal algorithm in composite SEA and show that it can concurrently achieve desirable bounds for three kinds of functions;

Finally, we discuss implications of our theoretical findings for two common intermediate examples with composite loss functions, and derive favorable bounds specific to these practical scenarios.

2 Related work

In this section, we briefly review related work on adversarial and stochastic composite optimization, the SEA model and universal online learning.

Adversarial composite optimization. In this setting, f t ( · ) is assumed to be chosen by the environments arbitrarily or even adversarially, and the performance is measured by regret shown in brackets of (3). In the literature, the seminal work [Duchi and Singer, 2009] proposes the FOBOS method with O ( √ T ) and O ( λ -1 log T ) regret bounds for general convex and λ -strongly convex f t ( · ) , respectively. Later, Xiao [2009] proposes the RDA method based on the primal-dual subgradient framework [Nesterov, 2009], and shows that RDA is able to generate sparser decisions than FOBOS when r ( x ) = µ ‖ x ‖ 1 for some µ > 0 . In the same time, another subsequent work [Duchi et al., 2010] introduces the COMID method under the mirror descent framework [Beck and Teboulle, 2003], and achieves the same bounds as FOBOS. Recently, for α -exp-concave f t ( · ) , Yang et al. [2024c] establish an O (( d/α ) log T ) regret bound by proposing the ProxONS method.

Besides, there exist other powerful methods [Yang et al., 2014, Mohri and Yang, 2016, Joulani et al., 2020, Scroccaro et al., 2023] equipped with problem-dependent bounds. These bounds can safeguard the above results in the worst-case scenarios and become tighter when environments have special properties, such as smoothness. Among them, the most related work is by Scroccaro et al. [2023], who develop OptCMD based on the optimistic online learning framework [Rakhlin and Sridharan, 2013]. The key idea is to make an estimate for the upcoming loss function, which ensures tighter bounds when the estimate is accurate and still maintains the worst-case bound otherwise. By utilizing the smoothness of f t ( · ) , OptCMD achieves O ( √ V T ) and O ( λ -1 log V T ) regret bounds for general convex and λ -strongly convex f t ( · ) , respectively, where V T = ∑ T t =1 max x ∈X ‖∇ f t ( x ) -∇ f t -1 ( x ) ‖ 2 2 denotes the gradient-variation and can be small when f t ( · ) gradually changes.

Stochastic composite optimization. In this setting, f t ( · ) is assumed to be i.i.d. sampled from a fixed distribution D , and the goal is to minimize the composite objective: min x ∈X E f ∼D [ f ( x )] + r ( x ) . The performance is measured by excess risk , which compares the solution with the optimal one, i.e., E f ∼D [ f ( x T )] + r ( x T ) -min x ∈X { E f ∼D [ f ( x )] + r ( x ) } .

In the literature, there are a substantial body of methods designed for the stochastic setting [Ghadimi and Lan, 2012, Lan, 2012, 2016, Lei and Tang, 2018, Lei et al., 2019], and by utilizing the online-tobatch conversion technique [Cesa-Bianchi et al., 2004], existing methods for adversarial composite optimization [Xiao, 2009, Duchi et al., 2010, Yang et al., 2024c] can also be extended to the stochastic

scenarios. Specifically, Lan [2012] first establishes an O (1 / √ T ) excess risk bound for general convex objective, and then Lan [2016] improves the rate to O (1 /T ) for strongly convex objective. By utilizing the online-to-batch conversion, both RDA [Xiao, 2009] and COMID [Duchi et al., 2010] achieve O (1 / √ T ) and O (log T/ ( λT )) excess risks for general convex and λ -strongly convex objective, respectively. When f ( · ) is α -exp-concave, ProxONS [Yang et al., 2024c] ensures an O ( d log T/ ( αT )) excess bound.

SEA model. The SEA model is originally introduced by Sachs et al. [2022] as an intermediate setting in online optimization without regularizer. Moreover, they also propose two versatile quantities σ 2 1: T and Σ 2 1: T to reflect the stochastic and adversarial aspect of the environments, respectively. Theoretically, for the SEA model, Sachs et al. [2022] prove that optimistic FTRL enjoys the bounds of O ( √ σ 2 1: T + √ Σ 2 1: T ) and O ( λ -1 ( σ 2 max + Σ 2 max ) log T ) for smooth and general convex, and smooth and λ -strongly convex loss functions, respectively. Later, Chen et al. [2023] demonstrate that optimistic OMD is able to attain the same bound for general convex losses and an improved O ( λ -1 ( σ 2 max +Σ 2 max ) log( σ 2 1: T +Σ 2 1: T )) bound for the strongly convex losses. Moreover, they also establish a new O (( d/α ) log( σ 2 1: T +Σ 2 1: T )) bound for smooth and α -exp-concave functions.

Universal online learning. The universal online learning is proposed to handle the uncertainty of the loss function types, when applying online algorithms to practical optimization problems. The center to universal algorithms is the powerful meta-expert structure [van Erven and Koolen, 2016, Mhammedi et al., 2019, van Erven et al., 2021], which is also widely-used in many other fields of online learning [Daniely et al., 2015, Jun et al., 2017a,b, Zhang et al., 2018, 2020, 2021, Cutkosky, 2020, Wan et al., 2021, 2022b, Wang et al., 2024, Wan et al., 2024a]. The key idea of meta-expert structure is to maintain multiple experts to process different types of loss functions and then, deploy a meta-algorithm to combine the decisions from experts [Wang et al., 2019, 2020, van Erven et al., 2021, Zhang et al., 2022, Yang et al., 2024a,b]. In the literature, the most related work is Yan et al. [2023], who propose a novel universal algorithm with a two-layer MSMWC [Chen et al., 2021] as the meta-algorithm. By maintaining multiple instances of optimistic OMD, their method can adapt to the SEA model and deliver the same bounds as those in Chen et al. [2023]. Recently, Yan et al. [2024] further improve this method by employing a simpler meta-algorithm while achieving optimal regret bounds for three types of functions. However, it should be noticed that these methods [Yan et al., 2023, 2024] heavily relies on the smoothness of the summation function (1), and thus cannot handle the composite structure in loss functions well, as previously discussed.

3 Preliminaries

In this section, we introduce some preliminaries, including standard assumptions and a brief review of OptCMD [Scroccaro et al., 2023].

3.1 Assumptions

We first list the common assumptions in prior studies [Duchi et al., 2010, Chen et al., 2023, 2024].

Assumption 1. The convex decision set X belongs to an Euclidean ball with the diameter D .

Assumption 2. At each round t , the random function f t ( · ) is G -Lipschitz over X , i.e.,

Assumption 3. The regularized function r ( · ) is convex and bounded over X , i.e.,

Assumption 4. At each round t , the expected function F t ( · ) is H -smooth over X , i.e.,

Assumption 5. At each round t , the expected function F t ( · ) is convex over X , i.e.,

Assumption 6. At each round t , the expected function F t ( · ) is λ -strongly convex over X , i.e.,

Algorithm 1 Optimistic Composite Mirror Descent (OptCMD)

Initialization : Let x 1 = ˆ x 1 be an arbitrary point in X .

1: for t = 1 to T do

2: Submit x t , and the environment chooses a distribution D t

3: Suffer the composite loss f t ( x t ) + r ( x t ) , where f t ( · ) is sampled from D t

4: Update ˆ x t +1 and x t +1 according to (4) and (5), respectively

5: end for

Assumption 7. All the variance of the gradients and the adversarial variations are bounded, i.e.,

Assumption 8. At each round t , the time-varying function f t ( · ) is α -exp-concave over X , i.e., ∀ x ∈ X , exp( -αf t ( x )) is concave.

3.2 Optimistic composite mirror descent

OptCMD is an algorithmic realization of the powerful optimistic online learning [Rakhlin and Sridharan, 2013] in online composite optimization. Specifically, in each round t , the learner submits a decision x t and suffers a loss f t ( x t ) + r ( x t ) . Then, the learner receives an optimism M t +1 that serves as an optimistic prediction for the gradient of subsequent function f t +1 ( · ) . After that, the learner performs the following update steps:

where B R t ( x , y ) = R t ( x ) -R t ( y ) -〈∇R t ( y ) , x -y 〉 denotes the Bregman divergence associated with a time-varying function R t , of which the specific form depends on the type of f t ( · ) and will be illuminated later. Note that in (4) and (5), r ( · ) remains non-linearilzed, which allows the decisions { x t } T t =1 to possess properties externally conferred by r ( · ) , such as the sparsity.

Remark. Scroccaro et al. [2023] specialize the estimate M t +1 = ∇ ˆ f t +1 (ˆ x t +1 ) , where ˆ f t +1 ( · ) denotes the function prediction of f t +1 ( · ) and is generally set as f t ( · ) in practice. We hightlight that this specification is unsuitable for the composite SEA model and inadvertently introduces a dependency issue during the analysis. More detailed discussions can be found in Sections 4.1 and 4.2.

4 OptCMD for composite SEA

In this section, we provide our results for general convex, strongly convex and exp-concave cases in the composite SEA model. Due to space limitations, all the proofs are deferred to Appendix B.

4.1 General convex case

Initially, we consider the case where the regualizer r ( · ) is convex and non-smooth, and the expected functions F t ( · ) are general convex and smooth. We choose the following configuration for this case.

where ¯ V t -1 = ∑ t -1 s =1 ‖∇ f s ( x s ) -∇ f s -1 ( x s -1 ) ‖ 2 2 and δ > 0 denotes the hyperparameter. With the configuration in (6), we establish the following regret bound for the general convex case.

Theorem 1. Under Assumptions 1, 2, 3, 4 and 5, Algorithm 1 ensures

with the configuration in (6) where δ = 6 √ 2 HD .

Remark. For the fully adversarial environments where Σ 2 1: T = V T and σ 2 1: T = 0 , our bound degenerates to O ( √ V T ) matching the previous result of Scroccaro et al. [2023]. For the fully stochastic environments where Σ 2 1: T = 0 and σ 2 1: T = σ 2 T with the stochastic variance σ 2 , our result becomes O ( √ T ) regret bound and further delivers an O (1 / √ T ) excess risk bound by the online-to-batch conversion [Cesa-Bianchi et al., 2004], which coincides with the results of Lan [2012]. Furthermore, our finding also aligns with existing bounds for the SEA model without r ( · ) [Sachs et al., 2022, Chen et al., 2023]. In other words, our method pays no price in regret bounds for handling the additional regularizer.

Beyond the configuration in (6), we also employ those adopted by Scroccaro et al. [2023]:

where ¯ D 0 = 0 and ¯ D t -1 = ∑ t -1 s =1 ‖∇ f s (ˆ x s ) -∇ f s -1 (ˆ x s ) ‖ 2 2 . By setting the configuration in (7), we obtain the following bound.

Theorem 2. Under Assumptions 1, 2, 3, 4 and 5, with configuration in (7) , Algorithm 1 ensures

where ˜ σ 2 1: T with ˜ σ 2 t = E f t ∼D t [sup x ∈X ‖∇ f t ( x ) -∇ F t ( x ) ‖ 2 2 ] .

Remark. This bound is less favorable than that in Theorem 1, as it scales with a new quantity ˜ σ 2 1: T , which also measures the stochasticity in composite SEA but is larger than σ 2 1: T because of the fact that E f t ∼D t [sup x ∈X ‖∇ f t ( x ) -∇ F t ( x ) ‖ 2 2 ] ≥ sup x ∈X E f t ∼D t [ ‖∇ f t ( x ) -∇ F t ( x ) ‖ 2 2 ] , where the inequality is due to the convexity of supremum operator. This discrepancy arises from the inappropriate choice of M t +1 = ∇ f t (ˆ x t +1 ) , where ˆ x t +1 is generated based on ∇ f t ( x t ) shown in (4), which implies that ˆ x t +1 has already incorporated partial information about f t ( · ) , leading to a dependence issue in analysis. To remove the dependency, we apply the supremum operator on ˆ x t over X (c.f. Lemma 4) but inevitably establish a reliance on ˜ σ 2 1: T in the final bound.

4.2 Strongly convex case

In this part, we focus on the case where the regualizer r ( · ) is non-smooth and convex, and the expected functions F t ( · ) are smooth and λ -strongly convex. We set OptCMD with the configurations:

where δ > 0 denotes the hyperparameter. The theoretical guarantee for this case is shown below.

Theorem 3. Under Assumptions 1, 2, 3, 4, 6 and 7, Algorithm 1 ensures

with the configuration in (8) where δ = 128 H 2 D 2 .

Remark. Similar to Theorem 1, this bound reduces to O ( λ -1 log V T ) matching that of Scroccaro et al. [2023] when environments become fully adversarial, and derives the same O (log T/ ( λT )) excess risk bound as that of Duchi and Singer [2009] and Xiao [2009] when environments become fully stochastic. Moreover, this bound also recovers that of Chen et al. [2023] for the SEA model.

Additionally, we further equip OptCMD with the original configuration of Scroccaro et al. [2023]:

and obtain the following theorem.

Theorem 4. Under Assumptions 1, 2, 3, 4 and 6, with the configuration in (9) , Algorithm 1 ensures

Remark. This bound suffers two limitations. Firstly, it depends on the unfavorable ˜ σ 2 1: T , which is due to the improper estimation M t +1 in (9). Secondly, it scales with O ( G 2 ) due to the choice of η t in (9), which is not tighter than O ( σ 2 max +Σ 2 max ) in Theorem 3.

Initialization : M top with η k = ( C 0 · 2 k ) -1 and ˆ q k 1 = ( η k ) 2 / ∑ K k =1 ( η k ) 2 ; for each k ∈ [ K ] , M k mid with η k,i = 2 η k and ˆ p k,i 1 = 1 /N , and experts { E k,i } i ∈ [ N ]

1: for t = 1 to T do

3: Compute x k t = ∑ i p k,i t x k,i t and submit x t = ∑ k q k t x k t

2: Receive x k,i t from expert E k,i , and update q t and p k t according to (11)

4: Suffer the loss f t ( x t ) + r ( x t ) and observe the gradient g t = ∇ f t ( x t )

5: Update ˆ q t +1 and ˆ p k t +1 according to (12)

6: Send the gradient g t and the regularizer r ( · ) to experts

7: end for

4.3 Exp-concave case

We further investigate the exp-concave case where the regualizer r ( · ) is non-smooth and convex, and the individual functions f t ( · ) are smooth and α -exp-concave.

Remark. In this case, we require the exponential concavity on the individual function f t ( · ) instead of the expected one F t ( · ) . This is due to the technical demand in analysis, and similar assumptions are also adopted by Chen et al. [2023] and Yang et al. [2024c].

In this case, we set OptCMD with the following configuration:

where δ > 0 denotes the hyperparameter, I denotes the d -dimension identity matrix, h t = ∇ f t ( x t ) ∇ f t ( x t ) glyph[latticetop] and β = (1 / 2) min { 1 / (4 GD ) , α } . The regret for this case is stated below.

Theorem 5. Under Assumptions 1, 2, 3, 4 and 8, Algorithm 1 ensures

with the configuration in (10) where δ = 1 .

Remark. For the fully adversarial setting, this bound degenerates to O (( d/α ) log V T ) , which is the first problem-dependent bound for exp-concave time-varying functions in online composite optimization, and tighter than O (( d/α ) log T ) by Yang et al. [2024c] in benign environments. For the fully stochastic setting, our result reduces to O (( d/α ) log T ) regret bound and further implies the same excess risk of O ( d log T/ ( αT )) as that of Yang et al. [2024c] through the online-to-batch conversion. Furthermore, it also matches existing results for the SEA model [Chen et al., 2023].

5 The universal strategy

Although we have established favorable theoretical results in the composite SEA model, achieving these guarantees requires prior knowledge of the function type to set appropriate configurations, e.g., (6) for the general convex case. This requirement is often impractical in real-world scenarios and motivates us to design a universal strategy. In the previous study of Yan et al. [2023], based on the meta-expert framework [van Erven and Koolen, 2016, Wang et al., 2020, Zhang et al., 2022], they propose a multi-layer universal algorithm, which can adapt to the SEA model without regularizer by choosing optimistic OMD [Chen et al., 2023] as the expert-algorithm. Unfortunately, their method [Yan et al., 2023] requires the smoothness of the (summation) loss functions, which does not necessarily hold for (1), so that it cannot be directly applied to the composite SEA model. In this paper, we propose a novel universal algorithm for composite SEA. For the expert-algorithm, we choose OptCMD due to its ability in handling three cases in composite SEA. For the meta-algorithm, we design a new method that is able to effectively manage the non-smooth component r ( · ) in (1) and can thus track experts according to their performances in the composite SEA model.

Our method is based on two key observations. Firstly, r ( · ) in (1) is fixed over time and available to the meta-algorithm from the beginning round. Secondly, the meta-algorithm can access expert

decisions for the current round before the performance estimation. Therefore, the information about r ( · ) can be introduced into the expert tracking process, securing that the regret of meta-algorithm will not deviate from that of the best expert. Specifically, inspired by Yan et al. [2023], we employ a two-layer meta-algorithm, with each layer running an MsMwC algorithm [Chen et al., 2021] that maintains two weight sequences { q t , ˆ q t ∈ ∆ N } t ∈ [ T ] , and a group of losses l t = ( l 1 t , · · · , l N t ) and optimisms m t = ( m 1 t , · · · , m N t ) to measure the performance of N experts. To handle the composite loss, we explicitly incorporate r ( · ) into both l t and m t . The incorporation serves two purposes: (i) it endows l t with a composite structure, facilitating the estimation of expert performance on composite losses; (ii) it utilizes the cancellation between l t and m t to eliminate the non-smooth component in loss functions, i.e., r ( · ) , so that the meta-algorithm is able to leverage the smoothness of f t ( · ) . In the following, we describe our universal algorithm, which is also summarized in Algorithm 2.

Meta-algorithm. Overall, our algorithm comprises two components: a two-layer meta-algorithm and a set of experts, together forming a three-layer structure. Specifically, on the top layer, we run one MsMwC named M top, which maintains the weights q ∈ ∆ K assigned to K = O (log T ) MsMwCs, named M mid. On the middle layer, each M k mid ( k ∈ [ K ] ) maintains the weights p k ∈ ∆ N assigned to N = O (log T ) experts. At the round t , we receive the decision x k,i t from each expert E k,i , and update the weights q t and p k t according to:

where ψ 1 ( q ) = ∑ k ( q k ln q k ) /η k and ψ 2 ( p ) = ∑ i ( p i ln p i ) /η k,i denote the negative entropy functions, and m k t and m t denote the optimism used by M k mid and M top, respectively. Then, we compute the weighted average decision x k t = ∑ i p k,i t x k,i t and submit x t = ∑ k q k t x k t . Next, we suffer the loss f t ( x t ) + r ( x t ) and observe the gradient g t = ∇ f t ( x t ) . We update ˆ q t +1 and ˆ p k t +1 by

where l t = ( l 1 t , · · · , l k t ) and l k t = ( l k, 1 t , · · · , l k,N t ) denote losses used by M top and M k mid , respectively, and a t ∈ R K and b k t ∈ R N denote the correction terms. At last, we send the gradient g t and the regularizer r ( · ) to each expert. Now, we specify the loss and optimism in (11) and (12). For M top , we choose

In the above, γ 1 , γ 2 denote hyperparameters, and γ 1 ‖ x k t -x k t -1 ‖ 2 2 and γ 2 ‖ x k,i t -x k,i t -1 ‖ 2 2 are injected for cancellation during the analysis [Yan et al., 2023]. It should be noticed that we explicitly introduce r ( · ) in (13) and (14). This is because experts are running over the composite losses and their performance assessment should similarly reflect the composite structure, ensuring an accurate measure for expert tracking. Moreover, the optimism m t with r ( · ) helps eliminate the non-smooth component in l t and thereby enables the meta-algorithm to effectively utilize the smoothness of time-varying functions. To make it clearer, we take the general convex case as an example.

First, the expected regret can be decomposed into the meta-regret and the expert-regret, as below:

where x k ∗ ,i ∗ t denotes the decision made by the best expert E k ∗ ,i ∗ . For the expert-regret, we can directly apply the result in Theorem 1. Therefore, the key is how to bound the meta-regret. By the fact that r ( x t ) ≤ ∑ k q k t r ( x k t ) and r ( x k t ) ≤ ∑ i p k,i t r ( x k,i t ) , the meta-regret is bounded by

where the first and second terms denote the regrets for tracking experts of M top and M k ∗ mid , respectively. While these two terms involve the losses l k t and l k ∗ ,i t that incorporate the regularizer, they can be simultaneously controlled by a favorable bound of O ( ∑ t ∈ [ T ] ( l k ∗ ,i ∗ t -m k ∗ ,i ∗ t ) 2 ) , which not only eliminates r ( · ) by the cancellation between l k ∗ ,i ∗ t and m k ∗ ,i ∗ t , but also suffices to achieve the desirable bound of ˜ O ( √ σ 2 1: T + √ Σ 2 1: T ) for general convex case and O (1) for other two cases. More details can be found in Appendix B.6.

Experts. Each expert is an instance of OptCMD that equips with certain configuration, i.e., (6), (8) or (10). To estimate the unknown curvature λ and α , we utilize the discretization strategy by Zhang et al. [2022], constructing the candidate sets as

where N = glyph[ceilingleft] log 2 T glyph[ceilingright] . Based on the two sets, we create following three types of experts:

Strongly convex experts, each of which is configured with (8) and a candidate λ i ∈ P str ;

Exp-concave experts, each of which is configured with (10) and a candidate α i ∈ P exp ;

General convex expert, which is configured with (6).

As demonstrated by Zhang et al. [2022], these experts are sufficient to identify the best one that runs for the true function type with the most accurate curvature. For instance, in the strongly convex case, there must exist an expert E i equipped with λ i ∈ P str that satisfies λ i ≤ λ ≤ 2 λ i . Moreover, instead of directly minimizing the original loss function (1), each expert runs over a new composite surrogate loss to avoid high-computational gradient query costs. Specifically, we choose h c t ( x ) = 〈 g t , x 〉 + r ( x ) , h sc t,i ( x ) = 〈 g t , x 〉 + r ( x ) + λ i ‖ x -x t ‖ 2 2 / 2 and h exp t,i ( x ) = 〈 g t , x 〉 + r ( x ) + β i 〈 g t , x -x t 〉 2 / 2 for general convex, strongly convex, and exp-concave experts, respectively. These surrogate losses not only inherit the composite structure and properties of f t ( x ) , e.g., the strongly convexity, but they also require only one gradient query for g t = ∇ f t ( x t ) in each round. The theoretical guarantees of our universal algorithm are presented below.

Theorem 6. Under Assumptions 1, 2, 3, 4 and 7, with proper hyperparameters, Algorithm 2 ensures the bounds of ˜ O ( √ σ 2 1: T + √ Σ 2 1: T ) , O (( σ 2 max +Σ 2 max ) log( σ 2 1: T +Σ 2 1: T )) , and O ( d log( σ 2 1: T +Σ 2 1: T )) for general convex, strongly convex, and exp-concave time-varying functions, respectively.

Remark. Theorem 6 indicates that Algorithm 2 can attain comparable regret bounds to those of OptCMD for the three cases, without knowing the function type and curvature in advance.

6 Implications

In the following, we discuss implications of our results for two common intermediate examples.

Composite adversarially corrupted stochastic data. We first focus on the composite adversarially corrupted stochastic model, of which a special case without regularizer has been widely investigated in learning with expert advice [Amir et al., 2020], bandits [Zimmert and Seldin, 2021, Ito, 2021], and online optimization [Sachs et al., 2022, Chen et al., 2023]. Specifically, each loss function consists of three terms: φ t ( · ) = h t ( · ) + c t ( · ) + r ( · ) , where h t ( · ) denotes the loss of i.i.d. data from a fixed distribution D with the variance σ , and c t ( · ) denotes an adversarial perturbation measured by a parameter C T > 0 , satisfying ∑ t ∈ [ T ] max x ∈X ‖∇ c t ( x ) ‖ 2 ≤ C T , and r ( · ) denotes the regularizer. In this model, the stochasticity and adversariality come from h t ( · ) and c t ( · ) , respectively. Therefore, σ 2 1: T and Σ 2 1: T can be formulated as

With the above specification, our theoretical results can naturally be extended to this scenarios, delivering the following corollary.

Corollary 1. With the specification (16) in the scenarios of composite adversarially corrupted stochastic data, we can obtain an O ( σ √ T + √ C T ) bound for the general convex case by Theorem 1, an O (log( σ 2 T + C T )) bound for the strongly convex case by Theorem 3, and an O ( d log( σ 2 T + C T )) bound for the exp-concave case by Theorem 5.

Adversarial online learning with limited resources. We then consider the another common scenarios where loss functions arrive in batches and the available computing resources are insufficient to process them all [Bottou and Cun, 2003, Benczúr et al., 2018, Li et al., 2024, Zhou, 2024]. Specifically, in each round t , we receive a group of functions F t = { f t ( · , i ) } i ∈ [ K t ] with the size K t , each of which is selected by the environments adversarially. We denote by F t ( · ) = K -1 t ∑ i ∈ [ K t ] f t ( · , i ) the average function. Due to limited computing resources, we can only sample a subset of functions for gradient estimation, and generate sparse decisions for efficient storage and inference. For this reason, our goal is to minimize φ t ( · ) = h t ( · ) + r ( · ) , where h t ( · ) = B -1 t ∑ i ∈ [ B t ] ˆ f t ( · , i ) denotes the approximation of F t ( · ) by i.i.d. sampling B t ∈ [ K t ] functions ˆ f t ( · , i ) from the group F t , and r ( · ) = ‖ · ‖ 1 denotes the glyph[lscript] 1 -norm regularizer. In this scenarios, it can be verified that

and Σ 2 1: T = E [ ∑ T t =1 sup x ∈X ‖∇ F t ( x ) -∇ F t -1 ( x ) ‖ 2 2 ] . Applying the above specification into our theorems delivers the following corollary.

Corollary 2. With the specification (17) in the scenarios of composite adversarially corrupted stochastic data, we can obtain an O ( √ ∑ t ∈ [ T ] B -1 t + √ Σ 2 1: T ) bound for the general convex case by Theorem 1, an O (log( ∑ t ∈ [ T ] B -1 t +Σ 2 1: T )) bound for the strongly convex case by Theorem 3, and an O ( d log( ∑ t ∈ [ T ] B -1 t +Σ 2 1: T )) bound for the exp-concave case by Theorem 5.

7 Conclusion and future work

In this paper, we investigate the intermediate setting between stochastic and adversarial composite optimization, named composite SEA, and demonstrate that OptCMD is able to attain the regret bounds of O ( √ σ 2 1: T + √ Σ 2 1: T ) , O (( σ 2 max +Σ 2 max ) log( σ 2 1: T +Σ 2 1: T )) and O ( d log( σ 2 1: T +Σ 2 1: T )) for the general convex, strongly convex and exp-concave cases, respectively. To deal with the unknown function type in real-world problems, we further propose a novel universal algorithm in online composite optimization, and show that our universal algorithm is able to achieve the desirable bounds in the three cases, simultaneously. Due to the versatility of σ 2 1: T and Σ 2 1: T , all our theoretical findings can recover previous results in fully stochastic and adversarial composite optimization. Finally, we explore several practical intermediate scenarios to demonstrate the implications of our results. Additionally, we also conduct empirical studies in Appendix A to verify our theoretical results.

There are many valuable directions for future research. First, our methods still rely on the domain and gradient bounded assumptions. We notice that there have been several methods designed for unbounded cases [Orabona, 2014, Orabona and Pál, 2016, Cutkosky and Orabona, 2018, Jacobsen and Cutkosky, 2022], but all of them focus on the setting without the reguralizer. Consequently, developing online algorithms with the unbounded domain and gradient for composite SEA remains an interesting research direction. Second, in the composite SEA model, we implicitly assume that the feedback (i.e., the loss function value and the gradient) is immediately revealed after making the decision, which, however, is not necessarily satisfied in practice [Joulani et al., 2013, Quanrud and Khashabi, 2015, Wan et al., 2022a,c, 2024b]. Therefore, it is also interesting to explore the composite SEA model with the delayed feedback. Thirdly, our proposed universal algorithm currently exhibits a three-layer structure, which presents several challenges in both analysis and practical implementation. Therefore, designing a simpler two-layer universal algorithm in composite SEA is another potential research direction in the future, which may require the advanced techniques in Yan et al. [2024].

Acknowledgments and Disclosure of Funding

The authors would thank Yu-Hu Yan for helpful discussions, and the anonymous reviewers for their constructive suggestions. This work was partially supported by National Science and Technology Major Project (2022ZD0114801), and NSFC (U23A20382, 62306275).

References

Amir, I. Attias, T. Koren, R. Livni, and Y . Mansour. Prediction with corrupted expert advice. In Advances in Neural Information Processing Systems 33 , pages 14315-14325, 2020.

Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters , 31(3):167-175, may 2003.

A. Benczúr, L. Kocsis, and R. Pálovics. Online machine learning in big data streams. ArXiv e-prints , arXiv:1802.05872, 2018.

Bottou and Y. L. Cun. Large scale online learning. In Advances in Neural Information Processing Systems 16 , pages 217-224, 2003.

Cesa-Bianchi, A. Conconi, and C. Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory , 50(9):2050-2057, 2004.

C.-C. Chang and C.-J. Lin. Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology , 2(27):1-27, 2011.

Chen, H. Luo, and C.-Y. Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Proceedings of the 34th Conference on Learning Theory , pages 1216-1259, 2021.

Chen, W.-W. Tu, P. Zhao, and L. Zhang. Optimistic online mirror descent for bridging stochastic and adversarial online convex optimization. In Proceedings of the 40th International Conference on Machine Learning , pages 5002-5035, 2023.

Chen, Y.-J. Zhang, W.-W. Tu, P. Zhao, and L. Zhang. Optimistic online mirror descent for bridging stochastic and adversarial online convex optimization. Journal of Machine Learning Research , 25 (178):1-62, 2024.

C.-K. Chiang, T. Yang, C.-J. Lee, M. Mahdavi, C.-J. Lu, R. Jin, and S. Zhu. Online optimization with gradual variations. In Proceedings of the 25th Conference on Learning Theory , pages 6.1-6.20, 2012.

Cutkosky. Parameter-free, dynamic, and strongly-adaptive online learning. In Proceedings of the 37th International Conference on Machine Learning , pages 2250-2259, 2020.

Cutkosky and F. Orabona. Black-box reductions for parameter-free online learning in banach spaces. In Proceedings of the 31st Conference on Learning Theory , pages 1493-1529, 2018.

Daniely, A. Gonen, and S. Shalev-Shwartz. Strongly adaptive online learning. In Proceedings of the 32nd International Conference on Machine Learning , pages 1405-1411, 2015.

Duchi and Y. Singer. Efficient online and batch learning using forward backward splitting. Journal of Machine Learning Research , 10:2899-2934, 2009.

C. Duchi, S. Shalev-Shwartz, Y. Singer, and A. Tewari. Composite objective mirror descent. In Proceedings of the 23rd Annual Conference on Learning Theory , pages 14-26, 2010.

Flammarion and F. Bach. Stochastic composite least-squares regression with convergence rate o (1 /n ) . In Proceedings of the 30th Conference on Learning Theory , pages 831-875, 2017.

Garber and A. Kaplan. Fast stochastic algorithms for low-rank and nonsmooth matrix problems. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics , pages 286-294, 2019.

Garber, G. Korcia, and K. Levy. Online convex optimization in the random order model. In Proceedings of the 37th International Conference on Machine Learning , pages 3387-3396, 2020.

Ghadimi and G. Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization I: A generic algorithmic framework. SIAM Journal on Optimization , 22 (4):1469-1492, 2012.

Hazan, A. Agarwal, and S. Kale. Logarithmic regret algorithms for online convex optimization. Foundations and Trends in Machine Learning , 69(2):169-192, 2007.

Hazan, T. Koren, and K. Y. Levy. Logistic regression: Tight bounds for stochastic and online optimization. In Proceedings of the 27th Conference on Learning Theory , pages 197-209, 2014.

Ito. On optimal robustness to adversarial corruption in online decision problems. In Advances in Neural Information Processing Systems 34 , pages 7409-7420, 2021.

Jacobsen and A. Cutkosky. Parameter-free mirror descent. In Proceedings of the 35th Conference on Learning Theory , pages 4160-4211, 2022.

Joulani, A. György, and C. Szepesvári. Online learning under delayed feedback. In Proceedings of the 30th International Conference on Machine Learning , pages 1453-1461, 2013.

Joulani, A. György, and C. Szepesvári. A modular analysis of adaptive (non-)convex optimization: Optimism, composite objectives, variance reduction, and variational bounds. Theoretical Computer Science , 808:108-138, 2020.

K.-S. Jun, F. Orabona, R. Willett, and S. Wright. Improved strongly adaptive online learning using coin betting. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , pages 943-951, 2017a.

K.-S. Jun, F. Orabona, S. Wright, and R. Willett. Online learning for changing environments using coin betting. Electronic Journal of Statistics , 11(2):5282-5310, 2017b.

Kulunchakov and J. Mairal. A generic acceleration framework for stochastic composite optimization. In Advances in Neural Information Processing Systems 32 , pages 12556-12567, 2019.

Lan. An optimal method for stochastic composite optimization. Mathematical Programming , 133: 365-397, 2012.

Lan. Gradient sliding for composite optimization. Mathematical Programming , 159:201-235, 2016.

Langford, L. Li, and T. Zhang. Sparse online learning via truncated gradient. Journal of Machine Learning Research , 10:777-801, 2009.

Lei and K. Tang. Stochastic composite mirror descent: Optimal bounds with high probabilities. In Advances in Neural Information Processing Systems 31 , pages 1519-1529, 2018.

Lei and D.-X. Zhou. Analysis of online composite mirror descent algorithm. Neural Computation , 29(3):825-860, 2017.

Lei, P. Yang, K. Tang, and D.-X. Zhou. Optimal stochastic and online learning with individual iterates. In Advances in Neural Information Processing Systems 32 , pages 5415-5425, 2019.

Li, T. Zhang, Z. Bu, S. Wang, H. He, J. Fu, Y. Wu, J. Bian, Y. Chen, and Y. Bengio. Map: Low-compute model merging with amortized pareto fronts via quadratic approximation. ArXiv e-prints , arXiv:2406.07529, 2024.

Mhammedi, W. M. Koolen, and T. van Erven. Lipschitz adaptivity with multiple learning rates in online learning. In Proceedings of the 32nd Conference on Learning Theory , pages 2490-2511, 2019.

Mohri and S. Yang. Accelerating online convex optimization via adaptive prediction. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , pages 848-856, 2016.

Nesterov. Primal-dual subgradient methods for convex problems. Mathematical Programming , 120(1):261-283, 2009.

Orabona. Simultaneous model selection and optimization through parameter-free stochastic learning. In Advances in Neural Information Processing Systems 27 , pages 1116-1124, 2014.

Orabona and D. Pál. Coin betting and parameter-free online learning. In Advances in Neural Information Processing Systems 29 , pages 577-585, 2016.

Pogodin and T. Lattimore. On first-order bounds, variance and gap-dependent bounds for adversarial bandits. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence , pages 894-904, 2019.

Quanrud and D. Khashabi. Online learning with adversarial delays. In Advances in Neural Information Processing Systems 28 , pages 1270-1278, 2015.

Rakhlin and K. Sridharan. Online learning with predictable sequences. In Proceedings of the 26th Conference on Learning Theory , pages 993-1019, 2013.

Sachs, H. Hadiji, T. van Erven, and C. Guzmán. Between stochastic and adversarial online convex optimization: Improved regret bounds via smoothness. In Advances in Neural Information Processing Systems 35 , pages 691-702, 2022.

Z. Scroccaro, A. S. Kolarijani, and P. M. Esfahani. Adaptive composite online optimization: Predictions in static and dynamic environments. IEEE Transactions on Automatic Control , 68(5): 2906-2921, 2023.

Sherman, T. Koren, and Y. Mansour. Optimal rates for random order online optimization. In Advances in Neural Information Processing Systems 34 , pages 2097-2108, 2021.

van Erven and W. M. Koolen. Metagrad: Multiple learning rates in online learning. In Advances in Neural Information Processing Systems 29 , pages 3666-3674, 2016.

van Erven, W. M. Koolen, and D. van der Hoeven. Metagrad: adaptation using multiple learning rates in online learning. Journal of Machine Learning Research , 22(161):1-61, 2021.

Wan, B. Xue, and L. Zhang. Projection-free online learning in dynamic environments. In Proceedings of the 35th AAAI Conference on Artificial Intelligence Advances , pages 10067-10075, 2021.

Wan, W.-W. Tu, and L. Zhang. Online frank-wolfe with arbitrary delays. In Advances in Neural Information Processing Systems 35 , pages 19703-19715, 2022a.

Wan, W.-W. Tu, and L. Zhang. Strongly adaptive online learning over partial intervals. Science China Information Sciences , 65(10):202101, 2022b.

Wan, Y. Wang, C. Yao, W.-W. Tu, and L. Zhang. Projection-free online learning with arbitrary delays. ArXiv e-prints , arXiv:2204.04964, 2022c.

Wan, C. Yao, M. Song, and L. Zhang. Non-stationary online convex optimization with arbitrary delays. In Proceedings of the 41st International Conference on Machine Learning , pages 4999150011, 2024a.

Wan, C. Yao, M. Song, and L. Zhang. Improved regret for bandit convex optimization with delayed feedback. In Advances in Neural Information Processing Systems 37 , 2024b.

Wang, S. Lu, and L. Zhang. Adaptivity and optimality: A universal algorithm for online convex optimization. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence , pages 659-668, 2019.

Wang, S. Lu, Y. Hu, and L. Zhang. Adapting to smoothness: A more universal algorithm for online convex optimization. In Proceedings of the 34th AAAI Conference on Artificial Intelligence , pages 6162-6169, 2020.

Wang, W. Yang, W. Jiang, S. Lu, B. Wang, H. Tang, Y. Wan, and L. Zhang. Non-stationary projection-free online learning with dynamic and adaptive regret guarantees. In Proceedings of the 38th AAAI Conference on Artificial Intelligence , pages 15671-15679, 2024.

Xiao. Dual averaging method for regularized stochastic learning and online optimization. In Advances in Neural Information Processing Systems 22 , pages 2116-2124, 2009.

Y.-H. Yan, P. Zhao, and Z.-H. Zhou. Universal online learning with gradual variations: A multi-layer online ensemble approach. In Advances in Neural Information Processing Systems 36 , pages 37682-37715, 2023.

Y.-H. Yan, P. Zhao, and Z.-H. Zhou. A simple and optimal approach for universal online learning with gradient variations. In Advances in Neural Information Processing Systems 37 , 2024.

Yang, M. Mahdavi, R. Jin, and S. Zhu. Regret bounded by gradual variation for online convex optimization. Machine Learning , 95:183-223, 2014.

Yang, W. Jiang, Y. Wang, P. Yang, Y. Hu, and L. Zhang. Small-loss adaptive regret for online convex optimization. In Proceedings of the 41st International Conference on Machine Learning , pages 56156-56195, 2024a.

Yang, Y. Wang, P. Zhao, and L. Zhang. Universal online convex optimization with 1 projection per round. In Advances in Neural Information Processing Systems 37 , 2024b.

Yang, P. Tian, X. Cheng, Y. Wan, and M. Song. Regularized online exponentially concave optimization. Neurocomputing , 595:127789, 2024c.

Zhang, S. Lu, and Z.-H. Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems 31 , pages 1323-1333, 2018.

Zhang, T. Yang, R. Jin, and Z.-H. Zhou. Relative error bound analysis for nuclear norm regularized matrix completion. Journal of Machine Learning Research , 20(97):1-22, 2019.

Zhang, S. Lu, and T. Yang. Minimizing dynamic regret and adaptive regret simultaneously. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics , pages 309-319, 2020.

Zhang, G. Wang, W.-W. Tu, W. Jiang, and Z.-H. Zhou. Dual adaptivity: A universal algorithm for minimizing the adaptive regret of convex functions. In Advances in Neural Information Processing Systems 34 , pages 24968-24980, 2021.

Zhang, G. Wang, J. Yi, and T. Yang. A simple yet universal strategy for online convex optimization. In Proceedings of the 39th International Conference on Machine Learning , pages 26605-26623, 2022.

Zhang, L. Zhang, Z. Jin, R. Jin, D. Cai, X. Li, R. Liang, and X. He. Sparse learning with stochastic composite optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(6): 1223-1236, 2017.

Z.-H. Zhou. Learnability with time-sharing computational resource concerns. National Science Review , 11(10):nwae204, 2024.

Zimmert and Y. Seldin. Tsallis-INF: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research , 22(28):1-49, 2021.

Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning , pages 928-936, 2003.

A Experiments

In this section, we conduct empirical studies to validate our theoretical results.

Setup. In this paper, we show that OptCMD with suitable configurations is able to achieve a series of favorable theoretical guarantees for the composite SEA model. Moreover, for the practical scenarios where the prior knowledge of loss functions is unavailable, we propose a novel universal strategy, called USC-SEA, which can still achieve the desired regret bounds for three cases in the composite SEA simultaneously. To verify our theoretical findings, we conduct experiments on the mushroom datasets from the LIBSVM repository [Chang and Lin, 2011], and consider the following online classification problem. Let T denote the number of the total rounds. At each round t ∈ [ T ] , the learner receives a sampled data ( x t , y t ) ∈ R d ×{-1 , 1 } with d = 112 . Then, the learner plays the decision w t from the ball X with the diameter D = 20 , and suffers a composite loss

where we set the hyper-parameter λ = 0 . 001 . The dataset used in the experiments is considered to be sampled from an unknown distribution, possessing the inherent stochastic property. To simulate the stochastically extended adversarial environments, we perturb the dataset by randomly flipping the labels of 10% data as the adversarial corruptions.

We consider the following three types of loss functions.

For the general convex case, we choose the smooth and convex cross-entropy function as the time-varying function:

where σ ( · ) denotes the sigmoid function, and utilize the glyph[lscript] 1 -norm regularizer r ( w t ) = ‖ w t ‖ 1 . Therefore, the composite function takes the form of:

For the strongly convex case, we employ the cross-entropy functions with the glyph[lscript] 2 -norm regularizer as the time-varying function:

which is 2 δ -strongly convex, and still leverage the glyph[lscript] 1 -norm regularizer. Hence, the composite loss function is in the form of:

where we set the hyper-parameter σ = 0 . 001 .

For the exp-concave case, we utilize the logistic function as the time-varying function:

which is exp-concave and smooth [Hazan et al., 2014], and still employ the glyph[lscript] 1 -norm regularizer. The composite loss function is shown below:

Contenders. For the general convex and strongly convex cases, we compare our methods with OGD [Zinkevich, 2003], COMID [Duchi et al., 2010] and Optimistic-OMD [Chen et al., 2023]. For the exp-concave case, we choose ONS [Hazan et al., 2007], ProxONS [Yang et al., 2024c] and Optimistic-OMD [Chen et al., 2023] as the contenders. All parameters of each method are set according to their theoretical suggestions. For instance, in the general convex case, the learning rate is set as η = ct -1 / 2 in OGD, and η = cT -1 / 2 in COMID, η t = D ( c + ¯ V t -1 ) -1 / 2 in Optimistic-OMD where c denotes the hyper-parameter selected from { 10 -3 , 10 -2 , · · · , 10 4 } .

Results. All experiments are repeated ten times, and we report the instantaneous loss, the cumulative loss and the average loss against the number of rounds in Figure 1 for the general convex case, Figure 2 for the strongly convex case and Figure 3 for the exp-concave case. From the experimental results, it is evident that adversarial corruptions cause considerable fluctuations in the instantaneous losses across different methods. Moreover, we also observe that, in the three cases of composite SEA, both OptCMD and USC-SEA suffer lower losses compared to baseline methods, demonstrating better performance. This phenomenon can be attributed to their ability to adapt to the composite SEA environment, and their explicit support for handling the non-smooth component r ( · ) .

0

1

2

3

4

5

Rounds

1e3

1

2

3

4

(a) Instantaneous Loss

0

1

2

3

4

5

Rounds

1e3

0

600

1200

1800

Cumulative Loss

(b) Cumulative Loss

0

1

2

3

4

5

Rounds

1e3

0.3

0.5

0.7

0.9

1.1

Average Loss

(c) Average Loss

OGD

COMID

Optimistic-OMD

OptCMD

USC-SEA

0

1

2

3

4

5

Rounds

1e3

1

2

3

4

5

(a) Instantaneous Loss

0

1

2

3

4

5

Rounds

1e3

700

1400

2100

2800

Cumulative Loss

(b) Cumulative Loss

0

1

2

3

4

5

Rounds

1e3

0.6

1.2

1.8

2.4

3.0

Average Loss

(c) Average Loss

OGD

COMID

Optimistic-OMD

OptCMD

USC-SEA

0

1

2

3

4

5

Rounds

1e3

1

2

3

4

(a) Instantaneous Loss

0

1

2

3

4

5

Rounds

1e3

400

800

1200

1600

Cumulative Loss

(b) Cumulative Loss

0

1

2

3

4

5

Rounds

1e3

0.2

0.4

0.6

0.8

1.0

Average Loss

(c) Average Loss

ONS

ProxONS

Optimistic-OMD

OptCMD

USC-SEA

B Theoretical analysis

B.1 Proof of Theorem 1

First, since the expected function F t ( · ) is convex in each round t , we could decompose the instantaneous regret as below:

The inequality is due the convexity of F t ( · ) and the last equality is due to the interchangeability of differentiation and integration by Leibniz integral rule. In the following, we shed the light on the right side of (18).

Then, we introduce the following lemma for OptCMD.

Lemma 1. Assume R t ( · ) is an γ -strongly convex function with respect to the norm ‖ · ‖ and denote by ‖ · ‖ ∗ the dual norm. According to the updating rule in (4) and (5) , for all x ∈ X , we have

In the general convex setting, we choose η t = min { D/ √ 1 + ¯ V t -1 , D/δ } , and ‖ · ‖ = 1 √ η t ‖ · ‖ 2 , and ‖ · ‖ ∗ = √ η t ‖ · ‖ 2 . Therefore, the corresponding Bregman divergence becomes B R t ( x , y ) = 1 2 η t ‖ x -y ‖ 2 2 with respect to the η -1 -strongly convex function R t ( x ) = 1 2 η t ‖ x ‖ 2 2 . Now, we make use of Lemma 1 with M t = ∇ f t -1 ( x t -1 ) , and obtain

Summing the above inequality over t = 1 , · · · , T , we have

Now, we have decomposed the upper bound into three terms and will analyze them separately.

To bound the term ( a ) , we introduce the following lemma.

Lemma 2. [Pogodin and Lattimore, 2019, Lemma 4.8] Let l 1 , · · · , l T be non-negative real numbers with the l t ∈ [0 , B ] . Then, we have

where for simplicity we define 0 / √ 0 = 0 .

By utilizing Lemma 2 and the fact that η t ≤ D/ √ 1 + ¯ V t -1 , the term ( a ) can be upper bounded as follows:

For the term ( b ) , a simple calculation delivers

For the term ( c ) , we utilize the fact that η t ≤ D/δ , and thus obtain

The last step is due to the domain bounded assumption, i.e., Assumption 1. Combining (21), (22) and (23), we obtain that

To bound the term ¯ V T , we exploiting the following lemma.

Lemma 3. Under Assumption 2 and 4, we have

Plugging (25) into (24) obtains

Then, we apply the AM-GM inequality, i.e., 6 HD √ ∑ T t =2 ‖ x t -x t -1 ‖ 2 2 ≤ 36 H 2 D 3 δ + δ 4 D ∑ T t =2 ‖ x t -x t -1 ‖ 2 2 , and obtain

By setting δ = 6 √ 2 HD , we have

where C 1 = 2( D + GD ) + 4 DG 2 +6 √ 2 HD 2 . Next, according to the definition of σ 2 1: T and Σ 2 1: T , the expected regret can be upper bound as follows:

B.2 Proof of Theorem 2

First, according to (18), we have

In this setting, we still choose ‖· ‖ = 1 √ η t ‖· ‖ 2 , and ‖· ‖ ∗ = √ η t ‖· ‖ 2 . Hence, the function R t ( x ) = 1 2 η t ‖ x ‖ 2 2 is η -1 t -strongly convex and the Bregman divergence becomes B R t ( x , y ) = 1 2 η t ‖ x -y ‖ 2 2 . Then, applying Lemma 1, we obtain

Following the configuration in Scroccaro et al. [2023], we set M t = ∇ f t -1 (ˆ x t ) and η t = (4 H 2 + ¯ D t ) -1 / 2 where ¯ D t = ∑ t s =1 ‖∇ f t (ˆ x t ) - ∇ f t -1 (ˆ x t ) ‖ 2 2 . According to [Scroccaro et al., 2023, Theorem 2.5], we have

To bound the term ¯ D T , we introduce the following lemma.

Lemma 4. Under Assumption 2, we have

By applying Lemma 4, we obtain

Substituting (27) into (26) and taking the expectation on both sides finishes the proof.

B.3 Proof of Theorem 3

In the beginning, we state the parameter configuration under the strongly convex setting (i.e., F t ( · ) is strongly convex and r ( · ) is general convex). The learning rate is set as η t = 2 δ + λt and the norm is chosen as ‖ · ‖ = 1 √ η t ‖ · ‖ 2 , with the dual norm ‖ · ‖ ∗ = √ η t ‖ · ‖ 2 . Therefore, the function becomes R t ( x ) = 1 2 η t ‖ x ‖ 2 2 and the corresponding Bregman divergence becomes B R t ( x , y ) = 1 2 η t ‖ x -y ‖ 2 2 .

Then, we make use of the strong convexity of F t ( · ) , and decompose the instantaneous regret as below:

Similar to the analysis in Theorem 1, we then focus on the the right side of (28). By utilizing Lemma 1, we have

Next, we analyze the above three terms separately. For the term ( a ) , we substitute η ≤ 2 λt and obtain

For the term ( b ) , we exploit one result of Lemma 1 in (52), i.e.,

with ‖ · ‖ = 1 √ η t ‖ · ‖ 2 and ‖ · ‖ ∗ = η t ‖ · ‖ 2 . By utilizing (30), we can upper bound the term ( b ) as following:

For the term ( c ) , we make use of the non-increasing property of η t and η t ≤ 2 /δ , and obtain

Putting (29), (31) and (32) together, we obtain

Now, we make use of a byproduct (55) from Lemma 3, i.e.,

and thus (33) becomes

To bound the first two terms, we introduce the following lemma.

Lemma 5. [Yan et al., 2023, Lemma 9] For a sequence of { a t } T t =1 and b , where a t , b > 0 for any t ∈ [ T ] , denoting by a max glyph[defines] max t a t and A glyph[defines] glyph[ceilingleft] b ∑ T t =1 a t glyph[ceilingright] , we have

Let a t = 2 σ 2 t + sup x ∈X ‖∇ F t ( x ) - ∇ F t -1 ( x ) ‖ 2 2 , a max = 2 σ 2 max + Σ 2 max , b = λ and A = glyph[ceilingleft] λ (2 σ 2 1: T +Σ 2 1: t ) glyph[ceilingright] . Then, applying Lemma 5, we obtain

Let a t = ‖ x t -x t -1 ‖ 2 2 , a max = D 2 , b = λ and A = glyph[ceilingleft] λ ∑ T t =1 ‖ x t -x t -1 ‖ 2 2 glyph[ceilingright] . Applying Lemma 5, we obtain

where the last step is due to ln(1 + x ) ≤ x for any x ≥ 0 .

Substituting (35) and (36) into (34), we have

Setting δ = 128 H 2 D 2 finishes the proof.

B.4 Proof of Theorem 4

Similar to (28) in Theorem 3, the instantaneous regret is upper bounded by

According to [Scroccaro et al., 2023, Theorem 2.9], OptCMD with the configuration in (9) ensures

For the term log ( 1 + λ 4 HG 2 ¯ D T ) , we apply Lemma 4 and obtain

Taking expectation over both sides of the above inequality delivers

Plug the above result into (37) finishes the proof.

B.5 Proof of Theorem 5

For the exp-concave setting (i.e., f t ( · ) is exp-concave and r ( · ) is general convex), our parameter configuration is follow Chiang et al. [2012]. To be precise, we define H t = δI + β 2 G 2 I + β 2 ∑ t -1 r =1 h r , where h r = ∇ f r ( x r ) ∇ f r ( x r ) glyph[latticetop] . The norm is set as ‖ · ‖ = ‖ · ‖ H t and ‖ · ‖ ∗ = ‖ · ‖ H -1 t , and the function becomes R t ( x ) = 1 2 ‖ x ‖ 2 H t . With this R t ( x ) , the corresponding Bregman divergence becomes B R t ( x , y ) = 1 2 ‖ x -y ‖ 2 H t .

First, we introduce a common property of exp-concave function, as shown below.

Lemma 6. [Hazan et al., 2007, Lemma 3] Under Assumption 1 and 2, if f ( · ) is exp-concave over X , we have

Then, according to the exponential concavity of f t ( · ) shown in Lemma 6, we can decompose the instantaneous regret as below:

Recall that we denote h t = ∇ f t ( x t ) ∇ f t ( x t ) glyph[latticetop] . Hence, the last term in (38) can be rewrite as 〈∇ f t ( x t ) , x ∗ -x t 〉 2 = ‖ x ∗ -x t ‖ 2 h t .

Now, we focus on the right side of (38). By utilizing Lemma 1, we have

The term ( a ) can be upper bounded as shown in the following lemma.

Lemma 7. Let d be the dimension. Then, we have

For the term ( b ) , we exploit the fact that H t +1 -H t = β 2 h t and obtain

To proceed the proof, we introduce the following lemma.

Lemma 8. [Chiang et al., 2012, Proposition 1] For any y, z ∈ R N and any PSD H ∈ R N × N , we have ‖ y + z ‖ 2 H ≤ 2 ‖ y ‖ 2 H +2 ‖ z ‖ 2 H .

Applying Lemma 8 on (39) obtains:

where the penultimate step is due to H t glyph[followsequal] β 2 G 2 I glyph[followsequal] β 2 h t , and the last step is due to

which could be obtained from Lemma 1 with ‖ · ‖ = ‖ · ‖ H t and ‖ · ‖ ∗ = ‖ · ‖ H -1 t .

For the term ( c ) , we have

where the first inequality is due to H t glyph[followsequal] H t -1 glyph[followsequal] δI , ‖ ˆ x T +1 -x T ‖ 2 H T +1 ≥ 0 and ‖ x 1 -ˆ x 1 ‖ 2 H 1 ≥ 0 . Combining (38), (40) and (42) obtains

Next, we leverage Lemma 3 and arrive at

where the last step is due to the inequality

To simplify (43), we make use of the following lemma.

Lemma 9. [Chen et al., 2023, Lemma 7] Let A ≥ 0 , a ≥ 0 , b ≥ 0 and c > 0 , we have

From Lemma 9, we have

where we set δ = 1 in the last step. Combining (43) and (45), we have

Taking the expectation over both sides delivers

where the last step is due to the Jensen's inequality. Finally, we complete the proof by substitute the above result into (38).

B.6 Proof of Theorem 6

In this part, we analyze three types of functions separately.

B.6.1 For convex functions

First, since the expected function F t ( · ) is convex in each round t , the instantaneous dynamic regret could be upper bounded as below:

Then, we decompose the regret into meta-regret and expert-regret:

Expert-regret analysis. According to the above decomposition, we define the surrogate loss functions h c t ( x ) = 〈∇ f t ( x t ) , x 〉 + r ( x ) for the i -th expert. According to (24), we can bound the expert-regret by

where the last step is due to Lemma 3.

Meta-regret analysis. We first consider the instantaneous meta-regret:

where the second step is due to Jensen's inequality, i.e., r ( x t ) = r ( ∑ K k =1 q k t x k t ) ≤ ∑ K k =1 q k t r ( x k t ) and r ( x k t ) = r ( ∑ N i =1 p k,i t x k,i t ) ≤ ∑ N i =1 p k,i t r ( x k,i t ) , and the third step is due to the definition of l k t and l k,i t in (13) and (14). For brevity, we denote q t glyph[defines] ( q 1 t , · · · , q t K ) , l t glyph[defines] ( l 1 t , · · · , l K t ) ,

Then, we introduce the following lemma, which is similar to Yan et al. [2023, Lemma 3] but with a different composite losses and optimisms in (13) and (14).

Lemma 10. Let l k t , m k t be defined in (13) , and l k,i t , m k,i t be defined in (14) . Then, we have

Note that

where the second and third steps are due to the fact that ( a + b ) 2 ≤ 2 a 2 + 2 b 2 for any a, b ∈ R . Taking the expectation on the both sides, we obtain

where the inequality is due to Lemma 3. Therefore, combining the above results with (47), Lemma 10 and Lemma 3 delivers

To bound the first two terms, we exploit the following lemma.

Lemma 11. [Yan et al., 2023, Lemma 7] For the step size pool H = { η 1 , · · · , η K } with η 1 = 1 2 C 0 ≥ · · · ≥ η T = 1 2 C 0 T , if C 0 ≥ √ X 2 T , there exists η k ∈ H satisfying

Therefore, we have

Hence, by requiring C 0 ≥ 1 , the meta-regret is bounded as:

meta-regret

Combining the expert-regret and meta-regret, we get

where C 1 = 128 D 2 H 2 +64 G 2 +10 DH and the last step is due to the following lemma. Lemma 12. Let x t = ∑ K k =1 q k t x k t . Then, we have

By setting C 0 ≥ 1 , C 0 ≥ 8 D 2 C 1 , γ 1 ≥ C 1 and δ ≥ 256 G 2 D +4 Dγ 2 , we finish the proof.

B.6.2 For strongly convex functions

According to (28), we could decompose the regret as below:

where the first step is due to the fact that there exists an λ i ∈ H satisfying λ i ≤ λ ≤ 2 λ i .

Expert-regret analysis. We define the surrogate loss functions h sc t,i ( x ) = 〈∇ f t ( x t ) , x 〉 + r ( x ) + λ i 2 ∑ T t =1 ‖ x -x t ‖ 2 2 for the i -th expert. According to Theorem 3, we can bound the expert-regret by expert-regret

Meta-regret analysis. Similar to (46), the instantaneous meta-regret can be bounded by

For the first two terms, we exploit Lemma 10 and obtain

Note that

where the first step is due to the fact that ( a + b ) 2 ≤ 2 a 2 +2 b 2 for any a, b ∈ R . Substituting (50) and (51) into (49) obtains meta-regret

where the last step is due to Lemma 12 and C 0 ≥ 1 . To bound the first term in the above result, we employ the following lemma.

Lemma 13. [Yan et al., 2023, Lemma 8] For the step size pool H = { η 1 , · · · , η K } with η 1 = 1 2 C 0 ≥ · · · ≥ η T = 1 2 C 0 T , if C 0 ≥ 1 2 η ∗ T where η ∗ is the optimal step size, there exists η k ∈ H satisfying

Applying the above lemma with η ∗ = λ i 256 G 2 delivers

Then, we combine the expert-regret and meta-regret:

where C 2 = 16 D 2 ( H 2 +1 ) 2 . Setting C 0 ≥ 4 D 2 C 2 , C 0 ≥ 32 D 2 γ 1 , δ ≥ 64 D 2 ( H 2 +1) + 8 γ 2 , γ 1 ≥ 2 C 2 and γ 2 ≥ γ 1 completes the proof.

B.6.3 For exp-concave functions

According to (38), we could decompose the regret as below:

where g t = ∇ f t ( x t ) , β i = 1 2 min { 1 4 GD , α i } , and the first step is due to the fact that there exists an α i ∈ H satisfying α i ≤ α ≤ 2 α i .

Expert-regret analysis. We define the surrogate loss functions h exp t,i ( x ) = 〈 g t , x 〉 + r ( x ) + β i 2 〈 g t , x -x t 〉 2 for the i -th expert. According to Theorem 5, we can bound the expert-regret by expert-regret

where C 3 = 8 H +64 D G H +8 G .

Meta-regret analysis. Similar to (46), the instantaneous meta-regret can be bounded by

For the first two terms, we exploit Lemma 10 and obtain

Note that

Therefore, the meta-regret is bounded by meta-regret

Applying Lemma 13 with η ∗ = β i 256 delivers

Then, we combine the expert-regret and meta-regret:

where C 3 = 8 H 2 + 64 D 2 G 2 H 2 + 8 G 4 . Setting C 0 ≥ 96 dDC 3 /δ , C 0 ≥ 32 D 2 γ 1 , δ ≥ 4 √ 6 d ( G 4 +4 C 3 ) , γ 1 ≥ 48 dC 3 /δ and γ 2 ≥ 2 γ 1 completes the proof.

C Supporting lemmas

C.1 Proof of Lemma 1

First, we introduce two lemmas that will be used in the following proof.

Lemma 14. [Scroccaro et al., 2023, Lemma 3.1] Let X be a convex set, ϕ ( · ) : X → R be a convex function and η > 0 . Define

Then, for any z ∈ X , we have

where g ( u ) ∈ ∂ϕ ( u ) .

Lemma 15. [Scroccaro et al., 2023, Lemma 3.2] Let X be a convex set, R : X → R be an α -strongly convex function with respect to the norm ‖ · ‖ . Define

Then, by exploiting the convexity of r ( · ) , we have

To upper bound the term ( a ) , we exploit Lemma 15 and obtain

Next, we apply Lemma 14 with the update (4) and (5), and obtain

Combining (52), (53) and (54) finishes the proof.

C.2 Proof of Lemma 3

First, we consider the case t = 1 . Under the Assumption 2, we obtain

In the case t ≥ 2 , according to the inequality ‖ a + b ‖ 2 2 ≤ 2 ‖ a ‖ 2 2 +2 ‖ b ‖ 2 2 , we have

Under the Assumption 4, the above result becomes

Then, we have

Combining both cases, we reach at

Summing up both sides of the above inequality over t = 1 , · · · , T obtains

which completes the proof.

C.3 Proof of Lemma 4

Similar to the analysis in Lemma 3, we first consider the case t = 1 .

Then, for the case t ≥ 2 , we have

Summing up both sides of the above inequality over t = 1 , · · · , T completes the proof.

C.4 Proof of Lemma 7

The analysis is based on Lemma 19 in Chiang et al. [2012]. Recall the definition H t = I + β 2 G 2 δI + β 2 ∑ t -1 r =1 h r with h r = ∇ f r ( x r ) ∇ f r ( x r ) glyph[latticetop] . Then, we can proof that

The first step is due to Assumption 2, and the last step is due to ∇ f 0 ( x 0 ) the all0 vector. Next, according to the fact that

we have

Substituting (57) into (56) delivers

For brevity, we denote K t = δI + β 8 ∑ t r =1 ( ∇ f r ( x r ) -∇ f r -1 ( x r -1 )) ( ∇ f r ( x r ) -∇ f r -1 ( x r -1 )) glyph[latticetop] . From (58), we obtain that H -1 t glyph[precedesequal] K -1 t , which implies

Then, we introduce the following lemma:

Lemma 16. [Hazan et al., 2007, Lemma 11] Let u 1 , · · · , u T ∈ R d be any sequence vectors and glyph[epsilon1] > 0 be a positive real number. Denote V t = glyph[epsilon1]I + ∑ t i =1 u i u glyph[latticetop] i , then we have

By setting glyph[epsilon1] = δ and u t = √ β 8 ( ∇ f t ( x t ) -∇ f t -1 ( x t -1 )) , we have

We complete the proof by combining (59) and (60).

C.5 Proof of Lemma 10

Our proof starts from the following lemma for single-layer MsMwC.

Lemma 17. [Yan et al., 2023, Lemma 2] If max t ∈ [ T ] ,i ∈ [ N ] | l i t | , | m i t | ≤ 1 , then MsMwC enjoys

It can be verified that under Assumption 1, 2 and 3, all our composite losses and optimisms defined in (13) and (14) are bounded

and we can rescale them to [ -1 , 1] with only constant multiplicative factors in the constant hyperparameter γ 1 and γ 2 . Therefore, applying Lemma 17 with the first layer(i.e., with l k t as the surrogate losses), we have

For the first two terms, the initialization ˆ q k 1 = ( η k ) 2 / ∑ K k =1 ( η k ) 2 and η k = 1 / ( C 0 2 k ) imply

For the last term, due to η k = 1 / ( C 0 2 k ) ≤ 2 /C 0 , we have

Substitute (62) and (63) into (61) delivers

Then, we apply Lemma 17 again with l k,i t and m k,i t .

With the initialization ˆ p k,i 1 = 1 /N and η k,i = 2 η k , we obtain

Therefore, (65) becomes

Combining (64) and (66), we obtain

According to (13) and (14), the last term is bounded by

where ˆ l k t = (〈 ∇ f t ( x t ) , x k, 1 t 〉 , · · · , 〈 ∇ f t ( x t ) , x k,N t 〉) and the last step is due to Cauchy-Schwarz inequality. Combining (67) and (68) completes the proof.

C.6 Proof of Lemma 12

where the second step is due to ( a + b ) 2 ≤ 2 a 2 +2 b 2 and the last step is due to the triangle inequality.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions have been clearly claimed in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations have been discussed in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the assumptions are listed in Section 3.1 and the complete proofs are provided in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided detailed descriptions for experiments in Section A that are sufficient to reproduce the experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: While the code is not included in the submission, the complete detailed descriptions of the experiments are provided in Section A.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided detailed descriptions for experiments in Section A that are sufficient to reproduce the experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments are repeated ten times with different random seeds, and the mean and standard deviation are reported.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: The experiments are conducted on a machine equipped with the Intel Xeon E5-2620 CPU and 32G memory.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have diligently adhered to the NeurIPS Code of Ethics in all aspects of this research.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is mainly theoretical and there is no societal impact of this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper is mainly theoretical and poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper is mainly theoretical and does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of online composite optimization under the Stochastically Extended Adversarial (SEA) model, where the environments select loss functions in a manner that interpolates between fully stochastic and fully adversarial scenarios. Specifically, the authors establish regret bounds for three kinds of loss functions and introduce a multi-level universal algorithm that can achieve these bounds simultaneously, without prior knowledge of the loss functions. These theoretical guarantees are interesting as they can not only recover previous findings for online composite optimization in both fully stochastic and adversarial environments but also lead to new results, including the first problem-dependent bound for exp-concave cases in the full adversarial setting.\n\nSTRENGTHS:\n1. The presentation is excellent, and the proof is readily comprehensible.\n\n2. The investigated problem of online composite optimization between stochastic and adversarial environments is significant, as it not only bridges two previously separate fields in online composite optimization but also extends the special case without the regularizer.\n\n3. The theoretical results are comprehensive and generalize previous findings in two extreme settings. Moreover, the authors clearly discuss the inadequacies of directly applying the existing work by Scroccaro et al. [2023] in the intermediate setting.\n\nWEAKNESSES:\nMy only concern is that this article appears to be purely theoretical at present and lacks empirical validation. I believe that some experimental studies could further strengthen the theoretical results.\n\nQUESTIONS:\nFor questions please refer to the weaknesses section.\n\nsome suggestions:\n\n1. I recommend relocating the Implications Section to follow the Universal Strategy Section. Such reorganization will enhance the logical flow and ensure the continuity of the narrative.\n\n2. Currently, this paper primarily focuses on static regret minimization. I am curious whether we can establish the more general dynamic regret bounds in the investigated setting. I believe that theoretical insights into dynamic regret would also be significant for the field.\n\nLIMITATIONS:\nThe authors discuss some limitations & future works in the Conclusion. And there is not potential negative societal impact, since this paper is purely theoretical.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates online composite optimization in the regime between stochastic and adversarial environments, establishing theoretical guarantees for three types of time-varying loss functions: smooth and general convex, smooth and strongly convex, and smooth and exp-concave. These bounds not only extend previous results in online composite optimization but also recover those in the special setting without the regularizer. Furthermore, the paper introduces a universal algorithm for online composite optimization that is agnostic to the type of functions and attains optimal bounds for all three cases simultaneously.\n\nSTRENGTHS:\n- The paper is well-organized, with a clear presentation of the background, motivations, and methods. The proofs are also easy to follow.\n- The investigation of the intermediate setting between fully stochastic and adversarial environments is an important topic in the community. This paper enriches this line of work, especially in online composite optimization. The regret bounds relate to both stochastic and adversarial aspects of the environments (i.e., $\\sigma_{1:T}^2$ and $\\Sigma_{1:T}^2$), allowing adaptation to different environments automatically.\n- The authors provide intermediate examples and explain the implications of their bounds in these cases, further highlighting the significance of their findings.\n\nWEAKNESSES:\n- Although the theoretical results are substantial, the paper lacks experimental validation. It would be more compelling if the authors conducted experiments to validate their theoretical results.\n- This paper appears to be an extension of [Sachs et al., 2022; Chen et al., 2023]. The authors should highlight their technical contributions more clearly.\n\nQUESTIONS:\n- The proposed algorithms in this paper are based on optimistic OMD. Is it possible to achieve similar conclusions using optimistic FTRL?\n- When the function $f_t(\\cdot)$ is non-smooth, can we still apply the proposed algorithms?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper investigates online composite optimization within the SEA model. Specifically, it demonstrates that by appropriately adjusting the predictor and step size of the algorithm known as OptCMD, similar regret bounds to those in online optimization within the SEA model can be achieved for general convex functions, strongly convex functions, and exp-concave functions. Additionally, the paper proposes extending the universal algorithm for online optimization by Yan et al. [2023] to online composite optimization.

STRENGTHS:
- This paper studies a problem setting considered important for practical applications.
- By making simple adjustments to existing methods, the desired theoretical results are obtained.
- The comparison with existing research is detailed, especially with Scroccaro et al. [2023].
- Overall, it is well-written and easy to follow.

WEAKNESSES:
- The techniques proposed in this paper do not offer significant novelty.

QUESTIONS:
- If the regularizer $r(\cdot)$ is time-dependent, can the same regret upper bound be achieved with the current analysis? Or are there parts of the analysis that would require non-trivial adjustments?

LIMITATIONS:
The limitations of this study are discussed in the conclusion.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
The authors study the online composite optimization under the Stochastically Extended Adversarial (SEA) model proposed by Sachs et al., 2022. They show that optimistic composite mirror descent (OptCMD), a variant of OMD by [Scroccaro et al., 2023], can achieve regret guarantees that match existing bounds for the SEA model without the regularizer. The bounds are derived for various classes of convex losses including convex functions, exp-concave functions, and strongly convex functions. Furthermore, the authors also propose a universal algorithm that achieves optimal bounds without requiring prior information of the function class.

STRENGTHS:
The paper is well-written and ideas are presented clearly. The authors extend the existing algorithm OMD by [Scroccaro et al., 2023] to the composite loss setting. The main novelty lies in the careful selection of the algorithm's parameters that lead to a good regret bound. I have not gone through the technical details very carefully but the results look interesting and extend the known results from noncomposite to composite loss setting.

WEAKNESSES:
While the results look nice, my main concern lies in evaluating the novelty of the paper. I would be interested in knowing the challenges faced by the authors to extend the existing analysis of OMD by [Scroccaro et al., 2023] to the composite setting. Especially, what technical challenges does the new setting throw beyond a different selection of algorithm's parameters? Do these challenges require the authors to come up with any technical innovation in their proofs?

QUESTIONS:
Please see the above.

LIMITATIONS:
Please see the above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-20