Optimal Rates for Random Order Online Optimization

We study online convex optimization in the random order model , recently proposed by Garber et al . [ 8 ] , where the loss functions may be chosen by an adversary , but are then presented to the online algorithm in a uniformly random order . Focusing on the scenario where the cumulative loss function is ( strongly ) convex , yet individual loss functions are smooth but might be non-convex , we give algorithms that achieve the optimal bounds and significantly outperform the results of Garber et al . [ 8 ] , completely removing the dimension dependence and improving their scaling with respect to the strong convexity parameter . Our analysis relies on novel connections between algorithmic stability and generalization for sampling without-replacement analogous to those studied in the with-replacement i.i.d . setting , as well as on a refined average stability analysis of stochastic gradient descent . 1 Introduction . Online convex optimization [ 25 , 12 ] studies the iterative process of decision making as data arrives in an online fashion . The model posits a game of ) rounds , where in each round the learner chooses a decision FC from a convex set , ⊆ ℝ3 , after which she observes a loss function 5C : , → ℝ , and incurs loss 5C ( FC ) . The learner ’ s objective is to minimize her regret , defined as her cumulative loss minus that of the best decision in hindsight F∗ = arg minF ∈ , ∑ ) C=1 5C ( F ) . In the prototypical setting , the individual loss functions are assumed to be convex and adversarially chosen by an opponent—commonly known as nature or the adversary—who has knowledge of the learner ’ s algorithm . While this setup is fundamental enough to accommodate a diverse set of applications ( see , e.g. , [ 12 ] ) , studying variants of the basic model promotes modeling flexibility , and further broadens the set of problems to which optimization techniques may be applied . Recently , Garber et al . [ 8 ] consider relaxing the convexity assumption by requiring that only on average the loss is ( strongly ) convex—a property the authors refer to as cumulative ( strong ) convexity—but do not require that the losses are convex individually . It is well known ( e.g. , [ 4 ] ) that under these assumptions , if the losses are sampled i.i.d . from some distribution , stochastic gradient descent ( SGD ) obtains the optimal $ ( log ) ) regret in expectation . However , as it turns out , in the fully adversarial model the cumulative strong convexity assumption is too weak : Garber et al . [ 8 ] show that in this case there is a linear regret lower bound . Consequently , they propose the random order model , where ) losses are chosen adversarially but then revealed to the learner in uniformly random order . Within this model , under the relaxed convexity assumption Garber et al . [ 8 ] obtain sub-linear regret for a number of specialized settings , differing in their assumptions on the structure of the individual loss functions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In the most general case ( the one we consider in this paper ) , they prove online gradient descent obtains regret $ ( ( 3 2/λ3 ) log ) ) w.h.p . for -Lipschitz λ-cumulative-strongly convex losses . It is informative to compare the random order model with the i.i.d . stochastic case , where on every round a new loss is sampled uniformly and independently from the set of losses , that is , with-replacement . By contrast , the random order model specifies that on every round a new loss is sampled uniformly without-replacement , an in particular not independently . Concretely , let L be an arbitrary set of ) smooth and Lipschitz continuous loss functions . Set ( F ) B 1 ) ∑ 5 ∈L 5 ( F ) , and assume is λ-strongly-convex over , . The learner ’ s goal is to minimize her regret on the loss sequence 51 , . . . , 5 ) obtained from a uniformly random ordering of L. As noted previously , if the losses 5C were drawn i.i.d. , SGD obtains the optimal $ ( log ) ) regret even though the losses are not individually convex . In a nutshell , when losses are i.i.d. , the gradients used in the SGD update are conditionally unbiased estimates of the gradient of the average ( strongly convex ) loss , hence the optimal regret is achieved in expectation . The difficulty in the random order model stems from the fact that random order gradients are not conditionally unbiased ; given any set of past losses 51 , . . . , 5C−1 , the next loss is uniform over the complement L \ { 51 , . . . , 5C−1 } , and thus ∇ 5C ( FC ) is biased . To overcome this complication , Garber et al . [ 8 ] work via the uniform convergence route , and build on concentration bounds applied to Hessians of the losses . As a result , they achieve suboptimal bounds , particularly in the general case where a dimension factor is introduced by a discretization argument necessary to ensure convergence over the entire domain . Here we choose a different strategy , and draw connections to notions of algorithmic stability and generalization studied in statistical learning theory . Our approach introduces significant improvements compared to prior work , and achieves regret bounds optimal up to additive factors . 1.1 Our results . We present and analyze two algorithms for random order online optimization , the first of which obtains the optimal regret up to additive factors . Let 51 , . . . , 5 ) be a random order sequence of -Lipschitz , β-smooth losses , where 5C : , → ℝ for all C ≤ ) . Assume the domain , ⊂ ℝ3 is convex , and has diameter bounded by . Further , assume the average loss 1 ) ∑ ) C=1 5C is λ-strongly convex . We prove ; Theorem ( informal ) . There exists an algorithm ( Algorithm 1 ) for random order online optimization that obtains regret of $ ( ( 2/λ ) log ) ) in expectation . Although the above result matches the optimal result for the individually strongly convex setting ( up to additive factors ) , Algorithm 1 requires memory linear in ) . This disadvantage motivates another algorithm , which trades off an extra factor of κ = β/λ in the regret for lower memory requirements . Theorem ( informal ) . There exists an algorithm ( Algorithm 2 ) for random order online optimization that requires memory linear in 3 , and obtains regret of $ ( ( β 2/λ2 ) log ) ) in expectation . Big- $ notation in both theorems hides additive factors polynomial in problem parameters β , and ( but does not hide multiplicative factors in these parameters ) . By comparison , Garber et al . [ 8 ] obtain a regret bound of $ ( ( 3 2/λ3 ) log ) ) w.h.p . for this setting . Both of our algorithms completely remove the dimension factor 3 , and reduce scaling w.r.t . the strong convexity parameter by a factor of 1/λ2 and 1/λ respectively . Moreover , their results require the losses to have a Lipschitz Hessian ( see 8 , Theorem A.3 ) , an assumption we do not make . In addition , we consider the case that is convex ( but not strongly convex ) , and apply our above results by means of regularization . As a corollary of the first theorem , we obtain for this setting regret that scales as $ ̃ ( √ ) ) , matching up to logarithmic and additive factors the optimal rate for the individually convex setting . Similarly , the second theorem implies a $ ̃ ( ) 2/3 ) regret algorithm which is also memory efficient for the convex case . Notably , a similar reduction applied to the results of Garber et al . [ 8 ] would yield a regret bound of $ ̃ ( 3 ) 3/4 ) . This highlights the significance of our improvement to the dependence on λ . Finally , we note it remains an open question to determine whether vanilla without-replacement SGD achieves optimal regret in the random-order model ( our results leave a multiplicative gap of order 1/λ between the upper and lower bounds ) . For the case of quadratic losses , however , using our techniques combined with a concentration argument it is possible to show that SGD indeed achieves optimal $ ( log ) /λ ) regret bound ( up to logarithmic terms ) . 1.2 Overview of techniques . Our approach builds on the observation that regret on a random order loss sequence may be expressed as the average generalization error w.r.t a without-replacement training sample . In light of this , we relate random order regret to a suitable notion of algorithmic stability , mimicking in a sense a well known argument previously employed in the context of i.i.d . sampled training sets [ 3 , 21 ] . At a high level , stability measures the sensitivity of an algorithm to small changes in its training set , and is a classical approach to proving generalization bounds [ 3 , 21 ] . More often than not , the particular notion used in practice is uniform stability , where sensitivity is measured w.r.t . the worst case small change in the training dataset ; max ( ⊆Z ( < ) ‖ 5 ( A ( ( ) − 5 ( A ( ( ′ ) ) ) ‖ . Our key insight is that while we can not hope for uniform stability as losses are not assumed to be convex individually , we may exploit strong convexity of the population loss to show SGD admits average stability ; E ( ∼Z ( < ) ‖ 5 ( A ( ( ) ) − 5 ( A ( ( ′ ) ) ‖ ≤ ² ( < ) . In particular , we prove the gradient update is contractive in expectation ; E ‖G−η∇ 5 ( G ) − ( H−η∇ 5 ( H ) ) ‖ E ‖G− H‖ under the cumulative strong convexity assumption . In turn , this yields a stability result which implies regret that scales with 1/C for the early rounds up to C ≈ ) /κ , where κ is the condition number of the problem . In short , as the game progresses the bias of the random order gradient estimates increases , and the gradient update becomes unstable . To overcome the loss of stability in later rounds , we devise a simple online sampling mechanism that generates i.i.d . uniform samples from a random order distribution , effectively ensuring unbiased gradient estimates throughout all ) rounds , and consequently optimal regret up to additive factors . Finally , our approach allows us to develop an analysis framework that naturally accommodates SGD based algorithms in the random order model , and in a broader sense establish stability of SGD in a new , relatively general setting . 1.3 Related work . Random-Order Online Optimization was proposed in the recent work of [ 8 ] ( where it is referred to as ROOCO ) , who establish an $ ( ( 3 2/λ3 ) log ) ) regret upper bound for -Lipschitz λ-cumulativestrongly convex losses . In the classical OCO setup [ 25 , 12 ] , under the assumption the losses are λ-strongly convex individually , it is well known the minimax regret scales as Θ ( ( 2/λ ) log ) ) , and that the lower bound also applies under the assumption the adversary is i.i.d . stochastic [ 14 , 13 ] . 1 In addition , it is well known ( see e.g. , [ 4 ] ) that the upper bound for the i.i.d . adversary is obtained in expectation by SGD also for non-convex losses , as long as the expected loss is strongly convex . In this work , we show that the same regret upper bound also holds for the random order adversary . Also relevant to our work is the study of stability and generalization [ 3 , 21 ] in modern learning theory , and in particular stability properties of SGD . Proving generalization bounds with stability arguments is a well known approach dating back at least to [ 19 , 5 , 6 ] . The specific notion of average stability in the i.i.d . setting we draw upon was defined in [ 21 ] , though many similar measures have appeared in the literature long before their work ( see [ 3 , 21 ] for an overview ) . The influential work of [ 11 ] gave the first generalization bounds for general forms of SGD with an analysis relying on the notion of uniform stability [ 3 ] . Since then , several works have used a similar approach to gain further insight into stability and generalization properties of SGD , e.g. , [ 15 , 7 , 2 ] . A related line of work [ 9 , 22 , 16 , 20 , 18 ] studies SGD without-replacement for solving finite-sum optimization problems , a setting commonly encountered in offline machine learning applications . Here , multiple epochs of SGD are executed over a given training set , with the objective to produce a single output ( approximately ) minimizing the average loss . However , the majority of the results are obtained under the ( vastly simplifying ) assumption that the individual loss functions are convex , and therefore do not apply in our setting . In addition , the performance metric of interest is convergence rate , and not regret which is the focus of our paper . In particular , Nagaraj et al . [ 16 ] employ the method of exchangeable pairs to relate the average and random order loss to a stability-like property , and obtain optimal ( up to polylogarithmic factors ) convergence rate for a single epoch , albeit only for individually convex loss functions . Recently , a number of papers study SGD without-replacement and attempt to relax the convexity assumption [ 10 , 17 , 1 ] , but the bounds they obtain are under conditions inapplicable for our setting . 1The lower bound given in Hazan and Kale [ 13 ] uses quadratic ( hence , smooth ) losses , and is easily extensible to without-replacement sampling , and therefore applies in our setting . Specifically , they impose a requirement that the number of epochs passes a certain threshold strictly larger than one . Moreover , state-of-the-art bounds achieved by [ 1 ] are suboptimal w.r.t ours even had we ignored the epoch requirement .

Summary: Problem description: The paper studies the problem of online convex optimization in the random order model. This problem was recently proposed by Garber et al. (ICML 2020). It is a generalization of online convex optimization, where at each step the learner incurs a loss function $f_t$ and the goal is to minimize the cumulative loss $\sum_{t=1}^{T} f_t$. The authors inspired by Garber et al. (2020), considered the scenario in which each $f_t$ is smooth and Lipschitz but not necessarily (strong) convex. However, they imposed this assumption that $\sum_{t=1}^{T} f_t$ is (strongly) convex. Garber et al. (2020) assumed that at each step $t$, a new loss function $f_t$ is sampled uniformly without replacement from $\{f_t\}_t=1^T$, and they call this setting random order online optimization. The same setting is peruse by the authors of this paper and they improved the results in Garber et al. (2020), under the same assumptions.  Assumptions: 1. $f_t$ is Lipschitz  2. $f_t$ is smooth 3. $\sum_{t=1}^{T} f_t$ is $\lambda$-strongly convex.  Approach: Their main idea is to use the stability properties of stochastic gradient descent, see Shalev-Shwartz et al. (JMLR 2010). The authors defined a new notion of stability (page 5, line 176). This new definition of stability is suitable for sampling without replacement, which is beneficial for random order online optimization. Note that relevant definitions of stability Shalev-Shwartz et al. (2010), works only for i.i.d sampling.   By this conviction, they proposed Algorithm 1 (page 7) which is inspired by Vitter (1985). They proved the stability property of Algorithm 1 in Lemma 6 (page 7), which implies their main result in Theorem 1 (page 8). Since Algorithm 1 requires an intense memory usage, they proposed Algorithm 2 (page 9), which is more economical in that regard. Theorem 2 (page 9) uses Algorithm 2 but with a worse dependency w.r.t $\lambda$ (strong convexity parameter), compare to Theorem 1.  Contributions: With a different and a novel approach, they studied the problem proposed by Garber et al. (2020) and they could improve the results which I summarize as follows:  1. The upper bounds in Garber et al. (2020) have a linear dependency with respect to the dimension of $f_t$. They could remove this dependency and the rates are dimension free.  2. In Theorem 1 (page 8) they improved the results in Garber et al. (2020) up to a factor $1/\lambda^2$  3. In Theorem 1 (page 8) they improved the results in Garber et al. (2020) up to a factor $1/\lambda$  4. As a corollary (page 9) of Theorems 1 and 2, they could obtain similar results for the case when $\sum_{t=1}^{T} f_t$ is convex but not necessarily strongly convex.