Optimal Rates for Random Order Online Optimization

We study online convex optimization in the random order model , recently proposed by Garber et al . [ 8 ] , where the loss functions may be chosen by an adversary , but are then presented to the online algorithm in a uniformly random order . Focusing on the scenario where the cumulative loss function is ( strongly ) convex , yet individual loss functions are smooth but might be non-convex , we give algorithms that achieve the optimal bounds and significantly outperform the results of Garber et al . [ 8 ] , completely removing the dimension dependence and improving their scaling with respect to the strong convexity parameter . Our analysis relies on novel connections between algorithmic stability and generalization for sampling without-replacement analogous to those studied in the with-replacement i.i.d . setting , as well as on a refined average stability analysis of stochastic gradient descent . 1 Introduction . Online convex optimization [ 25 , 12 ] studies the iterative process of decision making as data arrives in an online fashion . The model posits a game of ) rounds , where in each round the learner chooses a decision FC from a convex set , ⊆ ℝ3 , after which she observes a loss function 5C : , → ℝ , and incurs loss 5C ( FC ) . The learner ’ s objective is to minimize her regret , defined as her cumulative loss minus that of the best decision in hindsight F∗ = arg minF ∈ , ∑ ) C=1 5C ( F ) . In the prototypical setting , the individual loss functions are assumed to be convex and adversarially chosen by an opponent—commonly known as nature or the adversary—who has knowledge of the learner ’ s algorithm . While this setup is fundamental enough to accommodate a diverse set of applications ( see , e.g. , [ 12 ] ) , studying variants of the basic model promotes modeling flexibility , and further broadens the set of problems to which optimization techniques may be applied . Recently , Garber et al . [ 8 ] consider relaxing the convexity assumption by requiring that only on average the loss is ( strongly ) convex—a property the authors refer to as cumulative ( strong ) convexity—but do not require that the losses are convex individually . It is well known ( e.g. , [ 4 ] ) that under these assumptions , if the losses are sampled i.i.d . from some distribution , stochastic gradient descent ( SGD ) obtains the optimal $ ( log ) ) regret in expectation . However , as it turns out , in the fully adversarial model the cumulative strong convexity assumption is too weak : Garber et al . [ 8 ] show that in this case there is a linear regret lower bound . Consequently , they propose the random order model , where ) losses are chosen adversarially but then revealed to the learner in uniformly random order . Within this model , under the relaxed convexity assumption Garber et al . [ 8 ] obtain sub-linear regret for a number of specialized settings , differing in their assumptions on the structure of the individual loss functions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In the most general case ( the one we consider in this paper ) , they prove online gradient descent obtains regret $ ( ( 3 2/λ3 ) log ) ) w.h.p . for -Lipschitz λ-cumulative-strongly convex losses . It is informative to compare the random order model with the i.i.d . stochastic case , where on every round a new loss is sampled uniformly and independently from the set of losses , that is , with-replacement . By contrast , the random order model specifies that on every round a new loss is sampled uniformly without-replacement , an in particular not independently . Concretely , let L be an arbitrary set of ) smooth and Lipschitz continuous loss functions . Set ( F ) B 1 ) ∑ 5 ∈L 5 ( F ) , and assume is λ-strongly-convex over , . The learner ’ s goal is to minimize her regret on the loss sequence 51 , . . . , 5 ) obtained from a uniformly random ordering of L. As noted previously , if the losses 5C were drawn i.i.d. , SGD obtains the optimal $ ( log ) ) regret even though the losses are not individually convex . In a nutshell , when losses are i.i.d. , the gradients used in the SGD update are conditionally unbiased estimates of the gradient of the average ( strongly convex ) loss , hence the optimal regret is achieved in expectation . The difficulty in the random order model stems from the fact that random order gradients are not conditionally unbiased ; given any set of past losses 51 , . . . , 5C−1 , the next loss is uniform over the complement L \ { 51 , . . . , 5C−1 } , and thus ∇ 5C ( FC ) is biased . To overcome this complication , Garber et al . [ 8 ] work via the uniform convergence route , and build on concentration bounds applied to Hessians of the losses . As a result , they achieve suboptimal bounds , particularly in the general case where a dimension factor is introduced by a discretization argument necessary to ensure convergence over the entire domain . Here we choose a different strategy , and draw connections to notions of algorithmic stability and generalization studied in statistical learning theory . Our approach introduces significant improvements compared to prior work , and achieves regret bounds optimal up to additive factors . 1.1 Our results . We present and analyze two algorithms for random order online optimization , the first of which obtains the optimal regret up to additive factors . Let 51 , . . . , 5 ) be a random order sequence of -Lipschitz , β-smooth losses , where 5C : , → ℝ for all C ≤ ) . Assume the domain , ⊂ ℝ3 is convex , and has diameter bounded by . Further , assume the average loss 1 ) ∑ ) C=1 5C is λ-strongly convex . We prove ; Theorem ( informal ) . There exists an algorithm ( Algorithm 1 ) for random order online optimization that obtains regret of $ ( ( 2/λ ) log ) ) in expectation . Although the above result matches the optimal result for the individually strongly convex setting ( up to additive factors ) , Algorithm 1 requires memory linear in ) . This disadvantage motivates another algorithm , which trades off an extra factor of κ = β/λ in the regret for lower memory requirements . Theorem ( informal ) . There exists an algorithm ( Algorithm 2 ) for random order online optimization that requires memory linear in 3 , and obtains regret of $ ( ( β 2/λ2 ) log ) ) in expectation . Big- $ notation in both theorems hides additive factors polynomial in problem parameters β , and ( but does not hide multiplicative factors in these parameters ) . By comparison , Garber et al . [ 8 ] obtain a regret bound of $ ( ( 3 2/λ3 ) log ) ) w.h.p . for this setting . Both of our algorithms completely remove the dimension factor 3 , and reduce scaling w.r.t . the strong convexity parameter by a factor of 1/λ2 and 1/λ respectively . Moreover , their results require the losses to have a Lipschitz Hessian ( see 8 , Theorem A.3 ) , an assumption we do not make . In addition , we consider the case that is convex ( but not strongly convex ) , and apply our above results by means of regularization . As a corollary of the first theorem , we obtain for this setting regret that scales as $ ̃ ( √ ) ) , matching up to logarithmic and additive factors the optimal rate for the individually convex setting . Similarly , the second theorem implies a $ ̃ ( ) 2/3 ) regret algorithm which is also memory efficient for the convex case . Notably , a similar reduction applied to the results of Garber et al . [ 8 ] would yield a regret bound of $ ̃ ( 3 ) 3/4 ) . This highlights the significance of our improvement to the dependence on λ . Finally , we note it remains an open question to determine whether vanilla without-replacement SGD achieves optimal regret in the random-order model ( our results leave a multiplicative gap of order 1/λ between the upper and lower bounds ) . For the case of quadratic losses , however , using our techniques combined with a concentration argument it is possible to show that SGD indeed achieves optimal $ ( log ) /λ ) regret bound ( up to logarithmic terms ) . 1.2 Overview of techniques . Our approach builds on the observation that regret on a random order loss sequence may be expressed as the average generalization error w.r.t a without-replacement training sample . In light of this , we relate random order regret to a suitable notion of algorithmic stability , mimicking in a sense a well known argument previously employed in the context of i.i.d . sampled training sets [ 3 , 21 ] . At a high level , stability measures the sensitivity of an algorithm to small changes in its training set , and is a classical approach to proving generalization bounds [ 3 , 21 ] . More often than not , the particular notion used in practice is uniform stability , where sensitivity is measured w.r.t . the worst case small change in the training dataset ; max ( ⊆Z ( < ) ‖ 5 ( A ( ( ) − 5 ( A ( ( ′ ) ) ) ‖ . Our key insight is that while we can not hope for uniform stability as losses are not assumed to be convex individually , we may exploit strong convexity of the population loss to show SGD admits average stability ; E ( ∼Z ( < ) ‖ 5 ( A ( ( ) ) − 5 ( A ( ( ′ ) ) ‖ ≤ ² ( < ) . In particular , we prove the gradient update is contractive in expectation ; E ‖G−η∇ 5 ( G ) − ( H−η∇ 5 ( H ) ) ‖ E ‖G− H‖ under the cumulative strong convexity assumption . In turn , this yields a stability result which implies regret that scales with 1/C for the early rounds up to C ≈ ) /κ , where κ is the condition number of the problem . In short , as the game progresses the bias of the random order gradient estimates increases , and the gradient update becomes unstable . To overcome the loss of stability in later rounds , we devise a simple online sampling mechanism that generates i.i.d . uniform samples from a random order distribution , effectively ensuring unbiased gradient estimates throughout all ) rounds , and consequently optimal regret up to additive factors . Finally , our approach allows us to develop an analysis framework that naturally accommodates SGD based algorithms in the random order model , and in a broader sense establish stability of SGD in a new , relatively general setting . 1.3 Related work . Random-Order Online Optimization was proposed in the recent work of [ 8 ] ( where it is referred to as ROOCO ) , who establish an $ ( ( 3 2/λ3 ) log ) ) regret upper bound for -Lipschitz λ-cumulativestrongly convex losses . In the classical OCO setup [ 25 , 12 ] , under the assumption the losses are λ-strongly convex individually , it is well known the minimax regret scales as Θ ( ( 2/λ ) log ) ) , and that the lower bound also applies under the assumption the adversary is i.i.d . stochastic [ 14 , 13 ] . 1 In addition , it is well known ( see e.g. , [ 4 ] ) that the upper bound for the i.i.d . adversary is obtained in expectation by SGD also for non-convex losses , as long as the expected loss is strongly convex . In this work , we show that the same regret upper bound also holds for the random order adversary . Also relevant to our work is the study of stability and generalization [ 3 , 21 ] in modern learning theory , and in particular stability properties of SGD . Proving generalization bounds with stability arguments is a well known approach dating back at least to [ 19 , 5 , 6 ] . The specific notion of average stability in the i.i.d . setting we draw upon was defined in [ 21 ] , though many similar measures have appeared in the literature long before their work ( see [ 3 , 21 ] for an overview ) . The influential work of [ 11 ] gave the first generalization bounds for general forms of SGD with an analysis relying on the notion of uniform stability [ 3 ] . Since then , several works have used a similar approach to gain further insight into stability and generalization properties of SGD , e.g. , [ 15 , 7 , 2 ] . A related line of work [ 9 , 22 , 16 , 20 , 18 ] studies SGD without-replacement for solving finite-sum optimization problems , a setting commonly encountered in offline machine learning applications . Here , multiple epochs of SGD are executed over a given training set , with the objective to produce a single output ( approximately ) minimizing the average loss . However , the majority of the results are obtained under the ( vastly simplifying ) assumption that the individual loss functions are convex , and therefore do not apply in our setting . In addition , the performance metric of interest is convergence rate , and not regret which is the focus of our paper . In particular , Nagaraj et al . [ 16 ] employ the method of exchangeable pairs to relate the average and random order loss to a stability-like property , and obtain optimal ( up to polylogarithmic factors ) convergence rate for a single epoch , albeit only for individually convex loss functions . Recently , a number of papers study SGD without-replacement and attempt to relax the convexity assumption [ 10 , 17 , 1 ] , but the bounds they obtain are under conditions inapplicable for our setting . 1The lower bound given in Hazan and Kale [ 13 ] uses quadratic ( hence , smooth ) losses , and is easily extensible to without-replacement sampling , and therefore applies in our setting . Specifically , they impose a requirement that the number of epochs passes a certain threshold strictly larger than one . Moreover , state-of-the-art bounds achieved by [ 1 ] are suboptimal w.r.t ours even had we ignored the epoch requirement .

Summary: The paper studies the problem of random order online convex optimization introduced previously by Garber et. al in 2020. Random order online convex optimization is a variant of standard online convex optimization where the loss functions may be chosen by an adversary (and do not even need to be individually convex), but the order in which they arrive is uniformly random.  This modification allows one to obtain efficient algorithms with stronger guarantees than for true worst-case analysis.    This paper improves upon the guarantees obtained by Garber et. al, getting nearly optimal rates in the general setting that essentially match offline/stochastic rates.  They remove a factor of $d$ and a factor of $1/\lambda^2$ (where $\lambda$ is the strong convexity parameter) from the regret guarantee.  However, their optimal algorithm requires remembering all previous observations (so $O(dT)$ memory).  Thus, the authors also give an efficient algorithm that requires only $O(d)$ memory but sacrifices a $1/\lambda$ factor in the regret guarantee (which is still better than the result in Garber et. al).   Roughly, the authors draw a parallel to stochastic online convex optimization.  The difference in the random order model is that the observations are not  independent but are instead sampled from a set \textit{without replacement}.  However, the authors are able to modify  techniques from the stochastic case to this without replacement setting.