Optimal Rates for Random Order Online Optimization

We study online convex optimization in the random order model , recently proposed by Garber et al . [ 8 ] , where the loss functions may be chosen by an adversary , but are then presented to the online algorithm in a uniformly random order . Focusing on the scenario where the cumulative loss function is ( strongly ) convex , yet individual loss functions are smooth but might be non-convex , we give algorithms that achieve the optimal bounds and significantly outperform the results of Garber et al . [ 8 ] , completely removing the dimension dependence and improving their scaling with respect to the strong convexity parameter . Our analysis relies on novel connections between algorithmic stability and generalization for sampling without-replacement analogous to those studied in the with-replacement i.i.d . setting , as well as on a refined average stability analysis of stochastic gradient descent . 1 Introduction . Online convex optimization [ 25 , 12 ] studies the iterative process of decision making as data arrives in an online fashion . The model posits a game of ) rounds , where in each round the learner chooses a decision FC from a convex set , ⊆ ℝ3 , after which she observes a loss function 5C : , → ℝ , and incurs loss 5C ( FC ) . The learner ’ s objective is to minimize her regret , defined as her cumulative loss minus that of the best decision in hindsight F∗ = arg minF ∈ , ∑ ) C=1 5C ( F ) . In the prototypical setting , the individual loss functions are assumed to be convex and adversarially chosen by an opponent—commonly known as nature or the adversary—who has knowledge of the learner ’ s algorithm . While this setup is fundamental enough to accommodate a diverse set of applications ( see , e.g. , [ 12 ] ) , studying variants of the basic model promotes modeling flexibility , and further broadens the set of problems to which optimization techniques may be applied . Recently , Garber et al . [ 8 ] consider relaxing the convexity assumption by requiring that only on average the loss is ( strongly ) convex—a property the authors refer to as cumulative ( strong ) convexity—but do not require that the losses are convex individually . It is well known ( e.g. , [ 4 ] ) that under these assumptions , if the losses are sampled i.i.d . from some distribution , stochastic gradient descent ( SGD ) obtains the optimal $ ( log ) ) regret in expectation . However , as it turns out , in the fully adversarial model the cumulative strong convexity assumption is too weak : Garber et al . [ 8 ] show that in this case there is a linear regret lower bound . Consequently , they propose the random order model , where ) losses are chosen adversarially but then revealed to the learner in uniformly random order . Within this model , under the relaxed convexity assumption Garber et al . [ 8 ] obtain sub-linear regret for a number of specialized settings , differing in their assumptions on the structure of the individual loss functions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In the most general case ( the one we consider in this paper ) , they prove online gradient descent obtains regret $ ( ( 3 2/λ3 ) log ) ) w.h.p . for -Lipschitz λ-cumulative-strongly convex losses . It is informative to compare the random order model with the i.i.d . stochastic case , where on every round a new loss is sampled uniformly and independently from the set of losses , that is , with-replacement . By contrast , the random order model specifies that on every round a new loss is sampled uniformly without-replacement , an in particular not independently . Concretely , let L be an arbitrary set of ) smooth and Lipschitz continuous loss functions . Set ( F ) B 1 ) ∑ 5 ∈L 5 ( F ) , and assume is λ-strongly-convex over , . The learner ’ s goal is to minimize her regret on the loss sequence 51 , . . . , 5 ) obtained from a uniformly random ordering of L. As noted previously , if the losses 5C were drawn i.i.d. , SGD obtains the optimal $ ( log ) ) regret even though the losses are not individually convex . In a nutshell , when losses are i.i.d. , the gradients used in the SGD update are conditionally unbiased estimates of the gradient of the average ( strongly convex ) loss , hence the optimal regret is achieved in expectation . The difficulty in the random order model stems from the fact that random order gradients are not conditionally unbiased ; given any set of past losses 51 , . . . , 5C−1 , the next loss is uniform over the complement L \ { 51 , . . . , 5C−1 } , and thus ∇ 5C ( FC ) is biased . To overcome this complication , Garber et al . [ 8 ] work via the uniform convergence route , and build on concentration bounds applied to Hessians of the losses . As a result , they achieve suboptimal bounds , particularly in the general case where a dimension factor is introduced by a discretization argument necessary to ensure convergence over the entire domain . Here we choose a different strategy , and draw connections to notions of algorithmic stability and generalization studied in statistical learning theory . Our approach introduces significant improvements compared to prior work , and achieves regret bounds optimal up to additive factors . 1.1 Our results . We present and analyze two algorithms for random order online optimization , the first of which obtains the optimal regret up to additive factors . Let 51 , . . . , 5 ) be a random order sequence of -Lipschitz , β-smooth losses , where 5C : , → ℝ for all C ≤ ) . Assume the domain , ⊂ ℝ3 is convex , and has diameter bounded by . Further , assume the average loss 1 ) ∑ ) C=1 5C is λ-strongly convex . We prove ; Theorem ( informal ) . There exists an algorithm ( Algorithm 1 ) for random order online optimization that obtains regret of $ ( ( 2/λ ) log ) ) in expectation . Although the above result matches the optimal result for the individually strongly convex setting ( up to additive factors ) , Algorithm 1 requires memory linear in ) . This disadvantage motivates another algorithm , which trades off an extra factor of κ = β/λ in the regret for lower memory requirements . Theorem ( informal ) . There exists an algorithm ( Algorithm 2 ) for random order online optimization that requires memory linear in 3 , and obtains regret of $ ( ( β 2/λ2 ) log ) ) in expectation . Big- $ notation in both theorems hides additive factors polynomial in problem parameters β , and ( but does not hide multiplicative factors in these parameters ) . By comparison , Garber et al . [ 8 ] obtain a regret bound of $ ( ( 3 2/λ3 ) log ) ) w.h.p . for this setting . Both of our algorithms completely remove the dimension factor 3 , and reduce scaling w.r.t . the strong convexity parameter by a factor of 1/λ2 and 1/λ respectively . Moreover , their results require the losses to have a Lipschitz Hessian ( see 8 , Theorem A.3 ) , an assumption we do not make . In addition , we consider the case that is convex ( but not strongly convex ) , and apply our above results by means of regularization . As a corollary of the first theorem , we obtain for this setting regret that scales as $ ̃ ( √ ) ) , matching up to logarithmic and additive factors the optimal rate for the individually convex setting . Similarly , the second theorem implies a $ ̃ ( ) 2/3 ) regret algorithm which is also memory efficient for the convex case . Notably , a similar reduction applied to the results of Garber et al . [ 8 ] would yield a regret bound of $ ̃ ( 3 ) 3/4 ) . This highlights the significance of our improvement to the dependence on λ . Finally , we note it remains an open question to determine whether vanilla without-replacement SGD achieves optimal regret in the random-order model ( our results leave a multiplicative gap of order 1/λ between the upper and lower bounds ) . For the case of quadratic losses , however , using our techniques combined with a concentration argument it is possible to show that SGD indeed achieves optimal $ ( log ) /λ ) regret bound ( up to logarithmic terms ) . 1.2 Overview of techniques . Our approach builds on the observation that regret on a random order loss sequence may be expressed as the average generalization error w.r.t a without-replacement training sample . In light of this , we relate random order regret to a suitable notion of algorithmic stability , mimicking in a sense a well known argument previously employed in the context of i.i.d . sampled training sets [ 3 , 21 ] . At a high level , stability measures the sensitivity of an algorithm to small changes in its training set , and is a classical approach to proving generalization bounds [ 3 , 21 ] . More often than not , the particular notion used in practice is uniform stability , where sensitivity is measured w.r.t . the worst case small change in the training dataset ; max ( ⊆Z ( < ) ‖ 5 ( A ( ( ) − 5 ( A ( ( ′ ) ) ) ‖ . Our key insight is that while we can not hope for uniform stability as losses are not assumed to be convex individually , we may exploit strong convexity of the population loss to show SGD admits average stability ; E ( ∼Z ( < ) ‖ 5 ( A ( ( ) ) − 5 ( A ( ( ′ ) ) ‖ ≤ ² ( < ) . In particular , we prove the gradient update is contractive in expectation ; E ‖G−η∇ 5 ( G ) − ( H−η∇ 5 ( H ) ) ‖ E ‖G− H‖ under the cumulative strong convexity assumption . In turn , this yields a stability result which implies regret that scales with 1/C for the early rounds up to C ≈ ) /κ , where κ is the condition number of the problem . In short , as the game progresses the bias of the random order gradient estimates increases , and the gradient update becomes unstable . To overcome the loss of stability in later rounds , we devise a simple online sampling mechanism that generates i.i.d . uniform samples from a random order distribution , effectively ensuring unbiased gradient estimates throughout all ) rounds , and consequently optimal regret up to additive factors . Finally , our approach allows us to develop an analysis framework that naturally accommodates SGD based algorithms in the random order model , and in a broader sense establish stability of SGD in a new , relatively general setting . 1.3 Related work . Random-Order Online Optimization was proposed in the recent work of [ 8 ] ( where it is referred to as ROOCO ) , who establish an $ ( ( 3 2/λ3 ) log ) ) regret upper bound for -Lipschitz λ-cumulativestrongly convex losses . In the classical OCO setup [ 25 , 12 ] , under the assumption the losses are λ-strongly convex individually , it is well known the minimax regret scales as Θ ( ( 2/λ ) log ) ) , and that the lower bound also applies under the assumption the adversary is i.i.d . stochastic [ 14 , 13 ] . 1 In addition , it is well known ( see e.g. , [ 4 ] ) that the upper bound for the i.i.d . adversary is obtained in expectation by SGD also for non-convex losses , as long as the expected loss is strongly convex . In this work , we show that the same regret upper bound also holds for the random order adversary . Also relevant to our work is the study of stability and generalization [ 3 , 21 ] in modern learning theory , and in particular stability properties of SGD . Proving generalization bounds with stability arguments is a well known approach dating back at least to [ 19 , 5 , 6 ] . The specific notion of average stability in the i.i.d . setting we draw upon was defined in [ 21 ] , though many similar measures have appeared in the literature long before their work ( see [ 3 , 21 ] for an overview ) . The influential work of [ 11 ] gave the first generalization bounds for general forms of SGD with an analysis relying on the notion of uniform stability [ 3 ] . Since then , several works have used a similar approach to gain further insight into stability and generalization properties of SGD , e.g. , [ 15 , 7 , 2 ] . A related line of work [ 9 , 22 , 16 , 20 , 18 ] studies SGD without-replacement for solving finite-sum optimization problems , a setting commonly encountered in offline machine learning applications . Here , multiple epochs of SGD are executed over a given training set , with the objective to produce a single output ( approximately ) minimizing the average loss . However , the majority of the results are obtained under the ( vastly simplifying ) assumption that the individual loss functions are convex , and therefore do not apply in our setting . In addition , the performance metric of interest is convergence rate , and not regret which is the focus of our paper . In particular , Nagaraj et al . [ 16 ] employ the method of exchangeable pairs to relate the average and random order loss to a stability-like property , and obtain optimal ( up to polylogarithmic factors ) convergence rate for a single epoch , albeit only for individually convex loss functions . Recently , a number of papers study SGD without-replacement and attempt to relax the convexity assumption [ 10 , 17 , 1 ] , but the bounds they obtain are under conditions inapplicable for our setting . 1The lower bound given in Hazan and Kale [ 13 ] uses quadratic ( hence , smooth ) losses , and is easily extensible to without-replacement sampling , and therefore applies in our setting . Specifically , they impose a requirement that the number of epochs passes a certain threshold strictly larger than one . Moreover , state-of-the-art bounds achieved by [ 1 ] are suboptimal w.r.t ours even had we ignored the epoch requirement .

Summary: The authors study a variant of OCO introduced by Garber, Korcia, and Levy in which the costs are adversarial but are revealed to the online learner in a uniformly random order. The authors propose a new gradient-based algorithm for this model and claim that the regret incurred by their algorithm has a strict improvment on the problem parameters compared to that of Garber et al. Specifically, the regret of the new algorithm does not depend on the ambient dimension $d$ at all and has improved dependence on the strong convexity parameter $\lambda$.