The Lazy Online Subgradient Algorithm is Universal on Strongly Convex Domains

√ N ) regret against adversarial opponents ; here we show it is universal in the sense that it also achieves O ( logN ) expected regret against i.i.d opponents . This improves upon the more complex metaalgorithm of Huang et al [ 20 ] that only gets O ( √ N logN ) and O ( logN ) bounds . In addition we show that , unlike for the simplex , order bounds for pseudo-regret and expected regret are equivalent for strongly convex domains . 1 Introduction . Online linear optimisation is a repeated game with one opponent . On turn n we know the cost vectors a1 , a2 , . . . , an−1 ∈ Rd and select an action xn from the domain X ⊂ Rd . The opponent observes xn and chooses the next cost vector an ∈ Rd and we pay cost an · xn . Our goal is to select actions to make the regret ∑N i=1 ai · ( xi − x∗ ) small for x∗ ∈ argmin { ∑N i=1 ai · x : x ∈ X } . If the cost vectors are selected by an adversarial opponent then a1 , a2 , . . . , an−1 give no information about an and selecting a good action seems like an impossible task . Despite this , there are algorithms that get O ( √ N ) regret against such opponents . The loss in performance against x∗ is O ( 1/ √ N ) per turn which vanishes as N → ∞ . The central algorithms in the field are Hedge [ 18 , 34 ] ; Prod [ 9 ] ; and ( sub ) Gradient Descent , which has Greedy [ 41 ] and Lazy [ 34 ] variants . All three algorithms get O ( √ N ) regret . The first two are specialised to X the simplex . The latter works for any compact convex domain . The bulk of online optimisation literature is refinements of the central algorithms . In many cases however the cost vectors are non adversarial . For example they might be determined by fluctuations in traffic , the weather or the stock market over a short period of time . The standard model of easy opponents is cost vectors drawn independently from some fixed distribution . In the easy setting the history a1 , . . . , an provides information about the next cost vector . Indeed ãn+1 = a1+ ... +ann is an increasingly reliable sequence of estimates for a = E [ an+1 ] and it seems reasonable to play xn+1 to minimise ãn+1 · x . This is called Follow-the-Leader ( FTL ) . For example if X is the simplex FTL gives pseudo-regret E [ ∑N i=1 a· ( xi−y∗ ) ] ≤ O ( 1 ) for y∗∈argmin { a·x : x∈X } . Note this is weaker than bounding the expected regret E [ ∑N i=1ai · ( xi − x∗ ) ] . On the other hand FTL gives no guarantee against adversarial opponents , and we might not know in advance whether the opponent is i.i.d or adversarial . Hence we would like an algorithm that is universal in that it always gets O ( √ N ) regret but specialises to a much stronger bound against i.i.d opponents . Recently Huang et al [ 20 ] solved this problem for smooth , strongly convex domains . For these domains they showed FTL gives O ( logN ) regret if ‖ ∑n i=1 ai‖ grows linearly.1 The result was generalised to nonsmooth domains by [ 22 , 26 ] . Huang et al then combine FTL with the ( A , B ) -Prod meta-algorithm of [ 31 ] to ensure O ( √ N logN ) regret against adversarial opponents . In this work 1 Without smoothness they show the growth condition occurs with high probability against i.i.d opponents . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Algorithm 1 : Online Lazy Gradient Descent Data : Cost vectors a1 , a2 , . . . ∈ Rd . Parameter η > 0 . Domain X ⊂ Rd . 1 for n = 0 , 1 , . . . do 2 xn+1 =ΠX ( − η√ n ∑n i=1 ai ) 3 Receive an+1 and pay cost an+1 · xn+1 we improve and extend their results . In particular we show the meta-algorithm is unnecessary as the much simpler Online Lazy Gradient Descent already achieves O ( √ N ) and O ( logN ) bounds . Terminology and Notation Throughout ‖ · ‖ is the Euclidean norm . Given a vector subspace V ⊂ Rd and linear operator A : V → V write the operator norm as ‖A‖ = max { ‖Av‖ : v ∈ V , ‖v‖ = 1 } . It is known ( see [ 10 ] Theorems 4.1 and 4.2 ) that ‖A‖ = maxj √ |λj | for λj the eigenvalues of ATA . Moreover if A is symmetric we have ‖A‖ = maxj |µj | for µj the eigenvalues of A . The set Z ⊂ Rd is called λ-strongly convex to mean for each x , y ∈ Z and α ∈ [ 0 , 1 ] the ballB ( αx+ ( 1−α ) y , r ) is contained in Z for r = λ2α ( 1− α ) ‖x− y‖ 2 . The pseudo-regret P [ RN ] and expected regret E [ RN ] are defined : P [ RN ] = E [ N∑ i=1 a·xi −min x∈X N∑ i=1 a·x ] E [ RN ] = E [ N∑ i=1 ai ·xi −min x∈X N∑ i=1 ai ·x ] ( 1 ) The Jensen Inequality implies P [ RN ] ≤ E [ RN ] . We write ΠX for the Euclidean projection onto the domain X . The Online Lazy Subgradient algorithm is given as Algorithm 1 . Note this is different from the so-called Greedy Subgradient algorithm with starting point x1 = 0 and recursive update xn+1 =ΠX ( xn−ηan/ √ n ) . Section 5 Figure 2 ( b ) suggests Greedy Subgradient is not universal . Summary of Contributions Throughout we make the following assumptions on the domain and cost vectors . Both assumptions are nontrivial . For example Assumption 1 holds on the Euclidean and ` p balls for p∈ ( 1 , 2 ] but not for the simplex or polytopes . Assumption 2 on the cost vectors is a typical model of an easy opponent and fails if the opponent is intelligent and adapts to our past moves . Assumption 1 . The domain X⊂Rd is compact , m-strongly convex and contains the origin . Write D = max { ‖x‖ : x , y∈X } . The boundaryM = ∂X is a ( d−1 ) -dimensional C2 manifold . Namely each z∈M has a neighborhood U in Rd and C2 function F : U→R with nonzero gradient such that M∩U= { x ∈ U : F ( x ) =0 } . Such a function is called a coordinate patch at z . Henceforth write N ( x ) for the outwards unit normal at x∈M . To represent N : M→Rd locally we can choose a coordinate patch F : U→R at x∗ and write∇N= ∇F‖∇F‖ . Since F is C 2 the matrix ∇N of partial derivatives exists . Assumption 2 . The cost vectors a1 , a2 , . . . are i.i.d with expectation E [ an ] = a . For all n we have ‖an‖ ≤ L and ‖an−a‖ ≤ R and E‖an−a‖ ≤ δ . Let x∗ ∈ argmin { a ·x : x ∈ X } be the expected minimiser . The domain X being strongly convex implies x∗ is unique . Our main result is Theorem 3 which says Online Lazy Gradient Descent achieves O ( L2 m‖a‖ logN ) expected-regret under Assumptions 1 and 2 . This bound is essentially tight by [ 20 ] Theorem 9 . The improved behaviour against i.i.d opponents is remarkable since the algorithm was designed with the adversarial case in mind , and predates the recent interest in universal algorithms . Our second contribution is Theorem 2 which states the gap between expected regret and pseudo-regret is O ( L2 m‖a‖ ) . Theorem 2 is of independent interest since it fails if the domain is the simplex . From a technical standpoint , our Gradient Descent analysis goes beyond that of [ 20 , 22 , 26 ] for FTL . Their central idea is that the normal vector toM at xn ∈ argmin { ( a1+ . . .+an−1 ) · x : x ∈X } points along An−1 =−a1+ ... +an−1n . Using ‖An−An−1‖=O ( 1/n ) and how strong convexity says the unit normal changes quickly as we vary the basepoint , they prove ‖Θn−Θn+1‖=O ( 1/n ) for the normal vectors Θn , Θn+1 at xn , xn+1 . Summing the resulting series gives a logarithmic regret bound . Unfortunately this method is insufficient to analyse Gradient Descent instead of FTL as it only gives ‖Θn−Θn+1‖=O ( 1/ √ n ) . and the resulting series give only an O ( √ N ) bound . Thus we follow a new approach where we prove a high probability bound ‖N ( xn ) −N ( x∗ ) ‖ = O ( 1/ √ n ) for the unit normals . For comparison existing works do not mention the expected minimiser x∗ at all . Then we use strong convexity to see the boundary surface locally looks like a quadratic with axis pointing along −a . Hence ‖xn−x∗‖ = O ( 1/ √ n ) and only O ( 1/n ) of the displacement is in the a-direction ; and the larger orthogonal component does not contribute to pseudo-regret . Summing the series gives an O ( logN ) bound for pseudo-regret . To convert this into a bound for expected regret we use our Theorem 2 which says the gap between expected regret and pseudo-regret is finite . Our differential geometry methods are more nuanced than existing works . Existing works consider only the magnitude ‖xn − xn+1‖ but we consider both the magnitude and direction of xn − x∗ . The relevant object is∇N ( z ) v in Lemmas 1-5 and Proposition 1 which captures the rate of change of the unit normal as we perturb the basepoint in the v-direction . 2 Related Work . The first strand of related work is universal algorithms . That is to say online algorithms with Õ ( √ N ) regret against adversarial opponents that specialise to get stronger bounds against i.i.d opponents . The original research focus was on the simplex and bandit feedback [ 4 , 8 , 32 , 33 , 37 , 40 ] . In the full information setting the FlipFlop algorithm [ 11 ] interpolates between FTL and Hedge on the simplex to get O ( 1 ) and O ( √ N logN ) regret against i.i.d and adversarial opponents respectively . The ( A , B ) -Prod meta-algorithm [ 31 ] does something similar in much greater generality . Another result comes from [ 16 ] . They design a variant of the Prod algorithm [ 9 ] with different learning rates for each arm . As a corollary they get O ( 1 ) expected regret against i.i.d opponents with a unique best arm . Apart from [ 31 ] all the above algorithms apply only to the simplex . However recent results suggest such complicated algorithms are unnecessary . In the full information setting it was recently proved that two of the central algorithms do much better against i.i.d opponents . Hedge gets O ( log ( d ) L2∞/∆ ) pseudo-regret on the simplex [ 27 ] ; and Online Lazy Gradient Descent gets O ( L22/∆ ) pseudo-regret on polytopes 2 [ 1 , 2 ] .The latter is notable since it works for domains other than the simplex . Here d is the dimension ; ∆ the suboptimality gap ; and we assume ‖an‖ ≤ L2 and ‖an‖∞ ≤ L∞ where appropriate . The above bounds are in addition to the O ( √ N ) regret bounds of Gradient Descent and Hedge ( see [ 34 ] Chapter 2 ) against adversarial opponents . The second strand of related work is the study of online algorithms on strongly convex domains . Typically these are unit balls with respect to some norm . For example [ 39 ] consider the ` p unit ball with p ∈ ( 1 , 2 ] in the full-information setting . Another setting of interest [ 5 , 12 , 19 ] is when predictions are available for the direction of the next cost vector . In the bandit setting [ 7 ] consider ` p balls ; [ 6 ] consider the Euclidean ball ; and [ 30 ] consider general strongly convex sets . They allow infinitely many arms but require the cost vectors be stochastically generated . In the offline setting [ 17 ] show the Frank-Wolfe method has O ( 1/n2 ) convergence rates − and hence finite regret − when the domain is strongly convex . [ 23 ] prove similar results for more general so-called uniformly convex sets . Under additional conditions the older works [ 13 , 14 , 25 ] show an exponential convergence rate .

Summary: The authors study online linear optimization and show that the Lazy Online Subgradient Algorithm is *universal* on strongly convex domains: it simultaneously achieves the optimal $O(\sqrt T)$ regret when the costs are adversarial and the optimal $O(\log T)$ regret when the costs are stochastic. This result is a clear improvement over previous work of Huang et al, who described a more complex algorithm and didn't achieve the optimal rates. The proof is based on an intricate analysis, drawing from differential geometry, of the points selected by the algorithm, and in particular, how much of each step affects regret against the minimizer $x^*$ and how much is orthogonal to the direction $x_i - x^*$ and doesn't affect regret. This technique may be of interest to the broader ML community.