The Lazy Online Subgradient Algorithm is Universal on Strongly Convex Domains

√ N ) regret against adversarial opponents ; here we show it is universal in the sense that it also achieves O ( logN ) expected regret against i.i.d opponents . This improves upon the more complex metaalgorithm of Huang et al [ 20 ] that only gets O ( √ N logN ) and O ( logN ) bounds . In addition we show that , unlike for the simplex , order bounds for pseudo-regret and expected regret are equivalent for strongly convex domains . 1 Introduction . Online linear optimisation is a repeated game with one opponent . On turn n we know the cost vectors a1 , a2 , . . . , an−1 ∈ Rd and select an action xn from the domain X ⊂ Rd . The opponent observes xn and chooses the next cost vector an ∈ Rd and we pay cost an · xn . Our goal is to select actions to make the regret ∑N i=1 ai · ( xi − x∗ ) small for x∗ ∈ argmin { ∑N i=1 ai · x : x ∈ X } . If the cost vectors are selected by an adversarial opponent then a1 , a2 , . . . , an−1 give no information about an and selecting a good action seems like an impossible task . Despite this , there are algorithms that get O ( √ N ) regret against such opponents . The loss in performance against x∗ is O ( 1/ √ N ) per turn which vanishes as N → ∞ . The central algorithms in the field are Hedge [ 18 , 34 ] ; Prod [ 9 ] ; and ( sub ) Gradient Descent , which has Greedy [ 41 ] and Lazy [ 34 ] variants . All three algorithms get O ( √ N ) regret . The first two are specialised to X the simplex . The latter works for any compact convex domain . The bulk of online optimisation literature is refinements of the central algorithms . In many cases however the cost vectors are non adversarial . For example they might be determined by fluctuations in traffic , the weather or the stock market over a short period of time . The standard model of easy opponents is cost vectors drawn independently from some fixed distribution . In the easy setting the history a1 , . . . , an provides information about the next cost vector . Indeed ãn+1 = a1+ ... +ann is an increasingly reliable sequence of estimates for a = E [ an+1 ] and it seems reasonable to play xn+1 to minimise ãn+1 · x . This is called Follow-the-Leader ( FTL ) . For example if X is the simplex FTL gives pseudo-regret E [ ∑N i=1 a· ( xi−y∗ ) ] ≤ O ( 1 ) for y∗∈argmin { a·x : x∈X } . Note this is weaker than bounding the expected regret E [ ∑N i=1ai · ( xi − x∗ ) ] . On the other hand FTL gives no guarantee against adversarial opponents , and we might not know in advance whether the opponent is i.i.d or adversarial . Hence we would like an algorithm that is universal in that it always gets O ( √ N ) regret but specialises to a much stronger bound against i.i.d opponents . Recently Huang et al [ 20 ] solved this problem for smooth , strongly convex domains . For these domains they showed FTL gives O ( logN ) regret if ‖ ∑n i=1 ai‖ grows linearly.1 The result was generalised to nonsmooth domains by [ 22 , 26 ] . Huang et al then combine FTL with the ( A , B ) -Prod meta-algorithm of [ 31 ] to ensure O ( √ N logN ) regret against adversarial opponents . In this work 1 Without smoothness they show the growth condition occurs with high probability against i.i.d opponents . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Algorithm 1 : Online Lazy Gradient Descent Data : Cost vectors a1 , a2 , . . . ∈ Rd . Parameter η > 0 . Domain X ⊂ Rd . 1 for n = 0 , 1 , . . . do 2 xn+1 =ΠX ( − η√ n ∑n i=1 ai ) 3 Receive an+1 and pay cost an+1 · xn+1 we improve and extend their results . In particular we show the meta-algorithm is unnecessary as the much simpler Online Lazy Gradient Descent already achieves O ( √ N ) and O ( logN ) bounds . Terminology and Notation Throughout ‖ · ‖ is the Euclidean norm . Given a vector subspace V ⊂ Rd and linear operator A : V → V write the operator norm as ‖A‖ = max { ‖Av‖ : v ∈ V , ‖v‖ = 1 } . It is known ( see [ 10 ] Theorems 4.1 and 4.2 ) that ‖A‖ = maxj √ |λj | for λj the eigenvalues of ATA . Moreover if A is symmetric we have ‖A‖ = maxj |µj | for µj the eigenvalues of A . The set Z ⊂ Rd is called λ-strongly convex to mean for each x , y ∈ Z and α ∈ [ 0 , 1 ] the ballB ( αx+ ( 1−α ) y , r ) is contained in Z for r = λ2α ( 1− α ) ‖x− y‖ 2 . The pseudo-regret P [ RN ] and expected regret E [ RN ] are defined : P [ RN ] = E [ N∑ i=1 a·xi −min x∈X N∑ i=1 a·x ] E [ RN ] = E [ N∑ i=1 ai ·xi −min x∈X N∑ i=1 ai ·x ] ( 1 ) The Jensen Inequality implies P [ RN ] ≤ E [ RN ] . We write ΠX for the Euclidean projection onto the domain X . The Online Lazy Subgradient algorithm is given as Algorithm 1 . Note this is different from the so-called Greedy Subgradient algorithm with starting point x1 = 0 and recursive update xn+1 =ΠX ( xn−ηan/ √ n ) . Section 5 Figure 2 ( b ) suggests Greedy Subgradient is not universal . Summary of Contributions Throughout we make the following assumptions on the domain and cost vectors . Both assumptions are nontrivial . For example Assumption 1 holds on the Euclidean and ` p balls for p∈ ( 1 , 2 ] but not for the simplex or polytopes . Assumption 2 on the cost vectors is a typical model of an easy opponent and fails if the opponent is intelligent and adapts to our past moves . Assumption 1 . The domain X⊂Rd is compact , m-strongly convex and contains the origin . Write D = max { ‖x‖ : x , y∈X } . The boundaryM = ∂X is a ( d−1 ) -dimensional C2 manifold . Namely each z∈M has a neighborhood U in Rd and C2 function F : U→R with nonzero gradient such that M∩U= { x ∈ U : F ( x ) =0 } . Such a function is called a coordinate patch at z . Henceforth write N ( x ) for the outwards unit normal at x∈M . To represent N : M→Rd locally we can choose a coordinate patch F : U→R at x∗ and write∇N= ∇F‖∇F‖ . Since F is C 2 the matrix ∇N of partial derivatives exists . Assumption 2 . The cost vectors a1 , a2 , . . . are i.i.d with expectation E [ an ] = a . For all n we have ‖an‖ ≤ L and ‖an−a‖ ≤ R and E‖an−a‖ ≤ δ . Let x∗ ∈ argmin { a ·x : x ∈ X } be the expected minimiser . The domain X being strongly convex implies x∗ is unique . Our main result is Theorem 3 which says Online Lazy Gradient Descent achieves O ( L2 m‖a‖ logN ) expected-regret under Assumptions 1 and 2 . This bound is essentially tight by [ 20 ] Theorem 9 . The improved behaviour against i.i.d opponents is remarkable since the algorithm was designed with the adversarial case in mind , and predates the recent interest in universal algorithms . Our second contribution is Theorem 2 which states the gap between expected regret and pseudo-regret is O ( L2 m‖a‖ ) . Theorem 2 is of independent interest since it fails if the domain is the simplex . From a technical standpoint , our Gradient Descent analysis goes beyond that of [ 20 , 22 , 26 ] for FTL . Their central idea is that the normal vector toM at xn ∈ argmin { ( a1+ . . .+an−1 ) · x : x ∈X } points along An−1 =−a1+ ... +an−1n . Using ‖An−An−1‖=O ( 1/n ) and how strong convexity says the unit normal changes quickly as we vary the basepoint , they prove ‖Θn−Θn+1‖=O ( 1/n ) for the normal vectors Θn , Θn+1 at xn , xn+1 . Summing the resulting series gives a logarithmic regret bound . Unfortunately this method is insufficient to analyse Gradient Descent instead of FTL as it only gives ‖Θn−Θn+1‖=O ( 1/ √ n ) . and the resulting series give only an O ( √ N ) bound . Thus we follow a new approach where we prove a high probability bound ‖N ( xn ) −N ( x∗ ) ‖ = O ( 1/ √ n ) for the unit normals . For comparison existing works do not mention the expected minimiser x∗ at all . Then we use strong convexity to see the boundary surface locally looks like a quadratic with axis pointing along −a . Hence ‖xn−x∗‖ = O ( 1/ √ n ) and only O ( 1/n ) of the displacement is in the a-direction ; and the larger orthogonal component does not contribute to pseudo-regret . Summing the series gives an O ( logN ) bound for pseudo-regret . To convert this into a bound for expected regret we use our Theorem 2 which says the gap between expected regret and pseudo-regret is finite . Our differential geometry methods are more nuanced than existing works . Existing works consider only the magnitude ‖xn − xn+1‖ but we consider both the magnitude and direction of xn − x∗ . The relevant object is∇N ( z ) v in Lemmas 1-5 and Proposition 1 which captures the rate of change of the unit normal as we perturb the basepoint in the v-direction . 2 Related Work . The first strand of related work is universal algorithms . That is to say online algorithms with Õ ( √ N ) regret against adversarial opponents that specialise to get stronger bounds against i.i.d opponents . The original research focus was on the simplex and bandit feedback [ 4 , 8 , 32 , 33 , 37 , 40 ] . In the full information setting the FlipFlop algorithm [ 11 ] interpolates between FTL and Hedge on the simplex to get O ( 1 ) and O ( √ N logN ) regret against i.i.d and adversarial opponents respectively . The ( A , B ) -Prod meta-algorithm [ 31 ] does something similar in much greater generality . Another result comes from [ 16 ] . They design a variant of the Prod algorithm [ 9 ] with different learning rates for each arm . As a corollary they get O ( 1 ) expected regret against i.i.d opponents with a unique best arm . Apart from [ 31 ] all the above algorithms apply only to the simplex . However recent results suggest such complicated algorithms are unnecessary . In the full information setting it was recently proved that two of the central algorithms do much better against i.i.d opponents . Hedge gets O ( log ( d ) L2∞/∆ ) pseudo-regret on the simplex [ 27 ] ; and Online Lazy Gradient Descent gets O ( L22/∆ ) pseudo-regret on polytopes 2 [ 1 , 2 ] .The latter is notable since it works for domains other than the simplex . Here d is the dimension ; ∆ the suboptimality gap ; and we assume ‖an‖ ≤ L2 and ‖an‖∞ ≤ L∞ where appropriate . The above bounds are in addition to the O ( √ N ) regret bounds of Gradient Descent and Hedge ( see [ 34 ] Chapter 2 ) against adversarial opponents . The second strand of related work is the study of online algorithms on strongly convex domains . Typically these are unit balls with respect to some norm . For example [ 39 ] consider the ` p unit ball with p ∈ ( 1 , 2 ] in the full-information setting . Another setting of interest [ 5 , 12 , 19 ] is when predictions are available for the direction of the next cost vector . In the bandit setting [ 7 ] consider ` p balls ; [ 6 ] consider the Euclidean ball ; and [ 30 ] consider general strongly convex sets . They allow infinitely many arms but require the cost vectors be stochastically generated . In the offline setting [ 17 ] show the Frank-Wolfe method has O ( 1/n2 ) convergence rates − and hence finite regret − when the domain is strongly convex . [ 23 ] prove similar results for more general so-called uniformly convex sets . Under additional conditions the older works [ 13 , 14 , 25 ] show an exponential convergence rate .

Summary: The authors study online linear optimization in the case of known strongly convex constraints sets. They show that the simple method of online subgradient descent, which is known to achieve $O(\sqrt{N})$ regret in the adversarial case, also achieves $O(\log(N))$ regret in the i.i.d. case. This result is more elegant than and slightly improves upon existing results in the strongly convex case.