Best-case lower bounds in online learning

1 Introduction . A typical work in online learning would develop algorithms that provably achieve low regret for some family of problems , where low regret means that a learning algorithm ’ s cumulative loss is not much larger than that of the best expert ( or action ) in hindsight . Such a work often focuses on algorithms that exhibit various forms of adaptivity , including anytime algorithms , which adapt to an unknown time horizon T ; timeless algorithms , which obtain “ first-order ” regret bounds that replace dependence on the time horizon by the cumulative loss of the best expert ; and algorithms like AdaGrad [ 8 ] , which adapt to the geometry of the data . These examples of adaptivity all involve competing with the best expert in hindsight , but adaptivity comes in many guises . Another form of adaptivity involves upgrading the comparator itself : in the shifting regret ( also known as the tracking regret ) [ 11 ] , the learning algorithm competes with the best sequence of experts that shifts , or switches , k times for some small k. Naturally , an algorithm with low shifting regret can potentially perform much better than the single best expert in hindsight , thereby obtaining classical regret that is substantially negative . Our work serves as a counterpoint to previous works : we show that a broad class of learning strategies provably fails , against any sequence of data , to substantially outperform the best expert in hindsight . Thus , these strategies are unable to obtain low shifting regret and , more generally , low regret against any comparator sequence that can be substantially better than the best expert in hindsight . More concretely , this paper initiates the study of best-case lower bounds in online convex optimization ( OCO ) for the general family of learning strategies known as Follow the Regularized Leader ( FTRL ) [ 1 ] . That is , we study the minimum possible regret of a learning algorithm over all possible sequences . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . As we will show , many instances of FTRL — including adaptive instances that are anytime , timeless , or adapt to gradients like AdaGrad — never have regret that is much less than the negation of the corresponding regret upper bounds . Thus , while these instances can be adaptive in some ways , they are in a sense prohibited from uniformly obtaining low regret for adaptive notions of regret like the shifting regret . For example , in the setting of decision-theoretic online learning ( DTOL ) with d experts [ 9 ] , the well-known anytime version of Hedge ( which uses the time-varying learning rate ηt √ log ( d ) /t ) enjoys O ( √ T log d ) worst-case regret and , as we show , has −O ( √ T log d ) best-case regret . Moreover , in the same setting under the restriction of two experts and binary losses , we exactly identify the best-case sequence , thereby showing that our best-case lower bound for this setting is tight . The structure of this sequence is surprisingly simple , but the arguments we use to pinpoint this sequence are playfully complex , bearing some similarity to the techniques of [ 20 ] and [ 15 ] . The latter work [ 15 ] considers the regret of Thompson Sampling in adversarial bit prediction ; they use swapping rules and identify best-case sequences , as do we . However , the algorithms and problem settings have important differences . A key motivation for our work is a recent result [ 4 ] which shows , in the setting of DTOL , that Hedge with constant learning rate has best-case regret lower bounded by −O ( √ T ) . This result , taken together with worst-case upper bounds of order O ( √ T ) , is then used to show that if each of finitely many experts approximately satisfies a certain notion of group fairness , then a clever use of the Hedge algorithm ( running it separately on each group ) also approximately satisfies the same notion of group fairness while still enjoying O ( √ T ) regret . However , we stress that their result is very limited in that it applies only to Hedge when run with a known time-horizon . The fixed time horizon assumption also implies that their notion of group fairness also is inherently tied to a fixed time horizon ( see Section 4.2 for a detailed discussion ) , and this latter implication can lead to experts that seem very unfair but which , based on a fixed horizon view of group fairness , are technically considered to be fair . Our best-case lower bounds enable the results of [ 4 ] to hold in much greater generality ; in particular , our results enable the use of an anytime version of group fairness , which we feel is truly needed . To our knowledge , our work is the first to study best-case lower bounds for Adaptive FTRL [ 16 ] , i.e. , FTRL with time-varying regularizers that can adapt to the learning algorithms ’ past observations . The most closely related work is a paper by Gofer and Mansour ( GM ) [ 10 ] which , in the setting of online linear optimization ( OLO ) and when using FTRL with a fixed regularizer,1 provides various lower bounds on the regret . For instance , they show that for any sequence of data , the regret is nonnegative ; we recover this result as a special case of our analysis , and our analysis extends to OCO as well . GM also lower bound what they call the anytime regret , which superficially may seem similar to our providing best-case lower bounds for anytime algorithms . Yet , as we explain in Section 3 , these two notions greatly differ . In short , their analysis lower bounds the maximum regret ( over all prefixes of a sequence ) for fixed horizon algorithms , whereas our analysis lower bounds the regret for all prefixes ( including the minimum ) for adaptively regularized algorithms , which includes anytime algorithms . A natural question is whether results similar to our results for FTRL also hold for online mirror descent ( OMD ) . In some situations , such as in OLO when the action space is the probability simplex , the regularizer is the negative Shannon entropy , and the learning rate is constant , our results automatically apply to OMD because the methods are then the same . More generally , it is known that OMD with a time-varying learning rate can fail spectacularly by obtaining linear regret ( see Theorem 4 of [ 18 ] ) . Since so much of our work is tied to obtaining anytime guarantees ( which would require a time-varying learning rate ) , we forego providing best-case lower bounds for OMD . Our main contributions are as follows : 1 . We give a general best-case lower bound on the regret for Adaptive FTRL ( Section 3 ) . Our analysis crucially centers on the notion of adaptively regularized regret , which serves as a potential function to keep track of the regret . 2 . We show that this general bound can easily be applied to yield concrete best-case lower bounds for FTRL with time-varying negative regularizers , one special case being the negative Shannon entropy . We also show that an adaptive gradient FTRL algorithm ( which can be viewed as a “ non-linearized ” version of the dual averaging version of AdaGrad [ 8 ] ; see Section 4.3 for details ) admits a best-case lower bound that is essentially the negation of its upper bound ( Section 4 ) . 1 [ 10 ] uses learners that follow the gradient of a concave potential , which is essentially equivalent to FTRL . 3 . A widely used variant of FTRL for OCO is to first linearize the losses , leading to linearized FTRL . This method works well with respect to upper bounds , as a basic argument involving convexity goes in the right direction . However , with regards to best-case lower bounds , we show a simple construction ( Section 5 ) for which linearized FTRL obtains −Ω ( T ) regret.2 4 . In the setting of DTOL with 2 experts and binary losses , we explicitly identify the best-case sequence , proving that our best-case lower bounds are tight in this setting ( Section 6 ) . The next section formalizes the problem setting and FTRL . We then develop the main results . 2 Problem Setting and General Prediction Strategies . Before giving the problem setting , we first set some notation . We denote the norm of a vector w ∈ W as ‖w‖ , the corresponding dual norm is denoted as ‖ · ‖∗ , and log is always the natural logarithm . Problem setting . We consider the OCO setting . This is a game between Learner and Nature . In each round t = 1 , 2 , . . . , T , Learner selects an action wt belonging to a closed , bounded , convex setW ⊆ Rd . Then , with knowledge of w1 , . . . , wt , Nature responds with a convex loss function ft : W 7→ R. Learner then observes ft and suffers loss ft ( wt ) . Learner ’ s goal is to minimize its regret , defined as RT : = sup w∈W T∑ t=1 [ ft ( wt ) − ft ( w ) ] , which is the gap between Learner ’ s cumulative loss and that of the best action in hindsight . This paper will cover several examples of OCO . The first example is the subclass of OLO problems . In OLO , the loss functions ft are linear , with ft ( w ) = 〈 ` t , w〉 for some loss vector ` t ∈ Rd . A noteworthy special case of OLO is DTOL , also known as the Hedge setting . In DTOL , we takeW to be equal to the simplex ∆d over d outcomes and restrict the loss vectors as ` t ∈ [ 0 , 1 ] d. We introduce some notation that will be useful in the DTOL setting . For any expert j ∈ [ d ] , let Lt , j : = ∑t s=1 ` s , j denote the cumulative loss of expert j until the end of round t. We denote the loss of Learner in round t as ˆ̀t : = 〈 ` t , wt〉 and Learner ’ s cumulative loss at the end of round t as L̂t : = ∑t s=1 ˆ̀ s. In this work , we consider the general prediction strategy of FTRL . FTRL . Let Φ1 , Φ2 , . . . be a possibly data-dependent sequence of regularizers where , for each t , the regularizer Φt is a mapping Φt : W → R which is allowed to depend on ( fs ) s≤t . Then FTRL chooses actions according to the past regularized cumulative loss : wt+1 = arg min w∈W { t∑ s=1 fs ( w ) + Φt ( w ) } . ( 1 ) We would like to emphasize that this is a very general template . It includes a fixed learning rate regularization , Φt ≡ 1ηΦ , as well as its variable learning rate counterpart , Φt = 1 ηt Φ , and arbitrary forms of adaptive choices of Φt based on the past . Despite this adaptivity , in the next section we will show that proving best-case lower bounds for this strategy is quite straightforward . The regret attained by FTRL is summarized in the following known result . Theorem 1 ( Theorem 1 of [ 16 ] ) . Let ( Φt ) t≥1 be a sequence of nonnegative regularizers such that , for each t ≥ 1 , Φt is 1-strongly convex with respect to a norm ‖ · ‖ ( t ) . The regret of the FTRL algorithm ( 1 ) with this sequence of regularizers is upper bounded as RT ≤ ΦT ( w∗ ) + 12 ∑T t=1 ‖∇ft ( wt ) ‖2 ( t−1 ) , ∗ , ( 2 ) where w∗ ∈ W is the best action in hindsight . If we do not require the regularizers to be nonnegative but instead assume that , for each w ∈ W , the sequence ( Φt ( w ) ) t≥1 is non-increasing , then ( 2 ) still holds if we replace ΦT ( w∗ ) by ΦT ( w∗ ) − infw∈W ΦT ( w ) . 2This negative construction , combined with the fact that the dual averaging version of AdaGrad is a linearized version of FTRL , is why we only prove best-case lower bounds for the adaptive gradient FTRL algorithm .

Summary: The paper establishes lower bounds on the best-case regret of the classical FTRL algorithm in online convex optimization. Specifically, the FTRL algorithm is considered in the general form where the regularizer can change over time (e.g., due to a time-varying learning rate). In the case of a fixed regularizer, it is known from earlier work that the regret is always non-negative, i.e., the algorithm never performs better than the best static expert. For time-varying regularizers, this is no longer the case, meaning that the best-case regret can be negative. The paper proves a simple generic lower bound theorem and applies it to several standard settings (Decreasing Hedge; a standard timeless method for decision-theoretic online learning (DTOL); an adaptive gradient version of FTRL), where it is found that the best-case lower bounds are roughly symmetrical to the known worst-case upper bounds on the regret (i.e., if A is a worst-case upper bound, then -O(A) is a best-case lower bound). In other words, these algorithms cannot outperform the best static expert by much, for any input. For DTOL with 2 experts, they show their lower bound to be tight.  For a different setting of the linearized FTRL algorithm, they show that an analogue result does not hold: Although the known regret upper bound is O(sqrt(T)), the best-case regret can be as low as -Omega(T).  An application and the original motivation for the work is in the setting of "group fairness" for DTOL: Here, time steps are partitioned into groups, and the algorithm knows which group the current time step belongs to. It is assumed that for each expert, the average loss per time step is balanced across the groups. Then recent work of Blum et al shows that the learning algorithm also has comparable loss in each group. This assumes that the time horizon T is known. This paper allows extending this result to an unknown time horizon, and shows an "anytime" extension of the result: If the assumption is strengthened so that for each expert, the losses are balanced across the groups not only after the last time step but also on every prefix of the input, then the learning algorithm also has balanced losses per group at all timesteps.