Best-case lower bounds in online learning

1 Introduction . A typical work in online learning would develop algorithms that provably achieve low regret for some family of problems , where low regret means that a learning algorithm ’ s cumulative loss is not much larger than that of the best expert ( or action ) in hindsight . Such a work often focuses on algorithms that exhibit various forms of adaptivity , including anytime algorithms , which adapt to an unknown time horizon T ; timeless algorithms , which obtain “ first-order ” regret bounds that replace dependence on the time horizon by the cumulative loss of the best expert ; and algorithms like AdaGrad [ 8 ] , which adapt to the geometry of the data . These examples of adaptivity all involve competing with the best expert in hindsight , but adaptivity comes in many guises . Another form of adaptivity involves upgrading the comparator itself : in the shifting regret ( also known as the tracking regret ) [ 11 ] , the learning algorithm competes with the best sequence of experts that shifts , or switches , k times for some small k. Naturally , an algorithm with low shifting regret can potentially perform much better than the single best expert in hindsight , thereby obtaining classical regret that is substantially negative . Our work serves as a counterpoint to previous works : we show that a broad class of learning strategies provably fails , against any sequence of data , to substantially outperform the best expert in hindsight . Thus , these strategies are unable to obtain low shifting regret and , more generally , low regret against any comparator sequence that can be substantially better than the best expert in hindsight . More concretely , this paper initiates the study of best-case lower bounds in online convex optimization ( OCO ) for the general family of learning strategies known as Follow the Regularized Leader ( FTRL ) [ 1 ] . That is , we study the minimum possible regret of a learning algorithm over all possible sequences . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . As we will show , many instances of FTRL — including adaptive instances that are anytime , timeless , or adapt to gradients like AdaGrad — never have regret that is much less than the negation of the corresponding regret upper bounds . Thus , while these instances can be adaptive in some ways , they are in a sense prohibited from uniformly obtaining low regret for adaptive notions of regret like the shifting regret . For example , in the setting of decision-theoretic online learning ( DTOL ) with d experts [ 9 ] , the well-known anytime version of Hedge ( which uses the time-varying learning rate ηt √ log ( d ) /t ) enjoys O ( √ T log d ) worst-case regret and , as we show , has −O ( √ T log d ) best-case regret . Moreover , in the same setting under the restriction of two experts and binary losses , we exactly identify the best-case sequence , thereby showing that our best-case lower bound for this setting is tight . The structure of this sequence is surprisingly simple , but the arguments we use to pinpoint this sequence are playfully complex , bearing some similarity to the techniques of [ 20 ] and [ 15 ] . The latter work [ 15 ] considers the regret of Thompson Sampling in adversarial bit prediction ; they use swapping rules and identify best-case sequences , as do we . However , the algorithms and problem settings have important differences . A key motivation for our work is a recent result [ 4 ] which shows , in the setting of DTOL , that Hedge with constant learning rate has best-case regret lower bounded by −O ( √ T ) . This result , taken together with worst-case upper bounds of order O ( √ T ) , is then used to show that if each of finitely many experts approximately satisfies a certain notion of group fairness , then a clever use of the Hedge algorithm ( running it separately on each group ) also approximately satisfies the same notion of group fairness while still enjoying O ( √ T ) regret . However , we stress that their result is very limited in that it applies only to Hedge when run with a known time-horizon . The fixed time horizon assumption also implies that their notion of group fairness also is inherently tied to a fixed time horizon ( see Section 4.2 for a detailed discussion ) , and this latter implication can lead to experts that seem very unfair but which , based on a fixed horizon view of group fairness , are technically considered to be fair . Our best-case lower bounds enable the results of [ 4 ] to hold in much greater generality ; in particular , our results enable the use of an anytime version of group fairness , which we feel is truly needed . To our knowledge , our work is the first to study best-case lower bounds for Adaptive FTRL [ 16 ] , i.e. , FTRL with time-varying regularizers that can adapt to the learning algorithms ’ past observations . The most closely related work is a paper by Gofer and Mansour ( GM ) [ 10 ] which , in the setting of online linear optimization ( OLO ) and when using FTRL with a fixed regularizer,1 provides various lower bounds on the regret . For instance , they show that for any sequence of data , the regret is nonnegative ; we recover this result as a special case of our analysis , and our analysis extends to OCO as well . GM also lower bound what they call the anytime regret , which superficially may seem similar to our providing best-case lower bounds for anytime algorithms . Yet , as we explain in Section 3 , these two notions greatly differ . In short , their analysis lower bounds the maximum regret ( over all prefixes of a sequence ) for fixed horizon algorithms , whereas our analysis lower bounds the regret for all prefixes ( including the minimum ) for adaptively regularized algorithms , which includes anytime algorithms . A natural question is whether results similar to our results for FTRL also hold for online mirror descent ( OMD ) . In some situations , such as in OLO when the action space is the probability simplex , the regularizer is the negative Shannon entropy , and the learning rate is constant , our results automatically apply to OMD because the methods are then the same . More generally , it is known that OMD with a time-varying learning rate can fail spectacularly by obtaining linear regret ( see Theorem 4 of [ 18 ] ) . Since so much of our work is tied to obtaining anytime guarantees ( which would require a time-varying learning rate ) , we forego providing best-case lower bounds for OMD . Our main contributions are as follows : 1 . We give a general best-case lower bound on the regret for Adaptive FTRL ( Section 3 ) . Our analysis crucially centers on the notion of adaptively regularized regret , which serves as a potential function to keep track of the regret . 2 . We show that this general bound can easily be applied to yield concrete best-case lower bounds for FTRL with time-varying negative regularizers , one special case being the negative Shannon entropy . We also show that an adaptive gradient FTRL algorithm ( which can be viewed as a “ non-linearized ” version of the dual averaging version of AdaGrad [ 8 ] ; see Section 4.3 for details ) admits a best-case lower bound that is essentially the negation of its upper bound ( Section 4 ) . 1 [ 10 ] uses learners that follow the gradient of a concave potential , which is essentially equivalent to FTRL . 3 . A widely used variant of FTRL for OCO is to first linearize the losses , leading to linearized FTRL . This method works well with respect to upper bounds , as a basic argument involving convexity goes in the right direction . However , with regards to best-case lower bounds , we show a simple construction ( Section 5 ) for which linearized FTRL obtains −Ω ( T ) regret.2 4 . In the setting of DTOL with 2 experts and binary losses , we explicitly identify the best-case sequence , proving that our best-case lower bounds are tight in this setting ( Section 6 ) . The next section formalizes the problem setting and FTRL . We then develop the main results . 2 Problem Setting and General Prediction Strategies . Before giving the problem setting , we first set some notation . We denote the norm of a vector w ∈ W as ‖w‖ , the corresponding dual norm is denoted as ‖ · ‖∗ , and log is always the natural logarithm . Problem setting . We consider the OCO setting . This is a game between Learner and Nature . In each round t = 1 , 2 , . . . , T , Learner selects an action wt belonging to a closed , bounded , convex setW ⊆ Rd . Then , with knowledge of w1 , . . . , wt , Nature responds with a convex loss function ft : W 7→ R. Learner then observes ft and suffers loss ft ( wt ) . Learner ’ s goal is to minimize its regret , defined as RT : = sup w∈W T∑ t=1 [ ft ( wt ) − ft ( w ) ] , which is the gap between Learner ’ s cumulative loss and that of the best action in hindsight . This paper will cover several examples of OCO . The first example is the subclass of OLO problems . In OLO , the loss functions ft are linear , with ft ( w ) = 〈 ` t , w〉 for some loss vector ` t ∈ Rd . A noteworthy special case of OLO is DTOL , also known as the Hedge setting . In DTOL , we takeW to be equal to the simplex ∆d over d outcomes and restrict the loss vectors as ` t ∈ [ 0 , 1 ] d. We introduce some notation that will be useful in the DTOL setting . For any expert j ∈ [ d ] , let Lt , j : = ∑t s=1 ` s , j denote the cumulative loss of expert j until the end of round t. We denote the loss of Learner in round t as ˆ̀t : = 〈 ` t , wt〉 and Learner ’ s cumulative loss at the end of round t as L̂t : = ∑t s=1 ˆ̀ s. In this work , we consider the general prediction strategy of FTRL . FTRL . Let Φ1 , Φ2 , . . . be a possibly data-dependent sequence of regularizers where , for each t , the regularizer Φt is a mapping Φt : W → R which is allowed to depend on ( fs ) s≤t . Then FTRL chooses actions according to the past regularized cumulative loss : wt+1 = arg min w∈W { t∑ s=1 fs ( w ) + Φt ( w ) } . ( 1 ) We would like to emphasize that this is a very general template . It includes a fixed learning rate regularization , Φt ≡ 1ηΦ , as well as its variable learning rate counterpart , Φt = 1 ηt Φ , and arbitrary forms of adaptive choices of Φt based on the past . Despite this adaptivity , in the next section we will show that proving best-case lower bounds for this strategy is quite straightforward . The regret attained by FTRL is summarized in the following known result . Theorem 1 ( Theorem 1 of [ 16 ] ) . Let ( Φt ) t≥1 be a sequence of nonnegative regularizers such that , for each t ≥ 1 , Φt is 1-strongly convex with respect to a norm ‖ · ‖ ( t ) . The regret of the FTRL algorithm ( 1 ) with this sequence of regularizers is upper bounded as RT ≤ ΦT ( w∗ ) + 12 ∑T t=1 ‖∇ft ( wt ) ‖2 ( t−1 ) , ∗ , ( 2 ) where w∗ ∈ W is the best action in hindsight . If we do not require the regularizers to be nonnegative but instead assume that , for each w ∈ W , the sequence ( Φt ( w ) ) t≥1 is non-increasing , then ( 2 ) still holds if we replace ΦT ( w∗ ) by ΦT ( w∗ ) − infw∈W ΦT ( w ) . 2This negative construction , combined with the fact that the dual averaging version of AdaGrad is a linearized version of FTRL , is why we only prove best-case lower bounds for the adaptive gradient FTRL algorithm .

Summary: Inspired by applications of fair online learning, the authors initiate the study of *lower bounds* on the regret of OCO algorithms. They provide a general lower bound for FTRL algorithms and show the implications for fixed regularizers and for some special kinds of time-varying regularizers (such as fixed regularizers with time-varying learning rates or regularizers dependent on the past losses). They also provide regret lower bounds on AdaGrad-like algorithms derived from the FTRL framework, and show how linearizing the losses can yield to algorithms with potentially $-\Omega(T)$ regret in $T$ rounds. They also explicitly build the best-case losses for the experts problem with 2 experts, giving sharp bounds for this special case;  Their main application is to fairness for decision theoretic online learning (experts' problem), strengthening previous fairness bounds. They show a way to get fairness guarantees without needing to know the size of the groups, and extending the fairness notion in online learning in a way that is independent of the length of the game.