Logarithmic Regret from Sublinear Hints

p T ) in the standard setting . In this work , we study the question of whether an algorithm really requires a hint at every time step . Somewhat surprisingly , we show that an algorithm can obtain O ( log T ) regret with just O ( p T ) hints under a natural query model ; in contrast , we also show that o ( p T ) hints can not guarantee better than ⌦ ( p T ) regret . We give two applications of our result , to the well-studied setting of optimistic regret bounds and to the problem of online learning with abstention . 1 Introduction . There has been a spate of work on improving the performance of online algorithms with the help of externally available hints . The goal of these works is to circumvent worst-case bounds and exploit the capability of machine-learned models that can potentially provide these hints . There have been two main lines of study . The first is for combinatorial problems , where the goal has been to be improve the competitive ratio of online algorithms ; problems considered here include ski-rental [ 9 , 17 ] , caching [ 13 , 19 , 26 ] , scheduling [ 1 , 12 , 17 , 22 ] , matching [ 16 , 18 ] , etc . The second is in the learning theory setting , where the goal has to been to improve the regret of online optimization algorithms . A series of recent papers showed how to achieve better regret guarantees , assuming that we have a hint about the cost function before the algorithm makes a choice . For many variants of the online convex optimization problem , works such as [ 25 , 8 , 2 , 3 ] studied the power of having prior information about a cost function . Works such as [ 29 ] have also studied improved regret bounds in partial information or bandit settings . In all these works , a desirable property is to ensure consistency , which demands better performance with better quality hints , and robustness , which guarantees a certain level of performance with poor quality or even adversarially bad hints . Recall the standard online optimization model [ 30 ] , which is a game between an algorithm and an adversary . In each round , the algorithm plays a point and the adversary responds with a cost function that is visible to the algorithm , and the cost in this round is measured by evaluating the cost function on the played point . In online linear optimization , the cost function is linear . The regret of an algorithm is the worst-case difference between the cost of the algorithm and the cost of an algorithm 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . that plays a fixed point in each round . A natural way to incorporate hints or prior information is to give the algorithm access to a hint about the cost function at a given round before it chooses a point to play at that round . In this sense , hints are present gratis [ 25 ] . The availability of a hint in each round seems natural in some settings , e.g. , when cost functions change gradually with time [ 25 , 4 ] ) . However , it can be prohibitive in many others , for instance , if hints are obtained by using expensive side information , or if they are generated by a running a computationally expensive ML model . Furthermore , hints can also be wasteful when the problem instance , or even a large sub-instance , is such that the algorithm can not really derive any substantial benefit from their presence . This leads to the question of strengthening the online learning with hints model by making it parsimonious . In this work , we pursue this direction , where we offer the algorithm the flexibility to choose to ask for a hint before it plays the point . It then becomes an onus on the algorithm to know when to ask for a hint and how to use it judiciously , while ensuring both consistency and robustness . The performance of such an algorithm is measured not only by its regret but also by the number of hints it uses . Our contributions . We now present a high level summary of our results . For ease of exposition , we will defer the formal statements to the respective sections . All our results are for the problem of online linear optimization ( OLO ) when the domain is the unit ` 2 ball ; the cost function at every step is defined using a cost vector ct ( and the cost or loss is the inner product of the point played with ct ) . We call a hint perfect if it is the same as the cost vector at that time step , good if it is weakly correlated with the cost vector , and bad otherwise . As our main result , we show that for OLO in which hints are guaranteed to be good whenever the algorithm asks for a hint , there is an efficient randomized algorithm that obtains O ( log T ) regret using only O ( p T ) hints . We extend our result to the case when |B| hints can be bad ( chosen in an oblivious manner , as we will discuss later ) , and give an algorithm that achieves a regret bound of O ( p |B| log T ) , while still asking for O ( p T ) hints . It is interesting to contrast our result with prior work : Dekel et al . [ 8 ] obtained an algorithm with O ( log T ) regret , when a good hint is available in every round . Bhaskara et al . [ 2 ] made this result robust , obtaining a regret bound of O ( p |B| log T ) , when there are |B| bad hints . Our result improves upon these works by showing the same asymptotic regret bounds , but using only O ( p T ) hints . Our result also has implications for optimistic regret bounds [ 25 ] ( where we obtain the same results , but with fewer hints ) and for online learning with abstention [ 24 ] ( where we can bound the number of abstentions ) . We also show two lower bounds that show the optimality of our algorithm . The first is regarding the minimum number of hints needed to get O ( log T ) regret : we show that any ( potentially randomized ) algorithm that uses o ( p T ) hints will suffer a regret of ⌦ ( p T ) . The second and more surprising result is the role of randomness : we show that any deterministic algorithm that obtains O ( log T ) regret must use ⌦ ( T/ log ( T ) ) hints , even if each of them is perfect . This shows the significance of having a randomized algorithm and an oblivious adversary . Finally we extend our results to the unconstrained OLO setting ( see Section 6 ) , where we design a deterministic algorithm to obtain O ( log3/2 T ) regret ( suitably defined for the unconstrained case ) when all hints are good , and a randomized algorithm to obtain O ( log T ) regret , which can be extended to the presence of bad hints . There are three aspects of our results that we find surprising . The first is even the possibility of obtaining O ( log T ) regret using only a sublinear number of hints . The second is the sharp threshold on the number of hints needed to obtain logarithmic regret ; the regret does not gracefully degrade when the number of hints is below O ( p T ) . The third is the deterministic vs randomized separation between the constrained and the unconstrained cases , when all queried hints are good . We now present some intuition why our result is plausible . Consider the standard “ worst-case ” adversary for OLO : random mean-zero costs . This case is “ hard ” because the learner achieves zero expected cost , but the competitor achieves p T total cost . However , if we simply play a hint ht on O ( p T/↵ ) rounds , each such round incurs ↵ cost , which is enough to cancel out the p T regret while making only O ( p T ) hint queries . Thus , the standard worst-case instances are actually easy with hints . More details on this example and an outline of our algorithm are provided at the start of Section 3 . Organization . Section 2 provides the necessary background . The main algorithm and analysis for the constrained OLO case are in Section 3 . The extensions and applications of this can be found in Section 4 . Section 5 contains the lower bounds and Section 6 contains the algorithms and analyses for the unconstrained case . All missing proofs are in the Supplementary Material . 2 Preliminaries . Let k · k denote the ` 2-norm and Bd = { x 2 Rd | kxk 1 } denote the unit ` 2 ball in Rd . We use the compressed sum notation and use c1 : t to denote Pt i=1 ci and kck21 : t to denote Pt i=1 kcik2 . Let ~c = c1 , . . . , cT be a sequence of cost vectors . Let [ T ] = { 1 , . . . , T } . OLO problem and Regret . The constrained online linear optimization ( OLO ) problem is modeled as a game over T rounds . At each time t 2 [ T ] , an algorithm A plays a vector xt 2 Bd , and then an adversary responds with a cost vector ct 2 Bd . The algorithm incurs cost ( or loss ) hxt , cti at time t. The total cost incurred by the algorithm is costA ( ~c ) = PT t=1hxt , cti . The regret of the algorithm A with respect to a ‘ comparator ’ or benchmark vector u 2 Bd is RA ( u , ~c ) = costA ( ~c ) costAu ( ~c ) = TX t=1 hxt u , cti , where Au is the algorithm that always plays u at every time step . The regret of an algorithm A is its worst-case regret with respect to all u 2 Bd : RA ( ~c ) = sup u2Bd RA ( u , ~c ) . Hints and query cost . Let ↵ > 0 be fixed and known . In this paper we consider the OLO setting where , at any round t before choosing xt , an algorithm A is allowed to obtain a hint ht 2 Bd . If hht , cti ↵kctk2 , we say that the hint is ↵-good . If A opts to obtain a hint at time t , then it incurs a query cost of ↵kctk2 ; the query cost is 0 if no hint was obtained at time t. The definition of regret stays the same and we denote it by RA , ↵ ( · ) . The total query cost of A is given by QA , ↵ ( ~c ) =PT t=1 t · ↵kctk2 where t is an indicator function used to denote whether A queried for a hint at time t. Note that the algorithm does not actually know the query cost for a round until the end of the round . If A is a randomized algorithm , the notions of expected regret and expected query cost follow naturally . We consider the setting when the adversarial choice of the hint ht and cost vector ct at time t is oblivious to whether the algorithm queries for a hint at time t but can depend adaptively on all previous decisions . More generally , we consider the case that some subset of the hints are “ bad ” in the sense that hct , hti < ↵kctk2 ; we let B denote the set of such indices t. Although we assume ↵ is known to our algorithms , we do not assume any information about B . Further , our algorithm is charged ↵kctk2 for querying a hint even if the hint was bad .

Summary:  Prior work has shown that in the online linear optimization problem, if at each round, the learner is provided with a hint on the cost prior to its selection by an adversary, then one can achieve a regret of order $O(\log(T))$. The authors consider the question of how many hints the learner requires in order to achieve such a regret rate. They show that $O(\sqrt{T})$ hints are sufficient by proposing an algorithm that achieves $O(\log(T))$. The proposed algorithm is a combination of Online Gradient Descent (OGD) and Follow The Regularized Leader (FTRL). OGD learns adaptively the probability by which the algorithm decides to query a hint or not, while, FTRL proposes which action to play whenever the algorithm decides not to query a hint. Conversely, the authors also show that with $o(\sqrt{T})$ hints, regret is no less than $\Omega(\sqrt{T})$.