Bandits with Knapsacks beyond the Worst Case

1 Introduction . We study multi-armed bandit problems with supply or budget constraints . Multi-armed bandits is a simple model for exploration-exploitation tradeoff , i.e. , the tension between acquiring new information and making optimal decisions . It is an active research area , spanning computer science , operations research , and economics . Supply/budget constraints arise in many realistic applications , e.g. , a seller who dynamically adjusts the prices or product assortment may have a limited inventory , and an algorithm that optimizes ad placement is constrained by the advertisers ’ budgets . Other motivating examples concern repeated auctions , crowdsourcing markets , and network routing . We consider a general model called Bandits with Knapsacks ( BwK ) , which subsumes the examples mentioned above . There are d 2 resources that are consumed over time , one of which is time itself . Each resource i starts out with budget Bi . In each round t , the algorithm chooses an action ( arm ) a = at from a fixed set of K actions . The outcome is a vector in [ 0 , 1 ] d+1 : it consists of a reward and consumption of each resource . This vector is drawn independently from some distribution over [ 0 , 1 ] d+1 , which depends on the chosen arm but not on the round , and is not known to the algorithm . The algorithm observes bandit feedback , i.e. , only the outcome of the chosen arm . The algorithm stops at a known time horizon T , or when the total consumption of some resource exceeds its budget . The goal is to maximize the total reward , denoted REW . The presence of supply/budget constraints makes the problem much more challenging . First , algorithm ’ s choices constrain what it can do in the future . Second , the algorithm is no longer looking for arms with maximal expected per-round reward ( because such arms may consume too much resources ) . Third , the best fixed distribution over arms can be much better than the best fixed arm . Accordingly , we compete with the best fixed distribution benchmark : the total expected reward of the best distribution , denoted OPTFD . All this complexity is already present even when d = 2 , i.e. , when there is only one resource other than time , and the minimal budget is B = mini Bi = ⌦ ( T ) . BwK were introduced in [ 14 , 16 ] and extensively studied since then . The optimal worst-case regret rate is well-understood . In particular , it is Õ ( p KT ) when B = ⌦ ( T ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We present several results that go beyond the worst-case perspective : 1 . We provide a full characterization for instance-dependent regret rates . In stochastic bandits , one obtains regret O K log T , where is the the reward-gap : the gap in expected reward between the best and the second-best arm . We work out whether , when and how such results extend to BwK . 2 . We show that simple regret , which tracks algorithm ’ s performance in a given round , can be small in all but a few rounds . Like in stochastic bandits , simple regret can be at least ✏ in at most Õ ( K/✏2 ) rounds , and this is achieved for all ✏ > 0 simultaneously . 3 . We improve all results mentioned above for a large number of arms , assuming some helpful structure . In fact , we provide a general “ reduction '' from BwK to stochastic bandits , and apply this reduction to three well-studied scenarios from stochastic bandits . Our algorithmic results focus on UcbBwK , a BwK algorithm from [ 3 ] which implements the “ optimism under uncertainty '' paradigm and attains the optimal worst-case regret bound . We provide new analyses of this algorithm along the above-mentioned themes . Related work . Background on multi-armed bandits can be found in books [ 23 , 54 , 42 ] . Stochastic bandits ( i.e. , BwK without resources ) is a basic , well-understood version . The dependence on and ✏ are optimal as stated above [ 41 , 10 , 11 ] , and is achieved simultaneously with the optimal worst-case regret eO ( KT ) , e.g. , in [ 10 ] . Various refinements are known for O ( log T ) regret [ 10 , 8 , 34 , 32 , 45 ] . Most relevant to this paper is O ( P a log ( T ) / ( a ) ) regret , where ( a ) is the gap in expected reward between arm a and the best arm [ 10 ] . Improving regret for large / infinite number of arms via a helpful structure is a unifying theme for several prominent lines of work , e.g. , linear bandits , convex bandits , Lipschitz bandits , and combinatorial ( semi- ) bandits . Bandits with Knapsacks were introduced in [ 14 , 16 ] , and optimally solved in the worst case . Subsequent work extended BwK to a more general notion of rewards/consumptions [ 3 ] , combinatorial semibandits [ 49 ] , and contextual bandits [ 15 , 6 , 4 ] . Several special cases with budget/supply constraints were studied separately ( and inspired a generalization to BwK ) : dynamic pricing [ 19 , 12 , 20 , 59 ] , dynamic procurement [ 13 , 52 ] , and dynamic ad allocation [ 53 , 28 ] . The adversarial version of BwK was studied by [ 35 , 36 ] . All this work considers worst-case regret bounds . Several papers achieve O ( log T ) regret in BwK , but with substantial caveats that we avoid . [ 61 ] assume deterministic consumption , whereas all motivating examples of BwK require stochastic consumption correlated with rewards ( e.g. , dynamic pricing consumes supply only if a sale happens ) . They posit d = 2 and no other assumptions , whereas we show that “ best-arm optimality '' is necessary with stochastic consumption . [ 31 ] assume “ best-arm-optimality '' as we do ( it is implicit in their version of reward-gap ) . However , their algorithm inputs an instance-dependent parameter which is “ hidden '' in BwK . Moreover , their O ( log T ) regret bound scales with cmin , minimal expected consumption among arms ( as c 4min ) . Their worst-case regret bound is suboptimal , since it also scales with cmin ( as c 2 min ) , and only applies for d = 2 . [ 58 ] study a contextual version of BwK with two arms , one of which does nothing ; this is meaningless when specialized to BwK . [ 44 ] , subsequent to our initial draft on arxiv.org , use extra parameters ( other than a version of reward-gap ) , which yield p T regret whenever our lower bounds apply ; 1 it is unclear when all their parameters are small . No worst-case regret bounds are provided ; their algorithm does not appear to achieve even o ( T ) regret in the worst case . Finally , [ 33 , 56 , 57 , 30 , 47 ] posit one constrained resource and T = 1 . This is an easier problem , e.g. , the best arm is the best distribution over arms . 2 Preliminaries : the problem , linear relaxation and UcbBwK algorithm The bandits with knapsacks ( BwK ) problem is as follows . There are K arms , d resources , and T rounds . Initially , each resource j 2 [ d ] is endowed with budget Bj . In each round t = 1 , . . . , T , an algorithm chooses an arm at , and observes an outcome vector ot = ( rt ; c1 , t , . . . , cd , t ) 2 [ 0 , 1 ] d+1 , where rt is the reward , and cj , t is the consumption of each resource j . The algorithm stops when the consumption of some resource j exceeds its budget Bj , or after T rounds , whichever is sooner . We maximize the total reward , REW = P⌧ t=1 rt , where ⌧ is the stopping time . We focus on the stochastic version : for each arm a , there is a distribution Da over [ 0 , 1 ] d+1 such that each outcome vector ot is 1Conceptually , our assumption of “ best-arm-optimality '' is replaced with another assumption : a lower bound on the positive entries of the optimal distribution x⇤ ( parameter in Section 3.3 of [ 44 ] ) . an independent draw from distribution Dat ( which depends only on the chosen arm at ) . A problem instance consists of parameters ( K , d , T ; B1 , . . . , Bd ) and distributions ( Da : arms a ) . Given a problem instance , the best dynamic policy benchmark OPTDP maximizes the total expected reward over all algorithms ; it is used in all worst-case regret bounds . The best fixed distribution benchmark OPTFD , used in some of our results , maximizes the total expected reward over all algorithms that always sample an arm from the same distribution . The worst-case optimal regret rate is [ 16 ] : OPTDP E [ REW ] = Õ ( p K OPTDP + OPTDP p K/B ) , B = minj2 [ d ] Bj . ( 2.1 ) Simplifications and notation . Following prior work , we make three assumptions without losing generality . First , all budgets are the same : B1 = . . . = Bd = B . This is w.l.o.g . because one can divide the consumption of each resource j by Bj/mini Bi ; dependence on the budgets is driven by the smallest Bj . Second , resource d corresponds to time : each arm deterministically consumes B/T units of this resource in each round . It is called the time resource and denoted time . Third , there is a null arm , denoted null , whose reward and consumption of all resources except time is always 0.2 Like most prior work on BwK , we use O ( · ) notation rather than track explicit constants in regret bounds . This improves clarity and emphasizes the more essential aspects of analyses and results . For n 2 N , let [ n ] = { 1 , . . . , n } and n = { all distributions on [ n ] } . Let [ K ] and [ d ] be , resp. , the set of all arms and the set of all resources . For each arm a , let r ( a ) and cj ( a ) be , resp. , the mean reward and mean resource-j consumption , i.e. , ( r ( a ) ; c1 ( a ) , . . . , cd ( a ) ) : = Eo⇠Da [ o ] . We sometimes write r = ( r ( a ) : a 2 [ K ] ) and cj = ( cj ( a ) : a 2 [ K ] ) as vectors over arms . Given a function f : [ K ] ! R , we extend it to distributions X over arms as f ( X ) : = Ea⇠X [ f ( a ) ] . Linear Relaxation . Following prior work , we consider a linear relaxation : maximize X · r such that X 2 [ 0 , 1 ] K , X · 1 = 1 8j 2 [ d ] X · cj B/T . ( 2.2 ) Here X is a distributions over arms , the algorithm does not run out of resources in expectation , and the objective is the expected per-round reward . Let OPTLP be the value of this linear program . Then OPTLP OPTDP/T OPTFD/T [ 16 ] . The Lagrange function L : K ⇥ Rd+ ! R defined as follows : L ( X , ) : = r ( X ) + P j2 [ d ] j [ 1 T/B cj ( X ) , ] . ( 2.3 ) where corresponds to the dual variables . Then ( e.g. , by Theorem D.2.2 in [ 17 ] ) : min 0 max X2 K L ( X , ) = max X2 K min 0 L ( X , ) = OPTLP . ( 2.4 ) The min and max in ( 2.4 ) are attained , so that ( X⇤ , ⇤ ) is maximin pair if and only if it is minimax pair ; such pair is called a saddle point . We ’ ll use L ( · , ⇤ ) to generalize reward-gap to BwK . Algorithm UcbBwK . We analyze an algorithm from [ 3 ] , defined as follows . In the LP ( 2.2 ) , rescale the last constraint , for each resource j 6= time , as ( B/T ) ( 1 ⌘LP ) , where ⌘LP : = 3 · ( p K/B log ( KdT ) + K/B ( log ( KdT ) ) 2 ) . ( 2.5 ) We call it the rescaled LP ( see ( C.1 ) ) . Its value is ( 1 ⌘LP ) OPTLP . At each round t , the algorithm forms an “ optimistic '' version of this LP , upper-bounding rewards and lower-bounding consumption : maximize P a2 [ K ] X ( a ) r + t ( a ) such that X 2 [ 0 , 1 ] K , P a2 [ K ] X ( a ) = 1 8j 2 [ d ] P a2 [ K ] X ( a ) c j , t ( a ) B ( 1 ⌘LP ) /T . ( 2.6 ) UcbBwK solves ( 2.6 ) , obtains distribution Xt , and samples an arm at independently from Xt . The algorithm achieves the worst-case optimal regret bound in ( 2.1 ) . The upper/lower confidence bounds r+t ( a ) , c j , t ( a ) 2 [ 0 , 1 ] are computed in a particular way specified in Appendix B . What matters to this paper is that they satisfy a high-probability event 0 r+t ( a ) r ( a ) Radt ( a ) and 0 cj ( a ) c j , t ( a ) Radt ( a ) , ( 2.7 ) 2Choosing the null arm is equivalent to skipping a round . One can take an algorithm ALG that uses null , and turn it into an algorithm that doesn ’ t : when ALG chooses null , just call it again until it doesn ’ t . for some confidence radius Radt ( a ) specified below . This event holds , simultaneously for all arms a , resources j and rounds t , with probability ( say ) at least 1 log ( KdT ) T 4 . For a 6= null , we can take Radt ( a ) = min ( 1 , p Crad/Nt ( a ) + Crad/Nt ( a ) ) , ( 2.8 ) where Crad = 3 · log ( KdT ) and Nt ( a ) is the number of rounds before t in which arm a has been chosen . There is no uncertainty on the time resource and the null arm , so we define c time , t ( · ) = B/T and Radt ( null ) = r+t ( null ) = c j , t ( null ) = 0 for all resources j 6= time .

Summary: This paper tackles the Bandits with Knapsacks (BwK) problems, where the learner is not just looking for the arm with per-round maximal expected reward, but has to take into account the consumption of ressources that can end prematurely the whole process. While existing work on BwK have already characterized the worst-case regret scaling in sqrt(T), this paper provides additional theoretical guarantees: problem depend upper bounds in log(T) for a UCB-like algorithm called UcbBwK, lower bounds matching the worst-case rate, as well as guarantees on the simple regret. Finally, when the problem instance presents some specific structure (e.g. in combinatorial semi-bandit, linear contextual bandits, multinomial-logit bandits), all these bounds for UcbBwK are improved.