Bandits with Knapsacks beyond the Worst Case

1 Introduction . We study multi-armed bandit problems with supply or budget constraints . Multi-armed bandits is a simple model for exploration-exploitation tradeoff , i.e. , the tension between acquiring new information and making optimal decisions . It is an active research area , spanning computer science , operations research , and economics . Supply/budget constraints arise in many realistic applications , e.g. , a seller who dynamically adjusts the prices or product assortment may have a limited inventory , and an algorithm that optimizes ad placement is constrained by the advertisers ’ budgets . Other motivating examples concern repeated auctions , crowdsourcing markets , and network routing . We consider a general model called Bandits with Knapsacks ( BwK ) , which subsumes the examples mentioned above . There are d 2 resources that are consumed over time , one of which is time itself . Each resource i starts out with budget Bi . In each round t , the algorithm chooses an action ( arm ) a = at from a fixed set of K actions . The outcome is a vector in [ 0 , 1 ] d+1 : it consists of a reward and consumption of each resource . This vector is drawn independently from some distribution over [ 0 , 1 ] d+1 , which depends on the chosen arm but not on the round , and is not known to the algorithm . The algorithm observes bandit feedback , i.e. , only the outcome of the chosen arm . The algorithm stops at a known time horizon T , or when the total consumption of some resource exceeds its budget . The goal is to maximize the total reward , denoted REW . The presence of supply/budget constraints makes the problem much more challenging . First , algorithm ’ s choices constrain what it can do in the future . Second , the algorithm is no longer looking for arms with maximal expected per-round reward ( because such arms may consume too much resources ) . Third , the best fixed distribution over arms can be much better than the best fixed arm . Accordingly , we compete with the best fixed distribution benchmark : the total expected reward of the best distribution , denoted OPTFD . All this complexity is already present even when d = 2 , i.e. , when there is only one resource other than time , and the minimal budget is B = mini Bi = ⌦ ( T ) . BwK were introduced in [ 14 , 16 ] and extensively studied since then . The optimal worst-case regret rate is well-understood . In particular , it is Õ ( p KT ) when B = ⌦ ( T ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We present several results that go beyond the worst-case perspective : 1 . We provide a full characterization for instance-dependent regret rates . In stochastic bandits , one obtains regret O K log T , where is the the reward-gap : the gap in expected reward between the best and the second-best arm . We work out whether , when and how such results extend to BwK . 2 . We show that simple regret , which tracks algorithm ’ s performance in a given round , can be small in all but a few rounds . Like in stochastic bandits , simple regret can be at least ✏ in at most Õ ( K/✏2 ) rounds , and this is achieved for all ✏ > 0 simultaneously . 3 . We improve all results mentioned above for a large number of arms , assuming some helpful structure . In fact , we provide a general “ reduction '' from BwK to stochastic bandits , and apply this reduction to three well-studied scenarios from stochastic bandits . Our algorithmic results focus on UcbBwK , a BwK algorithm from [ 3 ] which implements the “ optimism under uncertainty '' paradigm and attains the optimal worst-case regret bound . We provide new analyses of this algorithm along the above-mentioned themes . Related work . Background on multi-armed bandits can be found in books [ 23 , 54 , 42 ] . Stochastic bandits ( i.e. , BwK without resources ) is a basic , well-understood version . The dependence on and ✏ are optimal as stated above [ 41 , 10 , 11 ] , and is achieved simultaneously with the optimal worst-case regret eO ( KT ) , e.g. , in [ 10 ] . Various refinements are known for O ( log T ) regret [ 10 , 8 , 34 , 32 , 45 ] . Most relevant to this paper is O ( P a log ( T ) / ( a ) ) regret , where ( a ) is the gap in expected reward between arm a and the best arm [ 10 ] . Improving regret for large / infinite number of arms via a helpful structure is a unifying theme for several prominent lines of work , e.g. , linear bandits , convex bandits , Lipschitz bandits , and combinatorial ( semi- ) bandits . Bandits with Knapsacks were introduced in [ 14 , 16 ] , and optimally solved in the worst case . Subsequent work extended BwK to a more general notion of rewards/consumptions [ 3 ] , combinatorial semibandits [ 49 ] , and contextual bandits [ 15 , 6 , 4 ] . Several special cases with budget/supply constraints were studied separately ( and inspired a generalization to BwK ) : dynamic pricing [ 19 , 12 , 20 , 59 ] , dynamic procurement [ 13 , 52 ] , and dynamic ad allocation [ 53 , 28 ] . The adversarial version of BwK was studied by [ 35 , 36 ] . All this work considers worst-case regret bounds . Several papers achieve O ( log T ) regret in BwK , but with substantial caveats that we avoid . [ 61 ] assume deterministic consumption , whereas all motivating examples of BwK require stochastic consumption correlated with rewards ( e.g. , dynamic pricing consumes supply only if a sale happens ) . They posit d = 2 and no other assumptions , whereas we show that “ best-arm optimality '' is necessary with stochastic consumption . [ 31 ] assume “ best-arm-optimality '' as we do ( it is implicit in their version of reward-gap ) . However , their algorithm inputs an instance-dependent parameter which is “ hidden '' in BwK . Moreover , their O ( log T ) regret bound scales with cmin , minimal expected consumption among arms ( as c 4min ) . Their worst-case regret bound is suboptimal , since it also scales with cmin ( as c 2 min ) , and only applies for d = 2 . [ 58 ] study a contextual version of BwK with two arms , one of which does nothing ; this is meaningless when specialized to BwK . [ 44 ] , subsequent to our initial draft on arxiv.org , use extra parameters ( other than a version of reward-gap ) , which yield p T regret whenever our lower bounds apply ; 1 it is unclear when all their parameters are small . No worst-case regret bounds are provided ; their algorithm does not appear to achieve even o ( T ) regret in the worst case . Finally , [ 33 , 56 , 57 , 30 , 47 ] posit one constrained resource and T = 1 . This is an easier problem , e.g. , the best arm is the best distribution over arms . 2 Preliminaries : the problem , linear relaxation and UcbBwK algorithm The bandits with knapsacks ( BwK ) problem is as follows . There are K arms , d resources , and T rounds . Initially , each resource j 2 [ d ] is endowed with budget Bj . In each round t = 1 , . . . , T , an algorithm chooses an arm at , and observes an outcome vector ot = ( rt ; c1 , t , . . . , cd , t ) 2 [ 0 , 1 ] d+1 , where rt is the reward , and cj , t is the consumption of each resource j . The algorithm stops when the consumption of some resource j exceeds its budget Bj , or after T rounds , whichever is sooner . We maximize the total reward , REW = P⌧ t=1 rt , where ⌧ is the stopping time . We focus on the stochastic version : for each arm a , there is a distribution Da over [ 0 , 1 ] d+1 such that each outcome vector ot is 1Conceptually , our assumption of “ best-arm-optimality '' is replaced with another assumption : a lower bound on the positive entries of the optimal distribution x⇤ ( parameter in Section 3.3 of [ 44 ] ) . an independent draw from distribution Dat ( which depends only on the chosen arm at ) . A problem instance consists of parameters ( K , d , T ; B1 , . . . , Bd ) and distributions ( Da : arms a ) . Given a problem instance , the best dynamic policy benchmark OPTDP maximizes the total expected reward over all algorithms ; it is used in all worst-case regret bounds . The best fixed distribution benchmark OPTFD , used in some of our results , maximizes the total expected reward over all algorithms that always sample an arm from the same distribution . The worst-case optimal regret rate is [ 16 ] : OPTDP E [ REW ] = Õ ( p K OPTDP + OPTDP p K/B ) , B = minj2 [ d ] Bj . ( 2.1 ) Simplifications and notation . Following prior work , we make three assumptions without losing generality . First , all budgets are the same : B1 = . . . = Bd = B . This is w.l.o.g . because one can divide the consumption of each resource j by Bj/mini Bi ; dependence on the budgets is driven by the smallest Bj . Second , resource d corresponds to time : each arm deterministically consumes B/T units of this resource in each round . It is called the time resource and denoted time . Third , there is a null arm , denoted null , whose reward and consumption of all resources except time is always 0.2 Like most prior work on BwK , we use O ( · ) notation rather than track explicit constants in regret bounds . This improves clarity and emphasizes the more essential aspects of analyses and results . For n 2 N , let [ n ] = { 1 , . . . , n } and n = { all distributions on [ n ] } . Let [ K ] and [ d ] be , resp. , the set of all arms and the set of all resources . For each arm a , let r ( a ) and cj ( a ) be , resp. , the mean reward and mean resource-j consumption , i.e. , ( r ( a ) ; c1 ( a ) , . . . , cd ( a ) ) : = Eo⇠Da [ o ] . We sometimes write r = ( r ( a ) : a 2 [ K ] ) and cj = ( cj ( a ) : a 2 [ K ] ) as vectors over arms . Given a function f : [ K ] ! R , we extend it to distributions X over arms as f ( X ) : = Ea⇠X [ f ( a ) ] . Linear Relaxation . Following prior work , we consider a linear relaxation : maximize X · r such that X 2 [ 0 , 1 ] K , X · 1 = 1 8j 2 [ d ] X · cj B/T . ( 2.2 ) Here X is a distributions over arms , the algorithm does not run out of resources in expectation , and the objective is the expected per-round reward . Let OPTLP be the value of this linear program . Then OPTLP OPTDP/T OPTFD/T [ 16 ] . The Lagrange function L : K ⇥ Rd+ ! R defined as follows : L ( X , ) : = r ( X ) + P j2 [ d ] j [ 1 T/B cj ( X ) , ] . ( 2.3 ) where corresponds to the dual variables . Then ( e.g. , by Theorem D.2.2 in [ 17 ] ) : min 0 max X2 K L ( X , ) = max X2 K min 0 L ( X , ) = OPTLP . ( 2.4 ) The min and max in ( 2.4 ) are attained , so that ( X⇤ , ⇤ ) is maximin pair if and only if it is minimax pair ; such pair is called a saddle point . We ’ ll use L ( · , ⇤ ) to generalize reward-gap to BwK . Algorithm UcbBwK . We analyze an algorithm from [ 3 ] , defined as follows . In the LP ( 2.2 ) , rescale the last constraint , for each resource j 6= time , as ( B/T ) ( 1 ⌘LP ) , where ⌘LP : = 3 · ( p K/B log ( KdT ) + K/B ( log ( KdT ) ) 2 ) . ( 2.5 ) We call it the rescaled LP ( see ( C.1 ) ) . Its value is ( 1 ⌘LP ) OPTLP . At each round t , the algorithm forms an “ optimistic '' version of this LP , upper-bounding rewards and lower-bounding consumption : maximize P a2 [ K ] X ( a ) r + t ( a ) such that X 2 [ 0 , 1 ] K , P a2 [ K ] X ( a ) = 1 8j 2 [ d ] P a2 [ K ] X ( a ) c j , t ( a ) B ( 1 ⌘LP ) /T . ( 2.6 ) UcbBwK solves ( 2.6 ) , obtains distribution Xt , and samples an arm at independently from Xt . The algorithm achieves the worst-case optimal regret bound in ( 2.1 ) . The upper/lower confidence bounds r+t ( a ) , c j , t ( a ) 2 [ 0 , 1 ] are computed in a particular way specified in Appendix B . What matters to this paper is that they satisfy a high-probability event 0 r+t ( a ) r ( a ) Radt ( a ) and 0 cj ( a ) c j , t ( a ) Radt ( a ) , ( 2.7 ) 2Choosing the null arm is equivalent to skipping a round . One can take an algorithm ALG that uses null , and turn it into an algorithm that doesn ’ t : when ALG chooses null , just call it again until it doesn ’ t . for some confidence radius Radt ( a ) specified below . This event holds , simultaneously for all arms a , resources j and rounds t , with probability ( say ) at least 1 log ( KdT ) T 4 . For a 6= null , we can take Radt ( a ) = min ( 1 , p Crad/Nt ( a ) + Crad/Nt ( a ) ) , ( 2.8 ) where Crad = 3 · log ( KdT ) and Nt ( a ) is the number of rounds before t in which arm a has been chosen . There is no uncertainty on the time resource and the null arm , so we define c time , t ( · ) = B/T and Radt ( null ) = r+t ( null ) = c j , t ( null ) = 0 for all resources j 6= time .

Summary: The paper introduces a new algorithm for the BwK Problem achieving $O(\log T)$ regret for a "best-arm-optimal" instance. The algorithm is based on a variant of UcbBwK. Moreover, they show $\Omega(\sqrt{T})$ lower bound and they provide a general reduction from BwK to stochastic bandits.  Pros: (i) Characterizing the problem-dependent lower bound by a new quantity, Lagrangian gap. In the language of linear programming (LP), this is related to minimal non-zero reduced cost for the underlying LP that does not include the null arm. (ii) Better bound. From certain "best-arm-optimal" instances, their regret bound is better than the existing work.  (iii) One-round performance. It shows that the rewards collected by UcbBwK are approximately optimal for all but a few rounds.   (iv) Reduction from BwK to stochastic bandits.