An Efficient Pessimistic-Optimistic Algorithm for Stochastic Linear Bandits with General Constraints

´´ K0.75 δ ` d ¯ ? τ ¯ ( pseudo ) regret in round τ ď T , where K is the number of constraints , d is the dimension of the reward feature space , and δ is a Slater ’ s constant ; and zero constraint violation in any round τ ą τ 1 , where τ 1 is independent of horizon T. Second , the algorithm is computationally efficient . Our algorithm is based on the primal-dual approach in optimization and includes two components . The primal component is similar to unconstrained stochastic linear bandits ( our algorithm uses the linear upper confidence bound algorithm ( LinUCB ) ) . The computational complexity of the dual component depends on the number of constraints , but is independent of the sizes of the contextual space , the action space , and the feature space . Thus , the computational complexity of our algorithm is similar to LinUCB for unconstrained stochastic linear bandits . 1 Introduction . Stochastic linear bandits have a broad range of applications in practice , including online recommendations , job assignments in crowdsourcing , and clinical trials in healthcare . Most existing studies on stochastic linear bandits formulated them as unconstrained online optimization problems , limiting their application to problems with operational constraints such as safety , fairness , and budget constraints . In this paper , we consider a stochastic linear bandit with general constraints . As in a standard stochastic linear bandit , at the beginning of each round t P rT s , the learner is given a context cptq that is randomly sampled from the context set C ( a countable set ) , and takes an action Aptq P rJs . The learner then receives a reward Rpcptq , Aptqq “ rpcptq , Aptqq ` ηptq , where rpc , jq “ xθ˚ , φpc , jqy , φpc , jq P Rd is a d-dimensional feature vector for ( context , action ) pair pc , jq , θ˚ P Rd is an unknown underlying vector to be learned , and ηptq is a zero-mean random variable . For constrained stochastic linear bandits , we further assume when action Aptq is taken on context cptq , it incurs K different types of costs , denoted by W pkqpcptq , Aptqq . We assume W pkqpc , jq is a random variable with mean wpkqpc , jq that is unknown to the learner . This paper considers general cost functions and does not require wpkqpc , jq to have a linear form like rpc , jq . Denote the action taken by policy π in round t by Aπptq . The learner ’ s objective is to learn a policy π that maximizes the cumulative rewards over horizon T subject to anytime cumulative constraints : 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . max π E « T ÿ t “ 1 Rpcptq , Aπptqq ff ( 1 ) E « τ ÿ t “ 1 W pkq pcptq , Aπptqq ff ď 0 , @ τ P rT s , k P rKs . ( 2 ) The constraint ( 2 ) above may represent different operational constraints including safety , fairness , and budget constraints . Anytime cumulative constraints In the literature , constraints in stochastic bandits have been formulated differently . There are two popular formulations . The first one is a cumulative constraint over horizon T , including knapsack bandits [ 10 , 9 , 3 , 4 , 5 , 17 , 11 ] where the process terminates when the total budget has been consumed ; fair bandits where the number of times an action can be taken must exceed a threshold at the end of the horizon [ 12 ] ; and contextual bandits with a cumulative budget constraint [ 40 , 14 ] . In these settings , the feasible action set in each round depends on the history . In general , the learner has more flexibility in the earlier rounds , close to that in the unconstrained setting . Another formulation is anytime constraints , which either require the expected cost of the action taken in each round to be lower than a threshold [ 6 , 29 ] or the expected cost of the policy in each round is lower than a threshold [ 32 ] . We call them anytime action constraints and anytime policy constraints , respectively . Our constraint in the form of ( 2 ) is an anytime cumulative constraint , i.e. , it imposes a cumulative constraint in every round . This anytime cumulative constraint is most similar to the anytime policy constraint in [ 32 ] because the average cost of a policy is close to its mean after the policy has been applied for many rounds and the process converges , so it can be viewed as a cumulative constraint on actions over many rounds ( like ours ) . Furthermore , when our anytime cumulative constraint ( 2 ) is satisfied , it is guaranteed that the time-average cost is below a threshold in every round . In summary , our anytime cumulative constraint is stricter than a cumulative constraint over fixed horizon T but is less restrictive than anytime action constraint in [ 6 , 29 ] . Figure 1 provides a conceptual description of the relationship between these different forms of constraints . Main Contributions This paper presents a pessimistic-optimistic algorithm based on the primal-dual approach in optimization for the problem defined in ( 1 ) - ( 2 ) . The algorithm is efficient in two aspects . First , the algorithm yields Õ ´´ K0.75 δ ` d ¯ ? τ ¯ regret in round τ ď T and achieves zero constraint violation in any round τ ą τ 1 for a constant τ 1 independent of horizon T . Second , the algorithm is computationally efficient . For computational efficiency , the design of our algorithm is based on the primal-dual approach in optimization . The computation of the primary component is similar to unconstrained stochastic linear bandits [ 15 , 33 , 25 , 1 , 13 ] . The dual component includes a set of Lagrangian multipliers that are updated in a simple manner to keep track of the levels of constraint violations so far in each round ; the update depends on the number of constraints , but it is independent of the sizes of the contextual space , the action space , and the feature space . Thus , the overall computational complexity of our algorithm is similar to that of LinUCB in the unconstrained setting . This results in a much more efficient calculation comparing to OPLB proposed in [ 32 ] . OPLB needs to construct a safe policy set in each round , hence , its computational complexity is prohibitively high as the authors acknowledged . For constraint violation , our algorithm guarantees that for any τ ą τ 1 , the constraint holds with probability one . In other words , after a constant number of rounds , the constraint is always satisfied . This is in contrast to prior works [ 32 , 6 ] , where anytime constraints are proven to hold over horizon T with probability 1´ χ for a constant χ . In other words , the anytime constraints may be violated with probability χ , and it is not clear how often they are violated when it happens . Furthermore , beyond mean cost constraints considered in ( 2 ) and in [ 32 , 6 ] , we prove that a sample-path version of constraint ( 2 ) holds with probability 1´O ´ e´ δ ? τ 50K2.5 ¯ in round τ under our algorithm . To summarize , our algorithm is computationally efficient and provides strong guarantees on both regret and constraint violations . Additionally , our cost function is in a general form and does not need to be linear as those in [ 32 , 6 ] . We discuss more related work in the following . Related Work Stochastic linear bandits [ 2 , 8 ] are a special class of contextual bandits [ 39 , 23 ] , which generalize multi-armed bandits [ 22 ] . Besides [ 32 ] , [ 12 ] considered an adversarial contextual bandit with anytime policy constraint representing fairness . The proposed algorithm has Õp a |C|JT q regret when the context distribution is known to the learner ; otherwise it has Õp a |C|JT q regret and Õp a |C|T q constraint violation . [ 24 ] studied a combinatorial sleeping bandits problem under cumulative fairness constraints and proposed an algorithm based on UCB which they conjectured to have Õp ? T q regret and Õp ? T q constraint violation . Recent work studied unconstrained structured bandits and proposed primal-dual approach based on asymptotically lower bound problem in bandits [ 21 , 35 , 16 ] . However , our algorithm is different from them in three aspects . Our primal component is a greedy algorithm instead of a ( sub- ) gradient algorithm ( as in [ 35 ] ) . Our dual component does not solve a best response problem , which is a constrained optimization problem as in [ 21 , 16 ] . Our analysis is based on the Lyapunov-drift analysis for queueing systems , e.g. , we establish a bound on the exponential moment of the dual variable , which is not present in [ 21 , 35 , 16 ] . It is also worth mentioning that [ 20 , 19 , 30 ] studied “ conservative ” bandits which require that the reward or the cumulative reward exceeds a threshold at each step . Another line of related work is online convex optimization with constraints , studied in [ 28 , 27 , 41 , 42 , 38 , 36 ] , where online primal-dual with proximal regularized algorithms have been proposed to achieve Op ? T q regret and Op1q violation for static constraints in [ 41 ] and Op ? T q violation for stochastic constraints in [ 42 ] . Notation . fpnq “ Õpgpnqq denotes fpnq “ Opgpnqlogknq with k ą 0 ; rN s denotes the set t1 , 2 , ¨ ¨ ¨ , Nu ; x¨ , ¨y denotes the inner product ; p¨q : denotes the transpose of a vector or a matrix ; || ¨ || “ || ¨ ||2 , and ||x||Σ “ ? x : Σx . 2 A Pessimistic-Optimistic Algorithm . We consider a stochastic linear bandit over horizon T as described in the introduction . The learner ’ s objective is to maximize the cumulative reward over time horizon T subject to K anytime cumulative constraints as defined in ( 1 ) - ( 2 ) . To address the challenges on the unknown reward and cost in constraint , as well as the anytime cumulative constraints , we develop a pessimistic-optimistic algorithm based on the primal-dual approach for constrained optimization . We first give out the intuition of the algorithm and then provide the formal statement of the algorithm . To start , we consider a baseline , deterministic problem that replaces all the random variables with their expectations . Different from the conventional setup , we introduce a “ tightness ” constant ą 0 : max x ÿ cPC , jPrJs pcrpc , jqxc , j ( 3 ) s.t . ÿ jPrJs xc , j “ 1 , xc , j ě 0 , @ c P C , ( 4 ) ÿ cPC , jPrJs pcw pkqpc , jqxc , j ` ď 0 , @ k P rKs , ( 5 ) where xc , j can be viewed as the probability of taking action j on context c , and pc is the probability that context c is selected in each round . We will discuss in further details the importance of the tightness constant in Section 4 . The Lagrangian of the problem above is max x : ř j xc , j “ 1 , xc , jě0 ÿ c , j pcrpc , jqxc , j ´ ÿ k λpkq ¨ ˝ ÿ cPC , jPrJs pcw pkqpc , jqxc , j ` ˛ ‚ , ( 6 ) where λpkq is the Lagrange multiplier associated with the kth constraint in ( 5 ) . Fixing the values of the Lagrange multipliers , solving the optimization problem is equivalent to solving |C| separate subproblems ( 7 ) , one for each context c , because the optimization variables x are coupled through j only : max x : ř j xc , j “ 1 , xc , jě0 pc ˜ ÿ j rpc , jqxc , j ´ ÿ k λpkq ˜ ÿ j wpkqpc , jqxc , j ¸¸ . ( 7 ) Since the problem above is a linear programming , one of the optimal solutions is xc , j “ 1 for j “ j˚ and xc , j “ 0 otherwise , where j˚ P arg max j rpc , jq ´ ÿ k λpkqwpkqpc , jq ( 8 ) and a tie can be broken arbitrarily . If we call rpc , jq ´ ř k λ pkqwpkqpc , jq the action-value of context c , then the solution for fixed values of Lagrange multipliers is to choose an action with the highest action-value . We may view the action value here plays a similar role as the Q-value ( also called action-value function ) in Q-learning [ 37 ] . Now the challenges to find a solution according to ( 8 ) include : ( i ) both rpc , jq and wpkqpc , jq are unknown , and ( ii ) the optimal Lagrange multipliers λpkq are also unknown . To overcome these challenges , we develop a pessimistic-optimistic algorithm that • Uses LinUCB to estimate rpc , jq based on its linear structure . • Uses observed W pkqpcptq , jq to replace wpkqpcptq , jq at each round t. • Uses the following function to dynamically approximate the Lagrange multipliers ( Xjptq “ 1 if Aptq “ j ; otherwise Xjptq “ 0 ) : Qpkqpt ` 1q “ » –Qpkqptq ` ÿ jPrJs W pkqpcptq , jqXjptq ` t fi fl ` , @ k. In other words , we increase its value when the current cost exceeds the current “ budget , ” and decrease it otherwise . Therefore , Qpkqptq keeps track of the cumulative constraint violation by round t. • We further add a scaling parameter 1 { Vt to Qpkqptq , i.e . Qpkqptq { Vt , to approximate λpkq . With a carefully designed Vt , we can control the tradeoff between maximizing reward and minimizing constraint violation in the policy and achieve the regret and constraint violation bounds to be presented in the main theorem . Next , we formally state our algorithm . This algorithm takes the following information as input at the beginning of each round t : ( i ) historical observations Ft´1 “ tcpsq , Apsq , Rpcpsq , Apsqq , W pkqpcpsq , ApsqqusPrt´1s , kPrKs , ( ii ) current observations cpsq and tW pkqpcpsq , jqukPrKs , jPrJs , and ( iii ) system parameters : the feature map tφpc , jqucPC , jPrJs , time horizon T , and a pre-set constant δ . In the analysis of our algorithm , we will reveal the connection of this constant δ with Slater ’ s condition . The algorithm outputs the action in each round , observes reward Rpcptq , Aptqq , makes updates , and then moves to the next round t ` 1 .

Summary: This paper proposes a learning algorithm for a stochastic linear bandits with constraints. At each decision step, the decision maker can choose between J actions. Each action A_t then provides a context-dependent reward $r(C_t, A_t)$ and contributes a $W_k(C_t, A_t)$ to the "constraint k". The goal of the learner is to maximize the reward provided that the K constraint $\sum_{t=0}^\tau W_k(C_t, A_t) \le 0$ are respected.  The authors present an algorithm that guarantees a sublinear regret in $O(\sqrt{T})$ while respecting constraints for all time-steps except at most O(1) time-steps.  After presenting the algorithm, most of the paper is devoted to a careful analysis of the theoretical properties of the algorithm. No application nor experimental results are presented.