Congested bandits: Optimal routing via short-term resets

For traffic routing platforms , the choice of which route to recommend to a user depends on the congestion on these routes – indeed , an individual ’ s utility depends on the number of people using the recommended route at that instance . Motivated by this , we introduce the problem of Congested Bandits where each arm ’ s reward is allowed to depend on the number of times it was played in the past ∆ timesteps . This dependence on past history of actions leads to a dynamical system where an algorithm ’ s present choices also affect its future pay-offs , and requires an algorithm to plan for this . We study the congestion aware formulation in the multi-armed bandit ( MAB ) setup and in the contextual bandit setup with linear rewards . For the multi-armed setup , we propose a UCB style algorithm and show that its policy regret scales as Õ ( √ K∆T ) . For the linear contextual bandit setup , our algorithm , based on an iterative least squares planner , achieves policy regret Õ ( √ dT + ∆ ) . From an experimental standpoint , we corroborate the no-regret properties of our algorithms via a simulation study . 1 INTRODUCTION . The online multi-armed bandit ( MAB ) problem and its extensions have been widely studied and used to model many real world scenarios ( Robbins , 1952 ; Auer et al. , 2002 ; 2001 ) . In the basic MAB setup there are K arms with either stochastic or adversarially chosen reward profiles . The goal is to design an algorithm that achieves a cumulative reward that is as good as that of the best arm in hindsight . This is quantified in terms of the regret achieved by the algorithm over T time steps ( see Section 2 for formal definitions ) . In many real world scenarios the MAB setup as described above is not suitable as the reward obtained by playing an arm/action at a given time step may depend on the algorithm ’ s choices in the previous time steps . In particular , we are motivated by online routing problems where the reward of suggesting a particular edge to traverse along a path from source to destination often depends on the congestion on that edge . This congestion is a function of the number of times the particular edge has been recommended earlier ( potentially in a time window ) . In such scenarios , one would desire algorithms that can compete with the best policy , i.e. , the best sequence of actions , in hindsight as compared to a fixed best action . Classical multi-armed bandit formulation and associated no-regret algorithms ( Slivkins , 2019 ) , or their extensions to routing problems ( Kalai & Vempala , 2005 ; Awerbuch & Kleinberg , 2008 ) do not suffice for the above scenario as they only guarantee competitiveness with respect to the best fixed arm in hindsight . To overcome these limitations , we propose a new model , viz. , congested bandits which captures the above scenario . In our proposed model the reward of a given arm at each time step depends on how many times the arm has been played within a given time window of size ∆ . Hence over time , an arm ’ s expected reward may decay and reset dynamically . While our model is motivated by online routing problems , our proposed formulation is very general . As another example , consider a digital music platform that recommends artists to its end users . In order to maximize profit the recommendation algorithm may prefer to recommend popular artists , and at the same time the platform may want to promote equity and diversity by highlighting new and emerging artists as well . This scenario can be model via congested bandits where each artist is an arm and the reward for suggesting an artist is a function of how many times the artist has been recommended in the past time window of length ∆ . In both the scenarios above , the ability to reset the congestion cost ( by simply not playing an arm for ∆ time steps ) is a crucial part of the problem formulation . Our contributions . We propose and study the congested bandits model with short term resets under a variety of settings and design no-regret algorithms . In the most basic setup we consider aK-armed stochastic bandit problem where each arm a has a mean reward µa , and the mean reward obtained by playing arm a at time t equals fcong ( a , ht ) µa . Here ht denotes the history of the algorithm ’ s choices within the last ∆ time steps and fcong is a non-increasing congestion function . Recall that the algorithm ’ s goal in this setup is to compete with the best policy in hindsight . While the above setting can be formulated as a Markov Decision Process ( MDP ) , existing no regret algorithms for MDPs will incur a regret bound that scales exponential in the parameters ( scaling as K∆ ) ( Auer et al. , 2009 ; Jaksch et al. , 2010 ) . Instead , we carefully exploit the problem structure to design an algorithm with near-optimal regret scaling as Õ ( √ K∆T ) . Next , we extend our model to the case of online routing with congested edge rewards , again presenting near optimal no-regret algorithms that avoid exponential dependence on the size of the graph in the regret bounds . We then extend the multi-armed and the congested online routing formulations to a contextual setting where the mean reward for each arm/edge at a given time step equals fcong ( a , ht ) 〈θ∗ , φ ( xt , a ) ) 〉 , where xt ∈ Rd is a context vector and θ∗ is an unknown parameter . This extension is inspired from classical work in contextual bandits ( Li et al. , 2010 ; Chu et al. , 2011 ) and captures scenarios where users may have different preferences over actions/arms ( e.g. , a user who wants to avoid routes with tolls ) . Solving the contextual case poses significant hurdles as a priori it is not even clear whether the setting can be captured via an MDP . By exploiting the structure of the problem , we present a novel epoch based algorithm that , at each time step , plays a near optimal policy by planning for the next epoch . Showing that such planning can be done when only given access to the distribution of the contexts is a key technical step in establishing the correctness of the algorithm . As a result , we obtain algorithms that achieve Õ ( √ dT + ∆ ) regret in the contextual MAB setting . Finally , using simulations , we perform an empirical evaluation of the effectiveness of our proposed algorithms . Related work . Closest to our work are studies on multi-armed bandits with decaying and/or improving costs . The work of ( Levine et al. , 2017 ) proposes the rotting bandits model that has been further studied in ( Seznec et al. , 2019 ) . In this model the mean reward of each arm decays in a monotonic way as a function of the number of times the arm has been played in the past . There is no notion of a short-term reset as in our setting . As a result it can be shown that a simply greedy policy is optimal in hindsight . In contrast , in our setting greedy approaches can fail miserably as highlighted in Figure 1 . The work of Heidari et al . ( 2016 ) considers a setting where the mean reward of an arm can either improve or decay as a function of the number of times it has been played . However , similar to the rotting bandits setup there is no notion of a reset . Pike-Burke & Grünewälder ( 2019 ) considers a notion of reset/improvement by allowing the mean reward to depend on the number of time steps since an arm was last played . Their work considers a Bayesian setup where the mean reward function is modelled by a draw from a Gaussian process . On the other hand , in our work , the reward is a function of the number of times an arm was played in the past ∆ time steps . However their regret bounds are either with respect to the instantaneous regret , or with respect to a weaker policy class of d-step look ahead policies . The notion of instantaneous regret only compares with the class of greedy policies at each timestep as compared to the globally optimal policy . There also exist works studying the design of online learning algorithms againstm-bounded memory adversaries . The assumption here is that the reward of an action at each time step depends only on the previous m actions . While this is also true for our setting , in general the regret bounds provided for bounded-memory adversaries are either with respect to a fixed action or a policy class containing policies that do no switch often ( Arora et al. , 2012 ; Anava et al. , 2015 ) . Our problem formulation for the contextual setup is reminiscent of the recent line of works on contextual MDPs ( Azizzadenesheli et al. , 2016 ; Krishnamurthy et al. , 2016 ; Hallak et al. , 2015 ; Modi & Tewari , 2020 ) . However these works either assume that the context vector is fixed for a given episode ( allowing for easier planning ) , or make strong realizability assumptions on the optimal Q-function . Finally , our congested bandits formulation of the routing problem is a natural extension of classical work on the online shortest path problem Kalai & Vempala ( 2005 ) ; Awerbuch & Kleinberg ( 2008 ) ; Dani et al . ( 2007 ) . 2 CONGESTED MULTI-ARMED BANDIT . In this section , we model the congestion problem in a multi-armed bandit ( MAB ) framework.Our setup for congested multi-armed bandits models the congestion phenomenon by allowing the rewards of an arm to depend on the number of times it was played in the past . Let us consider a MAB setup with K arms , where each arm may represent a possible route . Let us denote by ∆ the size of the window which affects the reward at the current time step . For any time t , let ht ∈ H∆ : = [ K ] ∆ denote the history of the actions taken by an algorithm1 in the past ∆ time steps , that is , ht = [ at−∆ , . . . , at−1 ] where aτ is the action chosen by the algorithm at time τ . In order to model congestion arising from repeated plays of a single arm , we consider a function fcong : [ K ] × [ ∆ ] + → ( 0 , 1 ] . This congestion function takes in two arguments : an arm a and the number of times this arm was played in the past ∆ time steps , and outputs a value indicating the decay in the reward of arm a arising from congestion . Protocol for congestion in bandits . We consider the following online learning protocol for a learner in our congested MAB framework : At each round t , the learner picks an arm at ∈ [ K ] and observes reward r̃ ( ht , at ) = fcong ( at , # ( ht , at ) ) · µat + t where t ∼ N ( 0 , 1 ) ) . Finally , the history changes to ht+1 = [ ht,2 : ∆ , at ] where we have used the notation ht , i : j to denote the vector [ ht ( i ) , . . . , ht ( j ) ] and # ( h , a ) to denote the number of times the action a was played in history h. Each arm a is associated with a mean reward vector µa and the congestion function multiplicatively decreases the reward of that arm . We assume that the learner does not know the exact form of the function fcong as well as the the mean vector µ = { µa } [ K ] . The objective of the learner is to select the actions at which minimizes a notion of policy regret , which we define next . Policy regret for congested MAB . In the standard MAB setup , regret compares the cumulative reward of the algorithm to the benchmark of playing the arm with the highest mean reward at all time steps . However , as described in Section 1 , this benchmark is not suitable for our setup . Indeed , the asymptotically optimal algorithm is one which maximizes the average cumulative reward ρ∗ : = maxalg limT→∞ 1 T ∑T t=1 r ( ht , at ) , where the action at is the one chosen by the algorithm alg and history ht is the sequence of actions in the past ∆ time steps . This asymptotically algorithm corresponds to a stationary policy π∗ whose selection at only depends on the history ht . Accordingly , we consider the following class of stationary policies as the comparator for our regret Π = { π : H∆ → [ K ] } , with size |Π| = KK ∆ . Denote by hπt the history at time t by running policy π up to time t , we define the policy regret for any algorithm RT ( alg ; Π , fcong ) : = sup π∈Π T∑ t=1 r ( hπt , π ( h π t ) ) − T∑ t=1 r̃ ( halgt , at ) , ( 1 ) 1We usually suppress the dependence of this history on the algorithm , but make it explicit whenever it is not clear from context . Algorithm 1 : CARMAB : Congestion Aware Routing via Multi-Armed Bandits Input : Congestion window ∆ , confidence parameter δ ∈ ( 0 , 1 ) , action set [ K ] , time horizon T . Initialize : Set t = 1 for episodes e = 1 , . . . , E do Initialize episode Set start time of episode te = t. For all actions a and historical count j , set ne ( a , j ) = 0 and Ne ( a , j ) = ∑ s < e ns ( a , j ) . Set empirical reward estimate for each arm and historical count r̂e ( a , j ) = ∑te−1 τ=1 rτ · I [ aτ = a , jτ = j ] max { 1 , Ne ( a , j ) } . Compute optimistic policy Set the feasible rewards Re = { r ∈ [ 0 , 1 ] A× ( ∆+1 ) | for all ( a , j ) , |r ( a , j ) − r̂e ( a , j ) | ≤ 10 √ log ( A∆te/δ ) max { 1 , Ne ( a , j ) } } Find optimistic policy π̃e = arg maxπ∈Π , r∈Re ρ ( π , r ) Execute optimistic policy while ne ( a , j ) < max { 1 , Ne ( a , j ) } do Select arm at = π̃e ( st ) , obtain reward r̂t . Update ne ( a , # ( st , a ) ) = ne ( a , # ( st , a ) ) + 1. where at is the action chosen by alg at time t and r ( ht , at ) = fcong ( ht , ) · µat . This notion of regret is called policy regret ( Arora et al. , 2012 ) because the history sequence observed by the algorithm π∗ and the algorithm alg can be different from each other – this leads to a situation where choosing the same action a at time t can lead to different rewards for the algorithm and the comparator .

Summary: This paper proposes a novel multi-armed bandit framework to formulate the scenario of route recommendation based on dynamic traffic situations. The formulated model fits in both stochastic bandits and linear contextual bandits.  The paper first introduces the stochastic bandit setting with time-varying expected reward for each arm, which depends on the pulling history within a fixed-sized sliding window. The authors propose an algorithm CARMAB, with regret analysis indicating that the regret upper bound scales as $ O(\sqrt{K\Delta T\log(\Delta KT)})$ with high probability.  Then the paper discusses the contextual bandit setting with known contexts. The authors updated the model and propose an algorithm CARCB with regret analysis indicating the regret upper bound scales as $\tilde O(\sqrt{dT+\Delta})$ with high probability. Then the authors further relax the assumption on contexts by allowing unknown stochastic contexts samples from a known distribution. The regret analysis indicates an additional term with respect to  $T$ comparing to the known contexts setting.