Learning Expensive Coordination: An Event-Based Deep RL Approach

1 INTRODUCTION . Deep Multi-Agent Reinforcement Learning ( MARL ) has been widely used in coordinating cooperative agents to jointly complete certain tasks where the agent is assumed to be selfless ( fully cooperative ) , i.e. , the agent is willing to sacrifice itself to maximize the team reward . However , in many cases of the real world , the agents are self-interested , such as taxi drivers in a taxi company ( fleets ) and clubs in a league . For instance , in the example of taxi fleets ( Miao et al. , 2016 ) , drivers may prefer to stay in the area with high customer demand to gain more reward . It is unfair and not efficient to compel the taxi driver to selflessly contribute to the company , e.g. , to stay in the low customer demand area . Forcing the drivers to selflessly contribute may increase the income for the company in a short-term but it will finally causes the low efficient and unsustainable of that company in the ∗Indicates equal contribution . †Co-corresponding authors . long run because the unsatisfied drivers may be demotivated and even leave the company . Another important example is that the government wants some companies to invest on the poverty area to achieve the fairness of the society , which may inevitably reduce the profits of companies . Similar to previous example , the companies may leave when the government forces them to invest . A better way to achieve coordination among followers and achieve the leader ’ s goals is that the manager of the company or the government needs to provide bonuses to followers , like the taxi company pays extra bonuses for serving the customers in rural areas and the government provides subsidies for investing in the poverty areas , which we term as expensive coordination . In this paper , we solve the large-scale sequential expensive coordination problem with a novel RL training scheme . There are several lines of works related to the expensive coordination problem , including mechanism design ( Nisan & Ronen , 2001 ) and the principal-agent model ( Laffont & Martimort , 2009 ) . However , these works focus more on static decisions ( each agent only makes a single decision ) . To consider sequential decisions , the leader-follower MDP game ( Sabbadin & Viet , 2013 ; 2016 ) and the RL-based mechanism design ( Tang , 2017 ; Shen et al. , 2017 ) are introduced but most of their works only focus on matrix games or small-scale Markov games , which can not be applied to the case with the large-scale action or state space . The most related work is M3RL ( Shu & Tian , 2019 ) where the leader assigns goals and bonuses by using a simple attention mechanism ( summing/averaging the features together ) and mind ( behaviors ) tracking to predict the followers ’ behaviors and makes response to the followers ’ behaviors . But they only consider the rule-based followers , i.e. , followers with fixed preference , and ignore the followers ’ behaviors responding to the leader ’ s policy , which significantly simplifies the problem and leads the unreasonability of the model . In the expensive coordination problem , there are two critical issues which should be considered : 1 ) the leader ’ s long-term decision process where the leader has to consider both the long-term effect of itself and long-term behaviors of the followers when determining his action to incentivise the coordination among followers , which is not considered in ( Sabbadin & Viet , 2013 ; Mguni et al. , 2019 ) ; and 2 ) the complex interactions between the leader and followers where the followers will adapt their policies to maximize their own utility given the leader ’ s policy , which makes the training process unstable and hard , if not unable , to converge in large-scale environment , especially when the leader changes his actions frequently , which is ignored by ( Tharakunnel & Bhattacharyya , 2007 ; Shu & Tian , 2019 ) . In this work , we address these two issues in the expensive coordination problem through an abstraction-based deep RL approach . Our main contributions are threefold . ( 1 ) We model the leader ’ s decision-making process as a semiMarkov Decision Process ( semi-MDP ) and propose a novel event-based policy gradient to learn the leader ’ s policy considering the long-term effect ( leader takes actions at important points rather than at each step to avoid myopic decisions . ) ( Section 4.1 ) . ( 2 ) A well-performing leader ’ s policy is also highly dependent on how well the leader knows the followers . To predict the followers ’ behaviors precisely , we show the leader-follower consistency scheme . Based on the scheme , the follower-aware module , the follower-specific attention module , and the sequential decision module are proposed to capture these followers ’ behaviors and make accurate response to their behaviors ( Section 4.2 ) . ( 3 ) To accelerate the training process , we propose an action abstraction-based policy gradient algorithm for the followers . This approach is able to reduce followers ’ decision space and thus simplifies the interaction between the leader and followers as well as accelerates the training process of followers ( Section 4.3 ) . Experiments in resource collections , navigation and predatorprey show that our method outperforms the state-of-the-art methods dramatically . 2 RELATED WORKS . Our works are closely related to leader-follower RL , temporal abstraction RL , and event-based RL . Leader-follower RL . The leader-follower RL targets at addressing the issue of expensive coordination where the leader wants to maximize the social benefit ( or the leader ’ s self-benefit ) by coordinating non-cooperative followers through providing them bonuses . Previous works have investigated different approaches to solve the expensive coordination , including the vanilla leader-follower MARL ( Sabbadin & Viet , 2013 ; Laumônier & Chaib-draa , 2005 ) , leader semi-MDP ( Tharakunnel & Bhattacharyya , 2007 ) , multiple followers and sub-followers MARL ( Cheng et al. , 2017 ) , followers abstraction ( Sabbadin & Viet , 2016 ) , and Bayesian optimization ( Mguni et al. , 2019 ) . But most of them focus on simple tabular games or small-scale Markov games . The most related work ( Shu & Tian , 2019 ) leverages the deep RL approach to compute the leader ’ s policy of assigning goals and bonuses to rule-based followers . But their method performs poorly when the followers are RLbased . In this work , we aim to compute the leader ’ s policy against the RL-based followers in the complex and sequential scenarios . Temporal abstraction RL . Our methods are also related to temporal abstraction method ( Sutton et al. , 1998 ; Daniel et al. , 2016 ; Bacon et al. , 2017 ; Smith et al. , 2018 ; Zhang & Whiteson , 2019 ; Vezhnevets et al. , 2016 ) . The basic idea of temporal abstraction is to divide the original one-level decision process into a two-level decision process where the high-level part is to decide the meta goal while the low-level policy is to select the primitive actions . Our leader ’ s decision process is different from those methods mentioned above because the leader ’ s policy can naturally form as an intermittent ( temporal abstraction ) decision process ( semi-MDP ) ( Tharakunnel & Bhattacharyya , 2007 ) and it is unnecessary to design the two-level decision process for the leader ( since the low-level decision process is the follower ) . Based on the nature of the leader , a novel training method is introduced . Event-based RL & Planning . Previous studies also focus on using events to capture important elements ( e.g. , whether agent reaches a goal ) during the whole episode . Upadhyay et al . ( 2018 ) regard the leader ’ s action and the environment feedback as events in the continuous time environment . Becker et al . ( 2004 ) ; Gupta et al . ( 2018 ) leverage events to capture the fact that an agent has accomplished some goals . We adopt this idea by depicting the event as the actions taken by the leader at some time steps and design a novel event-based policy gradient to learn the long-term leader ’ s policy . 3 STACKELBERG MARKOV GAMES . Our research focuses on single-leader multi-follower Stackelberg Markov Games ( SMG ) ( Mguni et al. , 2019 ; Sabbadin & Viet , 2013 ) , which can be formulated as a tupleG = 〈N , S , A , Ω , P , R , γ〉 . N is the set of N followers , i.e. , |N | = N . S is the set of states . s0 ∈ S0 ⊂ S is an initial state and S0 is the set of initial states . A = ×k∈NAk is the set of joint actions for followers where ak ∈ Ak is an action for the k-th follower . ω ∈ Ω = ×k∈NΩk is an action for the leader and ωk = { gk , bk } ∈ Ωk is a goal and a bonus that the leader assigns to the k-th follower . P : S × A → ∆ ( S ) is the transition function1 and R = ×k∈N rk × rl is the reward function set where rk : S × A × Ω → R is the reward function for the k-th follower and rl : S × A × Ω→ R is the reward function for the leader . γ is the discount factor and a is a joint action of followers . The leader ’ s policy is defined as µ = 〈µk〉k∈N where µk : Ω × S → ∆ ( Ωk ) is the leader ’ s action to the k-th follower given the leader ’ s action in the previous timestep ωt−1 and the current state st. ∆ ( · ) is a probability distribution . The followers ’ joint policy is defined as π = 〈πk〉 where πk : Ωk ×S → ∆ ( Ak ) is the k-th follower policy given the leader ’ s action ωkt and the current state st . Given the policy profile of the leader and followers 〈µ , π〉 , the follower ’ s utility is defined as Jk ( µ , π ) = E [ ∑T t=0 γ trkt ( st , at , ωt ) ] and the leader ’ s utility is J ( µ , π ) = E [ ∑T t=0 γ trlt ( st , at , ωt ) ] . We assume that the leader and followers aim to maximize their own utilities . We define the trajectory τ as a sequence of state , leader ’ s action , and followers ’ actions 〈ω−1 , ( st , at , ωt ) Tt=0〉 where ω−1 is the first step leader ’ s action and is set to zero . 1Notice that the transition function does not depend on the leader ’ s action . 4 METHODOLOGY . In this section , we propose a novel training scheme to train a well-performing leader policy against both rule-based and RL-based followers in the expensive coordination problem . We address the two issues , the leader ’ s long-term decision process and the complex interactions between the leader and followers , with three key steps : ( a ) we model the leader ’ s decision-making process as a semiMarkov Decision Process ( semi-MDP ) and propose a novel event-based policy gradient to take actions only at important time steps to avoid myopic policy ; ( b ) to accurately predict followers ’ behaviors , we construct a follower-aware module based on the leader-follower consistency , including a novel follower-specific attention mechanism , and a sequential decision module to predict followers ’ behaviors precisely and make accurate response to these behaviors ; and ( c ) an action abstractionbased policy gradient method for followers is proposed to simplify the decision process for the followers and thus simplify the interaction between leader and followers , and accelerate the convergence of the training process .

Summary: This paper aims to improve the performance of leader agents in leader-follower Stackelberg Markov games (SMG) by introducing architectural components. an event-aware policy gradient for the temporally abstracted level of the leader, and a follower abstraction technique that mitigates the impact of follower distributional shift. The authors show significant improvements over the state of the art on several problems, and also evaluate the importance of the different components of their approach with thorough ablations.