Title: LEARNING EXPENSIVE COORDINATION: AN EVENT-BASED DEEP RL APPROACH

Abstract: Existing works in deep Multi-Agent Reinforcement Learning (MARL) mainly focus on coordinating cooperative agents to complete certain tasks jointly. However, in many cases of the real world, agents are self-interested such as employees in a company and clubs in a league. Therefore, the leader, i.e., the manager of the company or the league, needs to provide bonuses to followers for efficient coordination, which we call expensive coordination. The main difficulties of expensive coordination are that i) the leader has to consider the long-term effect and predict the followers' behaviors when assigning bonuses, and ii) the complex interactions between followers make the training process hard to converge, especially when the leader's policy changes with time. In this work, we address this problem through an event-based deep RL approach. Our main contributions are threefold.
(1) We model the leader's decision-making process as a semi-Markov Decision Process and propose a novel multi-agent event-based policy gradient to learn the leader's long-term policy. (2) We exploit the leader-follower consistency scheme to design a follower-aware module and a follower-specific attention module to predict the followers' behaviors and make accurate response to their behaviors.
(3) We propose an action abstraction-based policy gradient algorithm to reduce the followers' decision space and thus accelerate the training process of followers. Experiments in resource collections, navigation, and the predator-prey game reveal that our approach outperforms the state-of-the-art methods dramatically.

1. INTRODUCTION:

Deep Multi-Agent Reinforcement Learning (MARL) has been widely used in coordinating cooperative agents to jointly complete certain tasks where the agent is assumed to be selfless (fully cooperative), i.e., the agent is willing to sacrifice itself to maximize the team reward. However, in many cases of the real world, the agents are self-interested, such as taxi drivers in a taxi company (fleets) and clubs in a league. For instance, in the example of taxi fleets (Miao et al., 2016) , drivers may prefer to stay in the area with high customer demand to gain more reward. It is unfair and not efficient to compel the taxi driver to selflessly contribute to the company, e.g., to stay in the low customer demand area. Forcing the drivers to selflessly contribute may increase the income for the company in a short-term but it will finally causes the low efficient and unsustainable of that company in the long run because the unsatisfied drivers may be demotivated and even leave the company. Another important example is that the government wants some companies to invest on the poverty area to achieve the fairness of the society, which may inevitably reduce the profits of companies. Similar to previous example, the companies may leave when the government forces them to invest. A better way to achieve coordination among followers and achieve the leader's goals is that the manager of the company or the government needs to provide bonuses to followers, like the taxi company pays extra bonuses for serving the customers in rural areas and the government provides subsidies for investing in the poverty areas, which we term as expensive coordination. In this paper, we solve the large-scale sequential expensive coordination problem with a novel RL training scheme.

There are several lines of works related to the expensive coordination problem, including mechanism design (Nisan & Ronen, 2001) and the principal-agent model (Laffont & Martimort, 2009) . However, these works focus more on static decisions (each agent only makes a single decision). To consider sequential decisions, the leader-follower MDP game (Sabbadin & Viet, 2013; 2016) and the RL-based mechanism design (Tang, 2017; Shen et al., 2017) are introduced but most of their works only focus on matrix games or small-scale Markov games, which cannot be applied to the case with the large-scale action or state space. The most related work is M 3 RL (Shu & Tian, 2019) where the leader assigns goals and bonuses by using a simple attention mechanism (summing/averaging the features together) and mind (behaviors) tracking to predict the followers' behaviors and makes response to the followers' behaviors. But they only consider the rule-based followers, i.e., followers with fixed preference, and ignore the followers' behaviors responding to the leader's policy, which significantly simplifies the problem and leads the unreasonability of the model.

In the expensive coordination problem, there are two critical issues which should be considered: 1) the leader's long-term decision process where the leader has to consider both the long-term effect of itself and long-term behaviors of the followers when determining his action to incentivise the coordination among followers, which is not considered in (Sabbadin & Viet, 2013; Mguni et al., 2019) ; and 2) the complex interactions between the leader and followers where the followers will adapt their policies to maximize their own utility given the leader's policy, which makes the training process unstable and hard, if not unable, to converge in large-scale environment, especially when the leader changes his actions frequently, which is ignored by (Tharakunnel & Bhattacharyya, 2007; Shu & Tian, 2019) . In this work, we address these two issues in the expensive coordination problem through an abstraction-based deep RL approach.

Our main contributions are threefold. (1) We model the leader's decision-making process as a semi-Markov Decision Process (semi-MDP) and propose a novel event-based policy gradient to learn the leader's policy considering the long-term effect (leader takes actions at important points rather than at each step to avoid myopic decisions.) (Section 4.1). (2) A well-performing leader's policy is also highly dependent on how well the leader knows the followers. To predict the followers' behaviors precisely, we show the leader-follower consistency scheme. Based on the scheme, the follower-aware module, the follower-specific attention module, and the sequential decision module are proposed to capture these followers' behaviors and make accurate response to their behaviors (Section 4.2). (3) To accelerate the training process, we propose an action abstraction-based policy gradient algorithm for the followers. This approach is able to reduce followers' decision space and thus simplifies the interaction between the leader and followers as well as accelerates the training process of followers (Section 4.3). Experiments in resource collections, navigation and predatorprey show that our method outperforms the state-of-the-art methods dramatically.

2. RELATED WORKS:

Our works are closely related to leader-follower RL, temporal abstraction RL, and event-based RL.

Leader-follower RL. The leader-follower RL targets at addressing the issue of expensive coordination where the leader wants to maximize the social benefit (or the leader's self-benefit) by coordinating non-cooperative followers through providing them bonuses. Previous works have investigated different approaches to solve the expensive coordination, including the vanilla leader-follower MARL (Sabbadin & Viet, 2013; Laumônier & Chaib-draa, 2005) , leader semi-MDP (Tharakunnel & Bhattacharyya, 2007) , multiple followers and sub-followers MARL (Cheng et al., 2017) , followers abstraction (Sabbadin & Viet, 2016) , and Bayesian optimization (Mguni et al., 2019) . But most of them focus on simple tabular games or small-scale Markov games. The most related work (Shu & Tian, 2019) leverages the deep RL approach to compute the leader's policy of assigning goals and bonuses to rule-based followers. But their method performs poorly when the followers are RLbased. In this work, we aim to compute the leader's policy against the RL-based followers in the complex and sequential scenarios.

Temporal abstraction RL. Our methods are also related to temporal abstraction method (Sutton et al., 1998; Daniel et al., 2016; Bacon et al., 2017; Smith et al., 2018; Zhang & Whiteson, 2019; Vezhnevets et al., 2016) . The basic idea of temporal abstraction is to divide the original one-level decision process into a two-level decision process where the high-level part is to decide the meta goal while the low-level policy is to select the primitive actions. Our leader's decision process is different from those methods mentioned above because the leader's policy can naturally form as an intermittent (temporal abstraction) decision process (semi-MDP) (Tharakunnel & Bhattacharyya, 2007) and it is unnecessary to design the two-level decision process for the leader (since the low-level decision process is the follower). Based on the nature of the leader, a novel training method is introduced.

Event-based RL & Planning. Previous studies also focus on using events to capture important elements (e.g., whether agent reaches a goal) during the whole episode. Upadhyay et al. (2018) regard the leader's action and the environment feedback as events in the continuous time environment. Becker et al. (2004) ; Gupta et al. (2018) leverage events to capture the fact that an agent has accomplished some goals. We adopt this idea by depicting the event as the actions taken by the leader at some time steps and design a novel event-based policy gradient to learn the long-term leader's policy.

3. STACKELBERG MARKOV GAMES:

Our research focuses on single-leader multi-follower Stackelberg Markov Games (SMG) (Mguni et al., 2019; Sabbadin & Viet, 2013) , which can be formulated as a tuple G = N , S, A, Ω, P, R, γ . N is the set of N followers, i.e., |N | = N . S is the set of states. s 0 ∈ S 0 ⊂ S is an initial state and S 0 is the set of initial states. A = × k∈N A k is the set of joint actions for followers where a k ∈ A k is an action for the k-th follower. ω ∈ Ω = × k∈N Ω k is an action for the leader and

ω k = {g k , b k } ∈ Ω k is

a goal and a bonus that the leader assigns to the k-th follower. P : S × A → ∆(S) is the transition function1 and R = × k∈N r k × r l is the reward function set where r k : S × A × Ω → R is the reward function for the k-th follower and r l : S × A × Ω → R is the reward function for the leader. γ is the discount factor and a is a joint action of followers. The leader's policy is defined as µ = µ k k∈N where µ k : Ω × S → ∆(Ω k ) is the leader's action to the k-th follower given the leader's action in the previous timestep ω t-1 and the current state s t . ∆(•) is a probability distribution. The followers' joint policy is defined as π = π k where π k : Ω k × S → ∆(A k ) is the k-th follower policy given the leader's action ω k t and the current state s t . Given the policy profile of the leader and followers µ, π , the follower's utility is defined as (st, at, ωt) and the leader's utility is J(µ, π) = E T t=0 γ t r l t (st, at, ωt) . We assume that the leader and followers aim to maximize their own utilities. We define the trajectory τ as a sequence of state, leader's action, and followers' actions ω -1 , (s t , a t , ω t ) T t=0 where ω -1 is the first step leader's action and is set to zero. 

J k (µ, π) = E T t=0 γ t r k t

4. METHODOLOGY:

In this section, we propose a novel training scheme to train a well-performing leader policy against both rule-based and RL-based followers in the expensive coordination problem. We address the two issues, the leader's long-term decision process and the complex interactions between the leader and followers, with three key steps: (a) we model the leader's decision-making process as a semi-Markov Decision Process (semi-MDP) and propose a novel event-based policy gradient to take actions only at important time steps to avoid myopic policy; (b) to accurately predict followers' behaviors, we construct a follower-aware module based on the leader-follower consistency, including a novel follower-specific attention mechanism, and a sequential decision module to predict followers' behaviors precisely and make accurate response to these behaviors; and (c) an action abstractionbased policy gradient method for followers is proposed to simplify the decision process for the followers and thus simplify the interaction between leader and followers, and accelerate the convergence of the training process.

4.1. EVENT-BASED TRAJECTORY OPTIMIZATION FOR LEADER:

We first describe the event-based trajectory optimization for the leader. As we mentioned above, the leader's decision process can be naturally formulated as a semi-MDP (Tharakunnel & Bhattacharyya, 2007) . Therefore, we firstly describe the basic ideas of semi-MDP using the modified option structure.

We define the modified option as a tuple: µ, β k k∈N where µ is the leader's policy as we defined above and β k (s t , ω t-1 ) : S × Ω → [0, 1] is the termination function for the k-th follower, to indicate the probability whether the leader's action to the k-th follower changes (ω k t-1 = ω k t ). Based on these definitions, we formulate the one-step option-state transition function with decay as:

P γ (s t+1 , ω t , a t |s t , ω t-1 ) = γP (s t+1 |s t , a t ) π (a t |s t , ω t ) k { 1 -β k (s t , ω t-1 ) 1 ω k t-1 =ω k t + β k (s t , ω t-1 ) µ k ω k t |s t , ω t-1 }

, where 1 is the indicator function and π (a t |s t , ω t ) = k∈N π k a k t |s t , ω k t is the joint policy for followers. Notice that this is an extension of the augmented process mentioned in (Bacon et al., 2017) . Differently, we do not have the low-level policy here (the low-level policy is the follower) and since we only focus on the finite time horizon, γ is set to be 1. Our modified option is used to depict the long-term decision process for the leader as shown in Fig. 2 . Now we start to discuss our leader's policy gradient. In fact, it is not easy to directly optimize the leader's utility based on this multi-agent option-state transition function since this form includes leader's different action stages to different followers. Notice that for a sampled trajectory, the occurrence of the leader actions is deterministic. Therefore, we can regard the time step and the action the leader takes at that step as an event and define the (universal) event set

U T = { t i , ω k ti |t i ≤ T, k ∈ N }.

We use the notation e k i = t i , ω k ti to represent the leader's action to the k-th follower at step t i , i is the index of the event. Since we focus on the change of the actions from the leader, we further define a set that represents a collection of new actions (ω k t = ω k t-1 ) taken by the leader within that trajectory:

A T = {e k i |ω k ti = ω k ti-1 , t i ≤ T, k ∈ N } ⊆ U T

, where t i -1 is the previous time step. A T represents when and how the leader commits to a new action (an example can be found in Fig. 2a ). For brevity, e k j ∈ A T means e k j ∈ U T \A T . The probability of A T can be represented as:

P (AT ) = k∈N e k i ∈A T β k (st i , ωt i -1)µ k ω k t i |st i , ωt i -1 e k j ∈A T 1 -β k (st j , ωt j -1) ,

where t j -1 is the previous time step for t j . This equation illustrates that the probability of the occurrence of a certain leader's event set within a trajectory. Concretely, the leader changes action to the k-th follower at t i ∈ e k i while maintaining the same action within the interval from t i -1 ∈ e k i-1

to t i (s.t., t i ∈ e k i ). Similarly, we can further define the probability of the whole trajectory τ as:

P (τ ) = P (s0) k∈N e k i ∈A T β k (st i , ωt i -1)µ k ω k t i |st i , ωt i -1 π k a k t i |st i , ω k t i × e k j ∈A T 1 -β k (st j , ωt j -1) π k a k t j |st j , ω k t j T t=0 P (st+1|st, at).

Comparing with P (A T ), P (τ ) includes the probability of the followers as well as the state transition. Do note that our goal is to maximize

max A T E P (τ ) [R τ (T )] ,

indicating that the leader is required to select an action that can maximize the accumulated reward, where R τ (T ) = T t=0 γ t r l t is the accumulated reward and τ is to stress that its accumulated reward is from the trajectory τ . Following the REINFORCE trick (Sutton & Barto, 1998) , the policy gradient for the termination function and the leader's policy function can be formulated under the following proposition:

Proposition 1. The policy gradients for the termination function β k (s ti , ω ti ) and leader's policy function µ k ω k ti |s ti , ω ti-1 can be written as:

∇ θ J(θ) ≈ E τ ∼pτ (•) k∈N T i=0 I(e k i ) R τ (T ) ; ∇ ϑ J(ϑ) ≈ E τ ∼pτ (•) k∈N T i=0 I (e k i ) R τ (T ) ;

where θ and ϑ are the parameters for the termination function β k θ and leader's policy µ k ϑ . I(•) and I (•) are the piece-wise functions:

I(e k i ) =      -∇ θ β k (st i ,ωt i -1) 1-β k (st i ,ωt i -1) e k i ∈ A T , ∇ θ β k (st i ,ωt i -1 ) β k (st i ,ωt i -1) e k i / ∈ A T . I (e k i ) = ∇ ϑ log µ k ω k ti |s ti , ω ti-1 e k i ∈ A T , 0 e k i / ∈ A T .

All the proofs can be found in Appendix A. Proposition 1 implies that under the event-based method, whether the leader's commitment to a new action will induce different policy gradients for both termination function and the policy function.

However, from the empirical results, we find that the leader's policy function updates rarely during the whole episode because the policy only updates when the leader commits to a new action, which causes the sample inefficiency. Notice that in fact the leader commits to the same action when e k i / ∈ A T . Therefore, the policy indication function I (•) can be formulated in an alternative way:

I (e k i ) = ∇ ϑ log µ k ω k ti |s ti , ω ti-1 , e k i ∈ A T ; ∇ ϑ µ k ω k ti = ω k ti-1 |s t , ω ti-1 , e k i / ∈ A T .

This form considers both committing to a new action and maintaining the same actions (Details can be found in Remark 2), which we call the Event-Based Policy Gradient (EBPG) and the previous one as the sparse EBPG respectively.

Intuitively, the dense EBPG is better than the sparse EBPG because it updates the leader's policy function more frequently than the sparse one. For example, in time step t, supposing that the leader chooses a wrong action for follower k and receives a negative reward. Then, the leader should learn to diminish the action chosen that state by EBPG. The sparse EBPG only do one PG during before terminating the action (at the committing action step) while the dense one does PG in each step before terminating the action. The latter can provide more signal to correct the wrong action. Experiments also reveal that the dense one is better (Sec. D.3.3).

4.2. NEURAL NETWORK BASED LEADER:

The EBPG approach is able to improve leader's performance. However, it is still very hard for the leader to choose actions considering long-term effect only based on the current state information. This is because the followers change their behaviors over time according to the leader's policy. Therefore, we introduce new modules and training schemes so as to capture the change of the followers' behaviors as well as the global state. To abstract the complicated state information, we use neural networks to learn the state representation. To capture the followers' behaviors and make accurate response to their behaviors, we design three modules: (1) we exploit the leader-follower consistency under game regularization and policy bound conditions, (2) based on the consistency, a follower-aware module is introduced and (3) based on the follower-aware module, a novel attention mechanism, and sequential decision making module is designed to make accurate response to these followers' behaviors as shown in Fig. 1 .

Leader-Follower Consistency. In previous works, a surge of researches focus on predicting other agents' behaviors through historical information, where the other agents are assumed to be opponents of that agent, which is only suitable for zero-sum games (Zheng et al., 2018; Foerster et al., 2018; He et al., 2016) . However, these methods cannot be directly applied to our case because SMG is not zero-sum. We note that Shu & Tian (2019) attempt to directly use the followers' behavior prediction module (use the history of the followers to predict their future actions) but do not analyze when and how it works. To ensure that the leader can predict the followers' behaviors, we introduce the following assumptions. Assumption 1. (Game regularization) The leader-follower state-action space (A × Ω × S) is compact and r k is a continuous function w.r.t. µ bounded by R max .

This assumption is inspired by (Antos et al., 2008) . We only extend it into the multi-agent forms. This assumption indicates that the action and states space should be limited and the reward function for the leader action should be smooth. Assumption 2. (Policy Bound) For any agent k, reward function r k and policy is consistency, i.e.,

|J k ( π k , π -k , µ) -J k ( π k , π -k , µ )| ≥ C 2 |π k -π k | Where C 2 is a constant that satisfies C 2 > 0. π k is the k-th follower's new policy. µ is the leader's new policy.

This assumption is inspired by (Mguni et al., 2019) . π -k indicates the joint policy without the k-th agent's. This assumption indicates that the change of the leader causes only slightly changes on each followers policy.

Based on these two assumptions, we propose a proposition here: Proposition 2. (Leader-Follower Consistency.) If both the assumptions of game regularization and policy bound are satisfied, for ∀ > 0, k ∈ N , there exists δ > 0, such that |µ -µ | ≤ implies π k -π k ≤ δ, where µ and π k are the new policies for the leader and the k-th follower respectively.

This proposition reveals that the change of the leader causes only slightly changes on each follower's policy under the game regularization assumption and the policy bound assumption, which is fundamental to follower-aware learning. Roughly speaking, the game regularization requires that the states, actions, and rewards are bounded while the policy bound states that a little change of a follower's policy does not change its utility so much. The former is from the game itself and we only focus on the latter. To satisfy the latter, one possible method is to make the µ and µ close because the followers always find the best response to the leader's policy and if the leader changes a little, the followers do not change too much since the new best response to µ is not far away from best response to µ. One direct method is to slow down the learning rate of the leader to make µ and µ close. Moreover, for the leader part, taking the right actions is also an important way to guarantee the second assumption because the taken action will more precisely decrease the probability of huge change of the whole process and stabilize the training process. There is an interesting phenomenon that on one hand, knowing more about the followers can diminish the wrong decision and thus aids the establishment of the consistency. On the other hand, the consistency will further guarantee the accuracy of the follower-aware module. Therefore, they form a positive feedback.

Follower-Aware Module. Based on the leader-follower consistency, we can safely implement the follower-aware module to our network. Before we discuss this module in details, we first define the history for both the leader and followers. For the k-th follower, its history at time step t is a sequence of states, its own actions, and the leader's actions to it, i.e.,

h k t = (s t , a k t , ω k t ) t ≤t ∈ H k t

while the leader's history is the stack of all followers histories h t = h k t k∈N ∈ H l t . Then, we define the history-based leader's policy as:

µ k ht (s t ) = Z -1 p(ω k t |s t , ât , h t )p(â t |s t , h t ) ∝ p(ω k t |s t , ât , h t ) k p k âk t |s t , h k t ,

where Z is the normalization term, p k is the predicted action probability of the k-th follower and â defines the predicted action (predicted by the leader). p is to stress the output is a probability. Since we cannot directly obtain an accurate estimation of a k , we adopt an alternative way to leverage history information and imitation learning to make a prediction of other agents' action probability function p k âk t |s t , h k t (Implementation details can be found in Appendix B) and p(ω k t |s t , ât , h t ) is designed using the attention mechanism as well as the sequential decision module presented below.

Follower-Specified Attention Mechanism. Inspired by (Chen et al., 2018) , we introduce a follower-specific attention mechanism to identify the important followers where the important followers are followers who has just finished a task and the leader has to commit new actions to these followers. The attention mechanism is as follows:

w k t = exp f (A s t , âk t , h k t ) k∈N exp f (A s t , âk t , h k t ) ; c t = k∈N w k t A s t , âk t , h k t ; c k t = [c t , A s t , âk t , h k t ].

Where w k is the weight of the k-th follower, A(•) : R

ds×d a k t ×d h k t →

R dc is a function to blend various information of an agent together (d c means the dimension of the output of the A(•)), f (•) : R dc → R 1 is a function to map the blending information to a real number, and c k t is the k-th agent attention value (the compression of history, states and actions for follower k as well as other followers). âk t is the output of p k âk t |s t , h k t , the predicted the k-th follower's action. This attention mechanism is better because it quantifies the importance of each follower in each state through learning while the original methods only adds/averages all the features (s t , âk t , h k t ) together (Shu & Tian, 2019) . Another advantage is that its weights can be visualized to see which follower is important to the leader at current step. (Details can be found in Appendices D.2.1 & D.3.4). c k t then is used by β k and µ k .

Sequentially Determining Goals and Bonuses. Also notice that the goal and the bonus are sequentially correlated. Therefore, it is better for the leader to choose the bonus and the goal sequentially rather than select them independently. Therefore, to consider the goal and bonus jointly when making a decision, we build a probabilistic graph-based model as: 

p(ω k t |s t , ât , h t ) ≈ p(g k t ; b k t |c k t ) ∝ p(b k t |g k t , c k t ) × p(g k t |c k t ),

4.3. FOLLOWER ACTION ABSTRACTION POLICY GRADIENT:

These methods mentioned above are fully implemented can enhance the performance dramatically. But when facing the RL-based followers, the SMG is still hard to converge. This is because in SMG, the policies of the leader and followers are always changing depending on other agents' performance. To guarantee convergence, the leader can only update its policy when the followers reach (or are near to) the best response policy (Fiez et al., 2019) . However, when the followers are RL-based agents, there is no way to ensure the followers' policies are (near) the best response policies in large-scale SMG and the commonly-seen idea is to provide enough training time but it is unbearable in practice due to the limitation of computing power (Mguni et al., 2019) .

To accelerate the training process, inspired by the action abstraction approach which is commonlyseen in Poker (Brown & Sandholm, 2019; Tuyls et al., 2018) and action abstraction RL (Chandak et al., 2019) , we collect the followers' primitive actions sharing the same properties together as a meta policy. Then, the followers only need to select the meta action to make a decision. Therefore, the original game is converted into a meta game, which is easy to solve. Specifically, we define the policy for the k-th follower as:

π k t (a k |ŝ) = z π k meta (z|ŝ)π k lower (a|ŝ, z)

, where ŝ = s, ω k is the augmented state for the follower (the combination of current state and the leader's action). π k meta (z|ŝ) is the meta policy for the k-th follower and z is the high-level (meta) action. We hypothesize that the lower-level policy (the policy to choose the primitive actions) is already known (rule-based) and deterministic, i.e., π k lower (a k |ŝ, z) = 1. For instance, given the example of the navigation task, the π k meta can be the selection to which landmark to explore while π k lower is a specific route planning algorithm (such as Dijkstra Algorithm). Based on this assumption, we can design a novel policy gradient to train the meta policy:

∇ λ k J k ∝ E ∇ λ k log π k meta (z|ŝ)R k

, where λ k is the parameter for meta-policy π k meta (Details can be found in Lemma 3).

4.4. LOSS FUNCTIONS:

In this section, we discuss how to design the leader's and followers' loss functions.

Loss Functions for the Leaders. The basic structure for the leader is the actor-critic structure (Sutton & Barto, 1998) . We find that adding regularizers can enhance the leader's performance and we implement the maximum entropy for the leader's policy function as well as the L2 regularization for the termination function, i.e.,

L enp = -k ω k µ k (ω k |s, h) log µ k (ω k |s, h) and L reg = β 2 .

We also use imitation learning to learn the predicted action function p k . Following the same logic of (Shu & Tian, 2019) , two baseline functions φ g (c t ) and φ b (c t ) are also introduced to further reduce the variance. Details can be found in Appendix B.

Loss Functions for the RL-Based Followers. The basic structure for each follower is also based on the actor-critic structure. We leverage the action abstraction policy gradient as we mentioned above. The learning rate between the leader and follower should satisfy the two time-scale principle (Roughly speaking, the leader learns slower than the follower(s)), similar to (Borkar, 1997) . Details can be found in Appendix B and the pseudo-code can be found in Appendix C. Tasks.

5. EXPERIMENTAL RESULTS:



5.1. SETUP:

We evaluate the following tasks to testify the performance of our proposed method. All of these tasks are based on SMG mentioned above. (1) resource collections: each follower collects three types of resources including its preferred one and the leader can choose two bonuses levels (Shu & Tian, 2019) ; (2) multi-bonus resources collections: based on (1), the leader can choose four bonuses levels; (3) modified navigation: followers are required to navigate some landmarks and after one of the landmarks is reached, the reached landmark disappears and new landmark will appear randomly. (4) modified predator-prey: followers are required to capture some randomly moving preys, prizes will be given after touching them. Both (3) and ( 4) are based on (Lowe et al., 2017) and we modify them into our SMG setting. Moreover, to increase the difficulty, in each episode, the combinations of the followers will change, i.e., in each task, there are 40 different followers and at each episode, we randomly choose some followers to play the game. More details can be found in Appendix D.

Baselines & Ablations. To evaluate our method, we compare a recently proposed method as our baseline: M 3 RL (Shu & Tian, 2019) . We do not include other baselines because other methods cannot be used in our problems, as justified in (Shu & Tian, 2019) . For the ablations of the leader part, we choose: (1) ours: the full implementation of our method. (2) ours w/o EBPG: removing the event-based policy gradient part; (3) ours w/o Attention: replacing follower-specified attention model by the original attention model mentioned in (Shu & Tian, 2019) . For the follower part, we choose (a) with rule-based follower (b) with vanilla RL-based follower, and (c) with action abstraction RL-based follower to testify the ability of our methods when facing different followers.

Hyper-Parameters. Our code is implemented in Pytorch (Paszke et al., 2017) . If no special mention, the batch size is 1 (online learning). Similar to (Shu & Tian, 2019) , we set the learning rate as 0.001 for the leader's critic and followers while 0.0003 for the leader's policy. The optimization algorithm is Adam (Kingma & Ba, 2014). Our method takes less than two days to train on a NVIDIA Geforce GTX 1080Ti GPU in each experiment.

For the loss function, we set the λ 1 = 0.01 and λ 2 = 0.001. The total training episode is 250, 000 for all the tasks (including both the rule-based followers and the RL-based followers). To encourage exploration, we use the ι-greedy2 . For the leader, the exploration rate is set to 0.1 and slightly decreases to zero (5000 episode). For the followers, the exploration rate for each agent is always 0.3 (except for the noise experiments). The quantitative results with different tasks are shown in Figs. 3 & 4 . For the rule-based followers, from Fig. 3 , we find that our method outperforms the state-of-the-art method in all the tasks, showing that our method is sample efficient and fast to coverage. There is an interesting phenomenon that in the task of multi-bonus resource collections and navigation, only our method obtains a positive reward, indicating that our method can work well in complicated environments. For ablations, we can see that ours w/o attention and ours w/o EBPG are worse than ours, representing these components do enhance the performance. For the RL-based followers, from Fig. 4 , we observe that when facing the RL-based method with action abstraction, our approach outperforms the baseline method in all the tasks (in predator-prey game, the reward for ours is twice as that of the state-of-the-art). We also find that without action abstraction, the reward is less than zero, revealing that the abstraction does play a crucial role in stabilizing training.

5.2. LEARNING EFFICIENCY:



5.3. ROBUSTNESS:

This experiment is to evaluate whether our method is robust to the noise, i.e., the follower randomly takes actions. We make this experiment by introducing noise into the follower decision. From Table 1 , we can find that our method reaches a higher total reward (more than 5) among all the environment with noise than the state-of-the-art, indicating that our method is robust to the noise. We also observe that the total reward for the baseline method becomes lower with the increase of the noise while our method is more robust to the change. Moreover, for the incentive (the total gain), we find that our method gains much more incentive than the state-of-the-art method, showing that our method coordinates have a better coordination the followers than the state-of-the-art method.

5.4. MORE EXPERIMENTS:

We also do a substantial number of experiments. However, due to the space limitation, we can only provide some results here: (1) The total incentives: incentive can reveal the performance of successful rate interacting with the followers. Our method outperforms the state-of-the-art method, indicating that our method has a better ability to interact with the followers. (2) Sparse EBPG: we compare the performance gap between sparse EBPG and (dense) EBPG. This results show that the sparse one is worse than the dense one, supporting the assumption that the dense signal can improve the sample efficiency.

(3) Visualizing attention: We visualize the attention module to find what it actually learns and the result indicates that our attention mechanism does capture the followers whom the leader needs to assign bonuses to. (4) Two time-scale training: We testify whether our two timescale training scheme works and the ablation shows that this scheme does play an important role in improving the performance of both the leader and the followers. (5) The committing interval: We observe that the dynamic committing interval (our method) performs better than the one with fixed committing intervals. ( 6) Reward for RL-based followers: we show the reward for the followers, which can provide the situation of the followers. The result represents that our method aids the followers to gain more than the state-of-the-art method. ( 7) Number of RL-based followers: finally, we testify our method in cases with different number of RL-based followers. The result shows that our method always performs well. The full results can be found in Appendix D.

6. CONCLUSION REMARKS:

This paper proposes a novel RL training scheme for Stackelberg Markov Games with single leader and multiple self-interested followers, which considers the leader's long-term decision process and complicated interaction between followers with three contributions. 1) To consider the long-term effect of the leader's behavior, we develop an event-based policy gradient for the leader's policy. 2) To predict the followers' behaviors and make accurate response to their behaviors, we exploit the leader-follower consistency to design a novel follower-aware module and follower-specific attention mechanism. 3) We propose an action abstraction-based policy gradient algorithm to accelerate the training process of followers. Experiments in resource collections, navigation, and predator-prey game reveal that our method outperforms the state-of-the-art methods dramatically.

We are willing to highlight that SMGs contribute to the RL (especially MARL) community with three key aspects: 1). As we mentioned in the Introduction, most of the existing MARL methods assume that all the agents are willing to sacrifice themselves to maximize the total rewards, which is not true in many real-world non-cooperative scenarios. On the contrary, our proposed method realistically assumes that agents are self-interested. Thus, SMGs provide a new scheme focusing more on the self-interested agents. We think this aspect is the most significant contribution to the RL community. 2). The SMGs can be regarded as the multi-agent system with different roles (the leader and the followers) (Wilson et al., 2008) and our method provides a solution to that problem.

3). Our methods also contribute to the hierarchical RL, i.e., it provides a non-cooperative training scheme between the high-level policy (the leaders) and the low-level policy (the followers), which plays an important role when the followers are self-interested. Moreover, our EBPG also propose an novel policy gradient method for the temporal abstraction structure.

There are several directions we would like to investigate to further extend our SMG model: i) we will consider multiple cooperative/competitive leaders and multiple self-interested followers, which is the case in the labor market, ii) we will consider multi-level leaders, which is the case in the hierarchical organizations and companies and iii) we will consider the adversarial attacks to our SMG model, which may induce extra cost to the leader for efficient coordination. We believe that our work is a preliminary step towards a deeper understanding of the leader-follower scheme in both research and the application to society. A PROOFS Proposition 1. The policy gradients for termination function β k (s ti , ω ti ) and leader's policy function µ k ω k ti |s ti , ω ti-1 can be written as:

∇ θ J(θ) ≈ E τ ∼pτ (•) k∈N T i=0 I(e k i ) R τ (T ) ; ∇ ϑ J(ϑ) ≈ E τ ∼pτ (•) k∈N T i=0 I (e k i ) R τ (T ) ;

where θ and ϑ are the parameters for the termination function β k θ and the leader's policy µ k ϑ . I(•) and I (•) are the piece-wise functions:

I(e k i ) =      -∇ θ β k (st i ,ωt i -1) 1-β k (st i ,ωt i -1) e k i ∈ A T ∇ θ β k (st i ,ωt i -1) β k (st i ,ωt i -1 ) e k i / ∈ A T . I (e k i ) = ∇ ϑ log µ k ω k ti |s ti , ω ti-1 e k i ∈ A T 0 e k i / ∈ A T

Proof. First recall the utility for the leader:

J(θ) = E P (τ ) [R * (T )] = τ m P(|A T | = m)P (s 0 ) k∈N e k i ∈A T β k (s ti , ω ti-1 )µ k ω k ti |s ti , ω ti-1 π k a k ti |s ti , ω k ti × e k j ∈A T 1 -β k (s tj , ω tj -1 ) π k a k tj |s tj , ω k tj T t=0 P (s t+1 |s t , a t )R τ (T );

Where m is the number of the times taking new action, m ≤ T . If m = T , implying that the leader has taken new action at each time. P(|A T | = m) means the probability of times taking new action within an episode.

Take derivatives on both LHS and RHS, we get:

∇ θ J(θ) = τ m ∇ θ P(|A T | = m)P (s 0 ) k∈N   e k i ∈A T β k (s ti , ω ti-1 )µ k ω k ti |s ti , ω ti-1 π k a k ti |s ti , ω k ti   ×   e k j ∈A T 1 -β k (s tj , ω tj -1 ) π k a k tj |s tj , ω k tj   T t=0 P (s t+1 |s t , a t )R τ (T ) = τ m ∇ θ P(|A T | = m) k∈N e k i ∈A T β k (s ti , ω ti-1 ) e k j ∈A T (1 -β k (s tj , ω tj -1 )) P(|A T | = m) k∈N e k i ∈A T β k (s ti , ω ti-1 ) e k j ∈A T (1 -β k (s tj , ω tj -1 )) × P(|A T | = m)P (s 0 ) k∈N   e k i ∈A T β k (s ti , ω ti-1 )µ k ω k ti |s ti , ω ti-1 π k a k ti |s ti , ω k ti   ×   e k j ∈A T 1 -β k (s tj , ω tj -1 ) π k a k tj |s tj , ω k tj   T t=0 P (s t+1 |s t , a t )R τ (T )

For brevity, we use P (τ ) to represent P (s 0 ) k∈N

e k i ∈A T β k (s ti , ω ti-1 )µ k ω k ti |s ti , ω ti-1 π k a k ti |s ti , ω k ti × e k j ∈A T 1 -β k (s tj , ω tj -1 ) π k a k tj |s tj , ω k tj T t=0 P (s t+1 |s t , a t

), the trajectory probability. And we use i ∈ e k i and j ∈ e k j to represent i ∈ e k i ∈ A T to j ∈ e k j / ∈ A T with a slight abuse of notation. Thus the equation mentioned above can be further simplified as:

= τ m P(|A T | = m)∇ θ   k∈N i∈e k i log(1 -β k (s ti , ω ti-1 )) + k∈N j∈e k j log β k s tj , ω tj -1   P (τ )R τ (T ) ≈ E τ ∼pτ (•)    ∇ θ   k∈N i∈e k i log(1 -β k (s ti , ω ti-1 )) + k∈N j∈e k j log β k s tj , ω tj -1   R τ (T )    = E τ ∼pτ (•)      k∈N i∈e k i -∇ θ β k (s ti , ω ti-1 ) 1 -β k (s ti , ω ti-1 ) + k∈N j∈e k j ∇ θ β k s tj , ω tj -1 β k s tj , ω tj -1   R τ (T )   

The equation above is exactly the REINFORCE trick (Sutton & Barto, 1998) and the rule of derivations. The approximation indicates that one trajectory only has one A T3 . Also based on the definition of e k i and e k j , the equation can be rewritten in a more compact form:

∇ θ J(θ) = E τ ∼pτ (•)      k∈N i∈e k i -∇ θ β k (s ti , ω ti-1 ) 1 -β k (s ti , ω ti-1 ) + k∈N j∈e k j ∇ θ β k s tj , ω tj -1 β k s tj , ω tj -1   R τ (T )    = E τ ∼pτ (•) k∈N T i=0 I(e k i ) R τ (T ) ,

Where I(•) is the piece-wise function:

I(e k i ) =      -∇ θ β k (st i ,ωt i -1 ) 1-β k (st i ,ωt i -1) e k i ∈ A T ∇ θ β k (st i ,ωt i -1) β k (st i ,ωt i -1) e k i / ∈ A T .

This is the first part of the proof (the policy gradient for the termination function). Here, we start proving the second part (the policy gradient for the leader's action).

The proof of the second part is similar to the first part.

∇ ϑ J(ϑ) = τ m P(|A T | = m)P (s 0 ) k∈N   e k i ∈A T β k (s ti , ω ti-1 )µ k ω k ti |s ti , ω ti-1 π k a k ti |s ti , ω k ti   ×   e k j ∈A T 1 -β k (s tj , ω tj -1 ))   T t=0 P (s t+1 |s t , a t )R τ (T ); = τ m ∇ ϑ P(|A T | = m) k∈N e k i ∈A T µ k ω k ti |s ti , ω ti-1 P (|A T | = m) k∈N e k i ∈A T µ k ω k ti |s ti , ω ti-1 × P(|A T | = m) × P (s 0 ) k∈N i∈e k i β k (s ti , ω ti-1 )µ k ω k ti |s ti , ω ti-1 π k a k ti |s ti , ω k ti × e k j ∈A T (1 -β k (s tj , ω tj -1 ))π k a k tj |s tj , ω k tj R τ (T ) = τ m   k∈N e k i ∇ ϑ log µ k ω k ti |s ti , ω ti-1   P (τ )R τ (T ) = E τ ∼pτ (•)      k∈N i∈e k i ∇ ϑ log µ k ω k ti |s ti , ω ti-1   R τ (T )   

We rewrite it to a more compact form:

∇ ϑ J(ϑ) = E τ ∼pτ (•) k∈N T i=1 I (e k t ) R τ (T ) I (e k i ) = ∇ ϑ log µ k ω k ti |s ti , ω ti-1 e k i ∈ A T 0 e k i / ∈ A T

Remark 1. Some researches also focus on event-based RL but either on single-agent continuous time (Upadhyay et al., 2018) or reward representation (Gupta et al., 2018) . We are the first to develop and implement the event-based policy gradient into the multi-agent system.

Remark 2. In fact, the policy gradient for the leader actions might be somewhat sparse, i.e., we only update the policy when the leader changes its actions. Notice that the leader commits to the same action when e k i / ∈ A T . Therefore, the probability of leader's action P (A T ) can also represented as:

P (A T ) = k∈N e k i ∈A T β k (s ti , ω ti-1 )µ k ω k ti |s ti , ω ti-1 × e k j ∈A T 1 -β k (s tj , ω tj -1 )) µ k ω k tj = ω k tj -1 |s tj , ω tj -1 ;

Then the policy gradient for leader's policy ∇ ϑ J(ϑ) can thus be

∇ ϑ J(ϑ) ≈ E τ ∼pτ (•) k∈N T i=0 I (e k i ) R τ (T )

, where

I (e k i ) = ∇ ϑ log µ k ω k ti |s ti , ω ti-1 e k i ∈ A T ∇ ϑ µ k ω k ti = ω k ti-1 |s t , ω ti-1 e k i / ∈ A T

Lemma 1. (Reward Bound) For any agent k, the corresponding reward function r k w.r.t ω is C-Lipschitz continuous.

|r k t s t , a k t , a -k t , µ -r k t s t , a k t , a -k t , µ | ≤ C|µ -µ |, s t , a t = [a k t , a -k t ].

Where C is a constant that satisfies C > 0. 4 µ is leader's new policy. a -k is the joint action without agent k's.

Proof. Based on Assumption 1, we can build a compact metric space (A × Ω × S, R k ). From the Heine-Cantor theorem we know that the compact metric space induce uniformly continuous.

That is, for every > 0, there exists a δ > 0,

such that |µ -µ | ≤ implies |r k t s t , a k t , a -k t , µ - r k t s t , a k t , a -k t , µ | ≤ δ.

There exists at least one positive Constance C such that δ ≤ C/ . Then

|r k t s t , a k t , a -k t , µ -r k t s t , a k t , a -k t , µ | ≤ δ ≤ C ≤ C|µ -µ |.

This Lemma is similar to the assumption in (Mguni et al., 2019) . We prove it rather than make it an assumption. Lemma 2. If Assumption 1 is satisfied, the inequality is established:

|J k ( π k , π -k , µ) -J k ( π k , π -k , µ )| ≤ C|µ -µ | Proof.

Expand the utility function by the bellman equation:

4 The follower's reward is originally defined based on the r k t st, a k t , a -k t , ωt ∼ µ . Here to emphasize the relationship between leader's policy µ and reward, with a little abuse of the notation, we use r k t st, a k t , a -k t , µ to represent the reward function.

When π k lower (a|ŝ, z) is fixed and deterministic, the equation can be:

∇ λ k J k = E ∇ λ k log[ zt π k meta (z t |ŝ t )π k lower (a t |ŝ t , z t )]R k = E ∇ λ k [ zt π k meta (z t |s t )π k lower (a t |ŝ t , z t )] zt π k meta (z t |ŝ t )π k lower (a t |ŝ t , z t ) R k = E ∇ λ k [ zt π k meta (z t |ŝ t )π k lower (a t |ŝ t , z t )] zt π k meta (z t |ŝ t )π k lower (a t |ŝ t , z t ) R k = E [ zt ∇ λ k π k meta (z t |ŝ t )π k lower (a t |ŝ t , z t )] zt π k meta (z t |ŝ t )π k lower (a t |ŝ t , z t ) R k = E [ zt ∇ λ k π k meta (z t |ŝ t )π k lower (a t |ŝ t , z t )] Z R k ∝ E ∇ λ k log[π k meta (z t |ŝ t ) R k ] Where Z = z π k meta (z t |ŝ t )π k lower (a t |ŝ t , z t ) = 1

is the partition function. For brevity, with a slight abuse of notation, we omit the superscript for variables a and z which represents the index of an agent.

B LOSS FUNCTIONS B.1 LEADER LOSS FUNCTIONS:

We add a baseline function to reduce the variance of the event-based policy gradient for the leader. We adopt the idea of successor representation (Rabinowitz et al., 2018; Shu & Tian, 2019) as two expected baseline functions: φg (c t ) and φb (c t ). For the gain baseline function:

φ g = g∈G k∈N I g = g k t I s k t = s g v g ,

For the bonus-based baseline function:

φ b = - g∈G k∈N I g = g k t b k t ,

Two baseline neural network functions with parameters θg and θb are trained through minimizing the mean square error:

L baseline = (φ g (c t ; θg ) -φ g ) 2 + (φ b (c t ; θb ) -φ b ) 2 ,

Where c t is the attention-based latent variable.

To this end, the gradient for the leader can be formulated as:

∇ θ L policy l = -∇ θ J (θ) + ∇ ϑ (λ 1 L reg ); ∇ ϑ L policy l = -∇ ϑ J (ϑ) + ∇ ϑ (λ 2 L enp ).

Where

∇ θ J (θ) ≈ E τ ∼pτ (•) k∈N t I(e k t ) (R τ (T ) -φ g (c t ; θg ) + φ b (c t ; θb )) , ∇ ϑ J (ϑ) ≈ E τ ∼pτ (•) k∈N t I 1 (e k t ) (R τ (T ) -φ g (c t ; θg ) + φ b (c t ; θb )) ,

are the baseline policy gradients.

We also leverage the imitation learning to learn the action probability function p k a k t |s t , h k t , θI similar to (Shu & Tian, 2019) , where θ I is the parameters for follower-aware module:

L IL = E - 1 N k∈N log p k a k t |s t , h k t ; θI .

The history encoder, the state encoder and the attention module are updated with the leader's policy gradient end-to-end.

B.2 FOLLOWER LOSS FUNCTIONS:

The follower policy gradient is from Lemma 3 and we find that adding the history h k can improve the performance:

∇ λ k J k ∝ E ∇ λ k log π k meta (z|s, h k )R k ,

and the learning rate of the follower α and the leader β satisfy

t≥0 α t = t≥0 β t = ∞, t≥0 α 2 t + β 2 t < ∞, lim t→∞ β t /α t = 0.

Which indicates the follower's learning rate is much higher than the leader. β is the learning rate for the leader's critic function, which is the same as α. Followers do not have critic function.

C ALGORITHM The illustration of experimental scenarios can be found in Figure 5 . Here we give some details about these environments:

Resource Collections. This task is similar to (Shu & Tian, 2019) , which is based on the scene that the leader and the followers collect some resources. Each follower has its own preference which might be the same (or against) to the leader's preference. In order to make the followers obey the leader's instruction, the leader should pay the followers bonuses. There are total 4 types of resources and for different resources each agent has different preferences. The leader owns two type of bonus (1 or 2) and 4 types of goals (each resource is a goal). The number of leader is 1 while the number of followers is 4.

Multi-Bonus Resource Collections. This task is similar to Resource Collections. Except that the leader can take 4 level bonuses (a bonus from 1 to 4) while each agent owns one skill. The number of leader is 1 while the number of followers is 4.

Modified Navigation. This task is original from (Lowe et al., 2017) . We make some modifications here to make it suitable our SMG: the leader and the followers are going to navigate some landmarks. Each follower has its own preference which might be the same (or against) to the leader's preference. When a landmark has been navigated, it disappears immediately and a new landmark will appear. There are total 6 types of landmarks and for different landmarks, each agent has different preferences. The leader owns two type of bonuses and 6 types of goals (each landmark is a goal).

The number of leader is 1 while the number of followers is 8 and the number of the landmarks is 6.

Modified Predator-Prey. This task is also original from (Lowe et al., 2017) . We make some modification here to make it suitable our SMG: the leader and the followers are going to catch some preys. Each follower has its own preference which might be the same (or against) to the leader's preference. In each step, whether a prey has been caught, it randomly chooses a direction to go. Catching a prey will not make it disappear, which means that the preys can exist until the game ends. There are total 8 types of preys and for different preys, each agent has different preferences. The leader owns two type of bonuses and 8 types of goals (each prey is a goal). The followers are faster than the preys.

The number of leader is 1 while the number of followers is 10 and the number of the landmarks is 8.

Reward Design. The rewards mentioned in Section 3 are the general forms. Here, we define two specified forms of the leader and followers reward function in our experiments:

Leader Reward. We define v g as the prize (utility) for finishing task g. We set the reward function for the leader at step t as: r l t = g∈G k∈N I g = g k t I s k t = s g v g -b k t , formulated by the total gain g∈G k∈N I g = g k t I s k t = s g v g (the total prizes got by the leader at time t) minus the total payment -g∈G k∈N I g = g k t b k t (the total bonuses paid to the followers). We should emphasize that our leader reward is total different from the (Shu & Tian, 2019) : in their approaches, the leader changes its mind after signing a contract will not be punished. To make it suitable to the real world, we modify the reward as the leader should pay the followers bonuses immediately after signing the contract and cannot get back if it gives up the contract.

Follower Reward. For the followers, we set the reward for the k-th follower as:

r k t = g r k g,t = u k,g I s k t = s g + I g l t = g k t × b k t

, where u k,g reveals the payoff of the k-th follower when finishing task g (the preference). Specifically, r k t indicates that the follower can either follow the leader's instruction or just do what it prefers to. The followers will receive reward immediately after signing the contract (the leader and the followers achieve an agreement). I s k t = s g means that the follower finishes the task g at step t. A penalty is added to the followers if the followers betray the leader (the followers and the leader sign the contract but the followers do not agree to work).

D.2 TRAINING DETAILS:



D.2.1 NETWORK DESIGN:

Our network is based on (Shu & Tian, 2019) . Some do not suit our method. We do some modification here: (1) We change the vanilla attention mechanism (sum/average all the history and action of each follower together) to a follower-specified one: each follower has a weight which indicates how important the follower is at the current step. (2) The output for g and b are changed into the sequential form, i.e., we first calculate p(g k t |c k t ) to get g k t , then based on g k t , we choose p(b k t |g k t , c k t ).

(3) The history information h k t is compressed by the LSTM (Gers et al., 2000) . Leader's Network. For the history information, we are willing to stress that the history consists of two parts, one is the statistical information similar to (Shu & Tian, 2019) and the other is the past information given by the neural networks. We leverage the LSTM with 128 hidden units and two fully-connected layers to compress the past information and the statistical information and obtain h k t . Each layer contains 128 neurons. For the state information, we encode it into a 128-dimension hidden state by a state encoder. For resource collections task, the state encoder consist of a convolutional layer with 64 channels and kernels of 1×1 and two fully connected layers with 128 neurons. For navigation and predator-prey task, we encode the state using two fully connected layers with 128 neurons.

For the follower-specific attention module, we firstly predict the âk t from the follower-aware module which contains an LSTM with 128 hidden units and two fully-connected layers with 128 neurons. Secondly, we transform âk t , h k t and the output of the state encoder into A(s t , âk t , h k t ) using a fully connected layer with 128 neurons. The hidden vector is generated by concatenating all the A(s t , âk t , h k t ). Then the attention weight w is calculated using two fully connected layers with softmax activation. The c t is calculated as a weighted summation of w and A(•). Finally, we obtain the k-th attention value c k t by concatenating c t and A(s t , âk t , h k t ). For sequential graph-based model, we firstly calculate p(g k t |c k t ) using two fully connected layers. Each layer consists of 128 neuron units. The input of layers is c k t and the output is g k t . Then we obtain b k t by two fully connected layers. The input of layers is the concatenation of c k t and g k t . Follower's Network. We construct the RL-based followers by a state encoder and two fully connected layers. Each fully connected layer contains 128 neurons. For resource collections task, the state encoder consists of a convolutional layer with 64 channels and kernels of 1×1, two fully connected layers with 128 neurons and an LSTM with 128 hidden units. For navigation and predatorprey task, we encode the state using two fully connected layers with 128 neurons and an LSTM with 128 hidden units. Notice that the network structure of action-abstraction RL-based follower and non-action-abstraction RL-based follower is similar. The only difference is that the output of the former is high-level action z and the output of the latter is low-level action a k t . 

D.3 MORE EXPERIMENTAL RESULTS:



D.3.1 TOTAL INCENTIVE FUNCTION:

Total incentive (income) functions R in = t r l t + t k b k t can reveal how well the leader interacts with the followers; the higher the R in is, the more successful the coordination between the leader and the followers.

From Figure 6 and 7, comparing with the state-of-the-art method, we can see that our method far outperforms the state-of-the-art method, which reveals that our method does have a better ability to coordinate with the followers. In all of the scenarios, without the EBPG, the performance of our method is worse than ours with EBPG. Specifically, in some scenarios (e.g., multi-bonus resource collections, navigation), without the EBPG, the performance of our method is (or nearly) similar to the performance of M 3 RL, showing the effectiveness of our novel policy gradient. Moreover, we can notice that in navigation environment, without follower-specified attention, the performance of our method diminishes rapidly, which implies that in some scenarios, attention does play an important role. Figure 9 : The ablation study of sparse EBPG in the predator-prey task.

D.3.2 ROBUST OF OUR METHOD:

In this section, we are going to testify the robust of our method. Specifically, we evaluate whether our method is robust to the noise. We make this experiment by introducing noise into the follower decision. For example, if we set the noise function as 30%, indicating that there is 30% probability that the followers will choose action randomly.

D.3.3 SPARSE EVENT-BASED POLICY GRADIENT:

This experiment testifies whether the dense event-based policy gradient increases the leader's performance comparing with the sparse event-based policy gradient. We make ablations here: (1) Ours: the full structure of our method;

(2) Sparse Event-Based Policy Gradient (sparse EBPG): the fully structure of ours except that the EBPG is replaced by sparse event-based policy gradient; (3) sparse EBPG w/o attention: replacing the follower-specified attention mechanism by averaging the input features.

From Figure 8 and 9 we can find that if the policy gradient is sparse, its performance is worse than the dense one, implying that the dense method does improve the leader's performance. There is also an interesting phenomenon that sparse EBPG with follower-specified attention mechanism performs better than that without, revealing that the attention can stabilize training when the training signal is sparse. Figure 11 : The ablation study of reward curves for two time-scale update method in resource collections task.

D.3.4 VISUALIZING ATTENTION:

Following the same logic of (Iqbal & Sha, 2019) , we visualize the weight of the attention when the leader takes actions. From Figure 10 , we find that the attention mechanism does learn to strongly attend to the followers that the leader needs to take actions. The followers with leader's commitment obtain much higher attention weight than others, showing that the attention module actually learn to identify the important followers while leader committing new action. Thus, the attention mechanism does play an important role in improving the performance.

D.3.5 TWO TIME-SCALE UPDATING:

In order to evaluate the performance of our two time-scale update scheme (TTSU), we do an ablation study as shown in Fig 11 . We can find that the performance where the followers' learning rate α (1 × 10 -3 ) is much larger than the leader's β (3 × 10 -4 ) is better than the performance where the leader's learning rate is similar to the followers (1 × 10 -3 ). Moreover, without TTSU, the reward curves of training methods become unstable, revealing that TTSU can stabilize the training process. In fact, TTSU improves the rate of convergence and play an important role in improving performance.

D.3.6 COMMITTING INTERVAL:

We evaluate the leader's performance between static committing interval and our dynamic committing interval. As shown in Figure 12 , we observe that all the different fixed committing intervals only change the rate of convergence and do not enhance the leader's performance. All the fixed committing intervals are much worse than our dynamic committing approach, revealing the fact that our dynamic committing approach aids a lot in improving the leader's performance. The last four terms are so close to zero and we use the rewards from last episode as the final results.

D.3.7 REWARD FOR RL-BASED FOLLOWERS:

We are interesting in the reward for the RL-based follower(s). Intuitively, a well-performing leader can make the follower gain more. As shown in Figure 13 , the reward for RL follower is higher than M 3 RL follower in all the tasks. This represents the leader can coordinate the followers better and make them gain more reward than other methods, which forms a win-win strategy. 

D.3.8 NUMBER OF THE FOLLOWER AGENTS:

Finally, we evaluate the leader's performance with different number of RL-based followers. As shown in Figure 15 , we find that our method outperforms the state-of-the-art method when facing different number of RL-based workers.

D.3.9 DIFFERENT COMBINATIONS STUDY:

To further illustrate the performance of different combinations of our methods, we make an extra ablation here. We choose the resource collections as the environments. Our analysis is as follows:

For the RL based followers scenario, As shown in Table 2 , we find that the action abstraction policy gradient is very important to converge. Additionally, adding different modules can improve the performance and the method with all the modules reach the highest reward than other combinations. The contribution ranking for each module is: Action abstraction policy gradient > EBPG > Leaderfollower consistency. 

:

Notice that the transition function does not depend on the leader's action.

Normally it is called the decayed -greedy. We use ι instead of to avoid notation abuse.

We find thatUpadhyay et al. (2018) also implement this approximation but use different explanations.

Figures and Tables:

Figure 1: Overview of our framework. The details of the leader's module and the follower's module can be found in Section 4.2 and Section 4.3, respectively. The implement details of each module can be found in Appendix D.2.1.

(a) A simple example for the illustration of AT . Suppose that the whole step is 4, the AT = {e I 1 = 0, ω I 0 , e II 1 = 1, ω II 1 , e II 2 = 3, ω II 3 }. (b) The probabilistic graphical model of the proposed framework. Dotted line means that β affects the final result of ω indirectly. ω-1 is set to be zero.

Figure 2: An example and a probabilistic graphical model to illustrate our method.

Figure 3: Leader's reward curves for different tasks (rule-based followers).

Figure 4: The final reward for RL-based followers. No abstraction means the vanilla RL-based followers.

Figure 5: Illustration of different tasks.

Total incentive for the multi-bonus task (rulebased agent).

Figure 6: Total incentives for predator-prey task and multi-bonus resource collections task.

Figure 7: Total incentives for resource collections task and navigation task.

The total incentive curves for multi-bonus resource collections.

Figure 8: The ablation study of sparse EBPG in the multi-bonus resource collections task.

Figure 10: Visualization of the attention. The x-axis indicates the commit time for different agents and the y-axis indicates the corresponding weights w k when the leader commits to that agent.

commit (M 3 RL) 3 step interval commit (M 3 RL) 6 step interval commit (M 3 RL) (a) The ablation study of reward curves for different committing interval in resource collections. interval (M 3 RL) 3 step committing interval (M 3 RL) 6 step committing interval (M 3 RL) (b) The ablation study of reward curves for fixed committing interval method in navigation. commit (M 3 RL) 3 step interval commit (M 3 RL) 6 step interval commit (M 3 RL) (c) The ablation study of reward curves for fixed committing interval method in multi-bonus resource collections. interval (M 3 RL) 3 step committing interval (M 3 RL) 6 step committing interval (M 3 RL) (d) The ablation study of reward curves for fixed committing interval method in predator-prey.

Figure12: The ablation study of reward curves for fixed commitment time in resource collections task.

Figure 13: Reward curves for RL-based followers in different tasks.

Figure 14: Leader's reward curves for different tasks (RL-based followers).

Figure 15: The final reward of different number of follower agents in different tasks.

Robustness results in multi-bound resource collections. b% is the probability that followers randomly choose actions.

Table content: <table><tr><td>Methods</td><td colspan="3">0% Noise 30% Noise 50% Noise</td></tr><tr><td>Ours total incentive</td><td>18.32</td><td>17.63</td><td>17.28</td></tr><tr><td>M 3 RL total incentive</td><td>4.06</td><td>3.85</td><td>4.02</td></tr><tr><td>Ours total reward</td><td>10.06</td><td>5.36</td><td>5.30</td></tr><tr><td>M 3 RL total reward</td><td>-1.58</td><td>-3.23</td><td>-8.96</td></tr></table>

Shangtong Zhang and Shimon Whiteson. DAC: The double actor-critic architecture for learning options. arXiv preprint arXiv:1904.12691, 2019. Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjie Fan. A deep Bayesian policy reuse approach against non-stationary agents. In NeurIPS, pp. 954-964, 2018.

Table content: <table/>

θI -βL IL ∇ θI L IL ; // the follower-aware module's parameters

Table content: <table><tr><td>6</td><td>for k in N do</td><td/><td/></tr><tr><td>7</td><td colspan="4">The k-th follower receives its own bonus b k t and goal g k t ;</td></tr><tr><td>8</td><td colspan="4">each agent make a decision a k t = π k t (a k t |ŝ t ) ;</td></tr><tr><td>9</td><td>end</td><td/><td/></tr><tr><td>10</td><td>s t+1 , r l t , {r k</td><td/><td/></tr><tr><td>19</td><td>θ ← θ -β∇ θ L policy l</td><td colspan="2">;</td><td>// the termination function's parameters</td></tr><tr><td>20</td><td colspan="2">ϑ ← ϑ -β∇ ϑ L policy l</td><td>;</td><td>// the parameters of leader's policy;</td></tr><tr><td colspan="2">21 end</td><td/><td/></tr><tr><td/><td colspan="4">Algorithm 2: Action Choices for Leader</td></tr><tr><td/><td>Input: Leader's policy µ</td><td/><td/></tr><tr><td/><td colspan="3">Output: goals g and bonuses b;</td></tr><tr><td colspan="2">1 for k ∈ N do</td><td/><td/></tr><tr><td>2</td><td colspan="4">sample the termination function β k t ;</td></tr><tr><td>3 4</td><td colspan="4">if β k t terminates previous leader's action ω k t-1 then choose new action ω k t ;</td></tr><tr><td>5</td><td>else</td><td/><td/></tr><tr><td>6</td><td>ω k t = ω k t-1 ;</td><td/><td/><td>// maintain previous action.</td></tr></table>

Ablation results of RL-based followers for resource collections. ' ' means the module is used.

Table content: <table><tr><td>Modules</td><td/><td/><td/><td/></tr><tr><td>EBPG</td><td/><td/><td/><td/></tr><tr><td>Leader-follower consistency</td><td/><td/><td/><td/></tr><tr><td>Action abstraction policy gradient</td><td/><td/><td/><td/></tr><tr><td>Final reward</td><td>8.12 6.26 4.18 3.48 -0.02</td><td>-0.03</td><td>-0.05</td><td>-0.05</td></tr></table>

Peer review (iclr2020):
This paper proposes an interesting event-based policy gradient algorithm for single-leader multi-follower Stackelberg Markov Games (SMG). The authors claimed 3 contributions on their work: model the leader*s decision-making process as a semi-MDP and proposed an event-based policy gradient algorithm for the leader to consider long-term effect; exploit a leader-follower consistency scheme with a follower-aware module and a follower-specific attention module to better predict the follower*s behaviors; and propose an action-abstraction policy gradient algorithm in order to make SMG easier to converge. The proposed model is interesting and the authors have done sufficient empirical studies to show the success of their method, as well as some necessary theoretical derivations. My major concern about this work is whether the task that this paper follows is widely studied in the reinforcement learning community, as the authors mentioned that only [1] can be used as baseline methods since the other methods can not be applied in their problem. But in general I think this is an interesting paper. Some detailed comments and suggestions: 1. This paper proposed 3 major contributions: event-based policy gradient, leader-follower consistency scheme and action abstraction policy gradient. The authors show the empirical performance comparison for the case with and without each of these techniques in the experiments. It can be seen that all of the 3 techniques help the proposed method perform better. In addition, the proposed method performs better compared to the state-of-the-art method (in [1]). Hence the authors have shown the success of their algorithm empirically. One thing that can be done to improve this paper is to show a joint comparison for the 2 ^ 3 = 8 cases of with / without any of the 3 techniques so that we can gain a better understanding about the effect of these 3 techniques. 2. The authors provide some necessary theoretical derivations to compute the policy gradients. But there are some typos in the equations. For example, the numerator of the equation of the case e_i^k
otin A_T in I(e_i^k) at the bottom of page 4 is incorrect. In addition, the equation on the second last line of page 3 should be P_gamma(s_{t+1}, omega_t, a_t | s_t, omega_{t-1}), not P_gamma(s_{t+1}, omega_t | s_t, omega_{t-1}, a_t). Please check the details and solve these typos. 3. It will be better if the authors can provide more theoretical analysis or quantitative explanation about the proposed method. For example, why a dense EBPG better than a sparse one? Why the equations in the attention mechanism appear in their form, not some similar forms (e.g. scale the values from the function A before putting them into the softmax function)? Intuitively the proposed techniques make sense, but it will be great if more insights are provided. 4. As I mentioned, I am concerned if the proposed problem is widely studied in the reinforcement learning community. Also, since the authors mentioned in the appendix that two of the tasks are original from [2], but we can not apply the method in [2] as a baseline method. It is possible to modify the method in [2] to fit with the setting and compare with them? 5. It will be better if the authors can rearrange the paper for a little bit, since there are too many important details (e.g. experiment settings, assumptions in the theorems) are in appendix. Questions: 1. Can the authors try to modify the method in [2] and compare with it (details in my comment 4)? Reference: [1] Tianmin Shu and Yuandong Tian. M3RL: Mind-aware multi-agent management reinforcement learning. In ICLR, 2019. [2] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In NeurIPS, pp. 6379–6390, 2017.