LEARNING EXPENSIVE COORDINATION: AN EVENT-BASED DEEP RL APPROACH
Year: 2020

Abstract:
Existing works in deep Multi-Agent Reinforcement Learning (MARL) mainly focus on coordinating cooperative agents to complete certain tasks jointly. However, in many cases of the real world, agents are self-interested such as employees in a company and clubs in a league. Therefore, the leader, i.e., the manager of the company or the league, needs to provide bonuses to followers for efficient coordination, which we call expensive coordination. The main difficulties of expensive coordination are that i) the leader has to consider the long-term effect and predict the followers’ behaviors when assigning bonuses, and ii) the complex interactions between followers make the training process hard to converge, especially when the leader’s policy changes with time. In this work, we address this problem through an event-based deep RL approach. Our main contributions are threefold. (1) We model the leader’s decision-making process as a semi-Markov Decision Process and propose a novel multi-agent event-based policy gradient to learn the leader’s long-term policy. (2) We exploit the leader-follower consistency scheme to design a follower-aware module and a follower-specific attention module to predict the followers’ behaviors and make accurate response to their behaviors. (3) We propose an action abstraction-based policy gradient algorithm to reduce the followers’ decision space and thus accelerate the training process of followers. Experiments in resource collections, navigation, and the predator-prey game reveal that our approach outperforms the state-of-the-art methods dramatically.

heading: 1 INTRODUCTION
text: Deep Multi-Agent Reinforcement Learning (MARL) has been widely used in coordinating cooperative agents to jointly complete certain tasks where the agent is assumed to be selfless (fully cooperative), i.e., the agent is willing to sacrifice itself to maximize the team reward. However, in many cases of the real world, the agents are self-interested, such as taxi drivers in a taxi company (fleets) and clubs in a league. For instance, in the example of taxi fleets (Miao et al., 2016), drivers may prefer to stay in the area with high customer demand to gain more reward. It is unfair and not efficient to compel the taxi driver to selflessly contribute to the company, e.g., to stay in the low customer demand area. Forcing the drivers to selflessly contribute may increase the income for the company in a short-term but it will finally causes the low efficient and unsustainable of that company in the ∗Indicates equal contribution. †Co-corresponding authors.
long run because the unsatisfied drivers may be demotivated and even leave the company. Another important example is that the government wants some companies to invest on the poverty area to achieve the fairness of the society, which may inevitably reduce the profits of companies. Similar to previous example, the companies may leave when the government forces them to invest. A better way to achieve coordination among followers and achieve the leader’s goals is that the manager of the company or the government needs to provide bonuses to followers, like the taxi company pays extra bonuses for serving the customers in rural areas and the government provides subsidies for investing in the poverty areas, which we term as expensive coordination. In this paper, we solve the large-scale sequential expensive coordination problem with a novel RL training scheme.
There are several lines of works related to the expensive coordination problem, including mechanism design (Nisan & Ronen, 2001) and the principal-agent model (Laffont & Martimort, 2009). However, these works focus more on static decisions (each agent only makes a single decision). To consider sequential decisions, the leader-follower MDP game (Sabbadin & Viet, 2013; 2016) and the RL-based mechanism design (Tang, 2017; Shen et al., 2017) are introduced but most of their works only focus on matrix games or small-scale Markov games, which cannot be applied to the case with the large-scale action or state space. The most related work is M3RL (Shu & Tian, 2019) where the leader assigns goals and bonuses by using a simple attention mechanism (summing/averaging the features together) and mind (behaviors) tracking to predict the followers’ behaviors and makes response to the followers’ behaviors. But they only consider the rule-based followers, i.e., followers with fixed preference, and ignore the followers’ behaviors responding to the leader’s policy, which significantly simplifies the problem and leads the unreasonability of the model.
In the expensive coordination problem, there are two critical issues which should be considered: 1) the leader’s long-term decision process where the leader has to consider both the long-term effect of itself and long-term behaviors of the followers when determining his action to incentivise the coordination among followers, which is not considered in (Sabbadin & Viet, 2013; Mguni et al., 2019); and 2) the complex interactions between the leader and followers where the followers will adapt their policies to maximize their own utility given the leader’s policy, which makes the training process unstable and hard, if not unable, to converge in large-scale environment, especially when the leader changes his actions frequently, which is ignored by (Tharakunnel & Bhattacharyya, 2007; Shu & Tian, 2019). In this work, we address these two issues in the expensive coordination problem through an abstraction-based deep RL approach.
Our main contributions are threefold. (1) We model the leader’s decision-making process as a semiMarkov Decision Process (semi-MDP) and propose a novel event-based policy gradient to learn the leader’s policy considering the long-term effect (leader takes actions at important points rather than at each step to avoid myopic decisions.) (Section 4.1). (2) A well-performing leader’s policy is also highly dependent on how well the leader knows the followers. To predict the followers’ behaviors precisely, we show the leader-follower consistency scheme. Based on the scheme, the follower-aware module, the follower-specific attention module, and the sequential decision module are proposed to capture these followers’ behaviors and make accurate response to their behaviors (Section 4.2). (3) To accelerate the training process, we propose an action abstraction-based policy gradient algorithm for the followers. This approach is able to reduce followers’ decision space and thus simplifies the interaction between the leader and followers as well as accelerates the training process of followers (Section 4.3). Experiments in resource collections, navigation and predatorprey show that our method outperforms the state-of-the-art methods dramatically.

heading: 2 RELATED WORKS
text: Our works are closely related to leader-follower RL, temporal abstraction RL, and event-based RL.
Leader-follower RL. The leader-follower RL targets at addressing the issue of expensive coordination where the leader wants to maximize the social benefit (or the leader’s self-benefit) by coordinating non-cooperative followers through providing them bonuses. Previous works have investigated different approaches to solve the expensive coordination, including the vanilla leader-follower MARL (Sabbadin & Viet, 2013; Laumônier & Chaib-draa, 2005), leader semi-MDP (Tharakunnel & Bhattacharyya, 2007), multiple followers and sub-followers MARL (Cheng et al., 2017), followers abstraction (Sabbadin & Viet, 2016), and Bayesian optimization (Mguni et al., 2019). But most of them focus on simple tabular games or small-scale Markov games. The most related work (Shu
& Tian, 2019) leverages the deep RL approach to compute the leader’s policy of assigning goals and bonuses to rule-based followers. But their method performs poorly when the followers are RLbased. In this work, we aim to compute the leader’s policy against the RL-based followers in the complex and sequential scenarios.
Temporal abstraction RL. Our methods are also related to temporal abstraction method (Sutton et al., 1998; Daniel et al., 2016; Bacon et al., 2017; Smith et al., 2018; Zhang & Whiteson, 2019; Vezhnevets et al., 2016). The basic idea of temporal abstraction is to divide the original one-level decision process into a two-level decision process where the high-level part is to decide the meta goal while the low-level policy is to select the primitive actions. Our leader’s decision process is different from those methods mentioned above because the leader’s policy can naturally form as an intermittent (temporal abstraction) decision process (semi-MDP) (Tharakunnel & Bhattacharyya, 2007) and it is unnecessary to design the two-level decision process for the leader (since the low-level decision process is the follower). Based on the nature of the leader, a novel training method is introduced.
Event-based RL & Planning. Previous studies also focus on using events to capture important elements (e.g., whether agent reaches a goal) during the whole episode. Upadhyay et al. (2018) regard the leader’s action and the environment feedback as events in the continuous time environment. Becker et al. (2004); Gupta et al. (2018) leverage events to capture the fact that an agent has accomplished some goals. We adopt this idea by depicting the event as the actions taken by the leader at some time steps and design a novel event-based policy gradient to learn the long-term leader’s policy.

heading: 3 STACKELBERG MARKOV GAMES
text: Our research focuses on single-leader multi-follower Stackelberg Markov Games (SMG) (Mguni et al., 2019; Sabbadin & Viet, 2013), which can be formulated as a tupleG = 〈N ,S,A,Ω, P,R, γ〉. N is the set of N followers, i.e., |N | = N . S is the set of states. s0 ∈ S0 ⊂ S is an initial state and S0 is the set of initial states. A = ×k∈NAk is the set of joint actions for followers where ak ∈ Ak is an action for the k-th follower. ω ∈ Ω = ×k∈NΩk is an action for the leader and ωk = {gk, bk} ∈ Ωk is a goal and a bonus that the leader assigns to the k-th follower. P : S × A → ∆(S) is the transition function1 and R = ×k∈N rk × rl is the reward function set where rk : S × A × Ω → R is the reward function for the k-th follower and rl : S × A × Ω→ R is the reward function for the leader. γ is the discount factor and a is a joint action of followers.
The leader’s policy is defined as µ = 〈µk〉k∈N where µk : Ω × S → ∆(Ωk) is the leader’s action to the k-th follower given the leader’s action in the previous timestep ωt−1 and the current state st. ∆(·) is a probability distribution. The followers’ joint policy is defined as π = 〈πk〉 where πk : Ωk ×S → ∆(Ak) is the k-th follower policy given the leader’s action ωkt and the current state st. Given the policy profile of the leader and followers 〈µ,π〉, the follower’s utility is defined as Jk(µ,π) = E [∑T t=0 γ trkt (st,at,ωt) ] and the leader’s utility is J(µ,π) = E [∑T t=0 γ trlt (st,at,ωt) ] . We assume that the leader and followers aim to maximize their own utilities. We define the trajectory τ as a sequence of state, leader’s action, and followers’ actions 〈ω−1, (st,at,ωt)Tt=0〉 where ω−1 is the first step leader’s action and is set to zero.
1Notice that the transition function does not depend on the leader’s action.

heading: 4 METHODOLOGY
text: In this section, we propose a novel training scheme to train a well-performing leader policy against both rule-based and RL-based followers in the expensive coordination problem. We address the two issues, the leader’s long-term decision process and the complex interactions between the leader and followers, with three key steps: (a) we model the leader’s decision-making process as a semiMarkov Decision Process (semi-MDP) and propose a novel event-based policy gradient to take actions only at important time steps to avoid myopic policy; (b) to accurately predict followers’ behaviors, we construct a follower-aware module based on the leader-follower consistency, including a novel follower-specific attention mechanism, and a sequential decision module to predict followers’ behaviors precisely and make accurate response to these behaviors; and (c) an action abstractionbased policy gradient method for followers is proposed to simplify the decision process for the followers and thus simplify the interaction between leader and followers, and accelerate the convergence of the training process.

heading: 4.1 EVENT-BASED TRAJECTORY OPTIMIZATION FOR LEADER
text: We first describe the event-based trajectory optimization for the leader. As we mentioned above, the leader’s decision process can be naturally formulated as a semi-MDP (Tharakunnel & Bhattacharyya, 2007). Therefore, we firstly describe the basic ideas of semiMDP using the modified option structure. We define the modified option as a tuple: 〈µ, ( βk ) k∈N 〉 where µ is the leader’s policy as we defined above and β
k(st,ωt−1) : S × Ω → [0, 1] is the termination function for the k-th follower, to indicate the probability whether the leader’s action to the k-th follower changes (ωkt−1 6= ωkt ). Based on these definitions, we formulate the one-step option-state transition function with decay as: Pγ (st+1,ωt,at|st,ωt−1) = γP (st+1|st,at)π (at|st,ωt) ∏ k{ ( 1− βk (st,ωt−1) ) 1ωkt−1=ω k t +
βk (st,ωt−1)µ k ( ωkt |st,ωt−1 ) }, where 1 is the indicator function and π (at|st,ωt) =∏
k∈N π k ( akt |st, ωkt ) is the joint policy for followers. Notice that this is an extension of the augmented process mentioned in (Bacon et al., 2017). Differently, we do not have the low-level policy here (the low-level policy is the follower) and since we only focus on the finite time horizon, γ is set to be 1. Our modified option is used to depict the long-term decision process for the leader as shown in Fig. 2.
Now we start to discuss our leader’s policy gradient. In fact, it is not easy to directly optimize the leader’s utility based on this multi-agent option-state transition function since this form includes leader’s different action stages to different followers. Notice that for a sampled trajectory, the occurrence of the leader actions is deterministic. Therefore, we can regard the time step and the action the leader takes at that step as an event and define the (universal) event set UT = {〈ti, ωkti〉|ti ≤ T, k ∈ N}. We use the notation e k i = 〈ti, ωkti〉 to represent the leader’s action to the k-th follower at step ti, i is the index of the event. Since we focus on the change of the actions from the leader, we further define a set that represents a collection of new actions (ωkt 6= ωkt−1) taken by the leader within that trajectory: AT = {eki |ωkti 6= ω k ti−1, ti ≤ T, k ∈ N} ⊆ UT , where ti − 1 is the previous time step. AT represents when and how the leader commits to a new action (an example can be found in Fig. 2a). For brevity, ekj 6∈ AT means ekj ∈ UT \AT . The probability of
AT can be represented as:
P (AT ) = ∏
k∈N ∏ eki ∈AT βk(sti ,ωti−1)µ k ( ωkti |sti ,ωti−1 )∏ ekj 6∈AT ( 1− βk(stj ,ωtj−1) ) ,
where tj − 1 is the previous time step for tj . This equation illustrates that the probability of the occurrence of a certain leader’s event set within a trajectory. Concretely, the leader changes action to the k-th follower at ti ∈ eki while maintaining the same action within the interval from ti−1 ∈ eki−1 to ti (s.t., ti ∈ eki ). Similarly, we can further define the probability of the whole trajectory τ as:
P (τ) = P (s0) ∏
k∈N {[∏ eki ∈AT βk(sti ,ωti−1)µ k ( ωkti |sti ,ωti−1 ) πk ( akti |sti , ω k ti )] ×[∏
ekj 6∈AT
( 1− βk(stj ,ωtj−1) ) πk ( aktj |stj , ω k tj )]}∏T t=0 P (st+1|st,at).
Comparing with P (AT ), P (τ) includes the probability of the followers as well as the state transition. Do note that our goal is to maximize maxAT EP (τ) [Rτ (T )] , indicating that the leader is required to select an action that can maximize the accumulated reward, where Rτ (T ) = ∑T t=0 γ
trlt is the accumulated reward and τ is to stress that its accumulated reward is from the trajectory τ . Following the REINFORCE trick (Sutton & Barto, 1998), the policy gradient for the termination function and the leader’s policy function can be formulated under the following proposition:
Proposition 1. The policy gradients for the termination function βk (sti ,ωti) and leader’s policy function µk ( ωkti |sti ,ωti−1 ) can be written as:
∇θJ(θ) ≈ Eτ∼pτ (·) {[∑
k∈N ∑T i=0 I(eki ) ] Rτ (T ) } ; ∇ϑJ(ϑ) ≈ Eτ∼pτ (·) {[∑ k∈N ∑T i=0 I ′(eki ) ] Rτ (T ) } ;
where θ and ϑ are the parameters for the termination function βkθ and leader’s policy µ k ϑ. I(·) and

heading: I ′(·) are the piece-wise functions:
text: I(eki ) =  −∇θβk(sti ,ωti−1) 1−βk(sti ,ωti−1)
eki ∈ AT , ∇θβk(sti ,ωti−1) βk(sti ,ωti−1) eki /∈ AT . I ′(eki ) =
{ ∇ϑ logµk ( ωkti |sti ,ωti−1 ) eki ∈ AT ,
0 eki /∈ AT .
All the proofs can be found in Appendix A. Proposition 1 implies that under the event-based method, whether the leader’s commitment to a new action will induce different policy gradients for both termination function and the policy function.
However, from the empirical results, we find that the leader’s policy function updates rarely during the whole episode because the policy only updates when the leader commits to a new action, which causes the sample inefficiency. Notice that in fact the leader commits to the same action when eki /∈ AT. Therefore, the policy indication function I ′(·) can be formulated in an alternative way: I ′(eki ) = ∇ϑ logµk ( ωkti |sti ,ωti−1 ) , eki ∈ AT ;∇ϑµk ( ωkti = ω k ti−1|st,ωti−1 ) , eki /∈ AT . This form considers both committing to a new action and maintaining the same actions (Details can be found in Remark 2), which we call the Event-Based Policy Gradient (EBPG) and the previous one as the sparse EBPG respectively.
Intuitively, the dense EBPG is better than the sparse EBPG because it updates the leader’s policy function more frequently than the sparse one. For example, in time step t, supposing that the leader chooses a wrong action for follower k and receives a negative reward. Then, the leader should learn to diminish the action chosen that state by EBPG. The sparse EBPG only do one PG during before terminating the action (at the committing action step) while the dense one does PG in each step before terminating the action. The latter can provide more signal to correct the wrong action. Experiments also reveal that the dense one is better (Sec. D.3.3).

heading: 4.2 NEURAL NETWORK BASED LEADER
text: The EBPG approach is able to improve leader’s performance. However, it is still very hard for the leader to choose actions considering long-term effect only based on the current state information. This is because the followers change their behaviors over time according to the leader’s policy.
Therefore, we introduce new modules and training schemes so as to capture the change of the followers’ behaviors as well as the global state. To abstract the complicated state information, we use neural networks to learn the state representation. To capture the followers’ behaviors and make accurate response to their behaviors, we design three modules: (1) we exploit the leader-follower consistency under game regularization and policy bound conditions, (2) based on the consistency, a follower-aware module is introduced and (3) based on the follower-aware module, a novel attention mechanism, and sequential decision making module is designed to make accurate response to these followers’ behaviors as shown in Fig. 1.
Leader-Follower Consistency. In previous works, a surge of researches focus on predicting other agents’ behaviors through historical information, where the other agents are assumed to be opponents of that agent, which is only suitable for zero-sum games (Zheng et al., 2018; Foerster et al., 2018; He et al., 2016). However, these methods cannot be directly applied to our case because SMG is not zero-sum. We note that Shu & Tian (2019) attempt to directly use the followers’ behavior prediction module (use the history of the followers to predict their future actions) but do not analyze when and how it works. To ensure that the leader can predict the followers’ behaviors, we introduce the following assumptions.
Assumption 1. (Game regularization) The leader-follower state-action space (A×Ω×S) is compact and rk is a continuous function w.r.t. µ bounded by Rmax.
This assumption is inspired by (Antos et al., 2008). We only extend it into the multi-agent forms. This assumption indicates that the action and states space should be limited and the reward function for the leader action should be smooth.
Assumption 2. (Policy Bound) For any agent k, reward function rk and policy is consistency, i.e., |Jk( [ πk, π−k ] ,µ)− Jk( [ πk, π−k ] ,µ′)| ≥ C2|πk − π′k|
Where C2 is a constant that satisfies C2 > 0. π′k is the k-th follower’s new policy. µ′ is the leader’s new policy.
This assumption is inspired by (Mguni et al., 2019). π−k indicates the joint policy without the k-th agent’s. This assumption indicates that the change of the leader causes only slightly changes on each followers policy.
Based on these two assumptions, we propose a proposition here:
Proposition 2. (Leader-Follower Consistency.) If both the assumptions of game regularization and policy bound are satisfied, for ∀ > 0, k ∈ N , there exists δ > 0, such that |µ− µ′| ≤ implies ∣∣πk − π′k∣∣ ≤ δ, where µ′ and π′k are the new policies for the leader and the k-th follower respectively.
This proposition reveals that the change of the leader causes only slightly changes on each follower’s policy under the game regularization assumption and the policy bound assumption, which is fundamental to follower-aware learning. Roughly speaking, the game regularization requires that the states, actions, and rewards are bounded while the policy bound states that a little change of a follower’s policy does not change its utility so much. The former is from the game itself and we only focus on the latter. To satisfy the latter, one possible method is to make the µ and µ′ close because the followers always find the best response to the leader’s policy and if the leader changes a little, the followers do not change too much since the new best response to µ′ is not far away from best response to µ. One direct method is to slow down the learning rate of the leader to make µ′ and µ close. Moreover, for the leader part, taking the right actions is also an important way to guarantee the second assumption because the taken action will more precisely decrease the probability of huge change of the whole process and stabilize the training process. There is an interesting phenomenon that on one hand, knowing more about the followers can diminish the wrong decision and thus aids the establishment of the consistency. On the other hand, the consistency will further guarantee the accuracy of the follower-aware module. Therefore, they form a positive feedback.
Follower-Aware Module. Based on the leader-follower consistency, we can safely implement the follower-aware module to our network. Before we discuss this module in details, we first define the history for both the leader and followers. For the k-th follower, its history at time step t is a sequence of states, its own actions, and the leader’s actions to it, i.e., hkt = 〈(st′ , akt′ , ωkt′)t′≤t〉 ∈ Hkt
while the leader’s history is the stack of all followers histories ht = 〈hkt 〉k∈N ∈ Hlt. Then, we define the history-based leader’s policy as: µkht(st) = Z
−1p(ωkt |st, ât,ht)p(ât|st,ht) ∝ p(ωkt |st, ât,ht) ∏ k p k ( âkt |st, hkt ) , where Z is the normalization term, pk is the predicted action probability of the k-th follower and â defines the predicted action (predicted by the leader). p is to stress the output is a probability. Since we cannot directly obtain an accurate estimation of ak, we adopt an alternative way to leverage history information and imitation learning to make a prediction of other agents’ action probability function pk ( âkt |st, hkt ) (Implementation details can be found in Appendix B) and p(ωkt |st, ât,ht) is designed using the attention mechanism as well as the sequential decision module presented below.
Follower-Specified Attention Mechanism. Inspired by (Chen et al., 2018), we introduce a follower-specific attention mechanism to identify the important followers where the important followers are followers who has just finished a task and the leader has to commit new actions to these followers. The attention mechanism is as follows:
wkt = exp
( f(A ( st, â k t , h k t )) )∑
k∈N exp ( f(A ( st, âkt , h k t ) )) ; ct = ∑ k∈N wktA ( st, â k t , h k t ) ; ckt = [ct, A ( st, â k t , h k t ) ].
Where wk is the weight of the k-th follower, A(·) : Rds×dakt ×dhkt → Rdc is a function to blend various information of an agent together (dc means the dimension of the output of the A(·)), f(·) : Rdc → R1 is a function to map the blending information to a real number, and ckt is the k-th agent attention value (the compression of history, states and actions for follower k as well as other followers). âkt is the output of p k ( âkt |st, hkt ) , the predicted the k-th follower’s action. This attention mechanism is better because it quantifies the importance of each follower in each state through learning while the original methods only adds/averages all the features (st, âkt , h k t ) together (Shu & Tian, 2019). Another advantage is that its weights can be visualized to see which follower is important to the leader at current step. (Details can be found in Appendices D.2.1 & D.3.4). ckt then is used by βk and µk.
Sequentially Determining Goals and Bonuses. Also notice that the goal and the bonus are sequentially correlated. Therefore, it is better for the leader to choose the bonus and the goal sequentially rather than select them independently. Therefore, to consider the goal and bonus jointly when making a decision, we build a probabilistic graph-based model as: p(ωkt |st, ât,ht) ≈ p(gkt ; bkt |ckt ) ∝ p(bkt |gkt , ckt )× p(gkt |ckt ), the first approximate equation is established because ckt is the compression of (st, ât,ht). p(bkt |gkt , ckt ) and p(gkt |ckt ) means the policy for bonuses and goals (Implementation details can be found in Appendix D.2.1).

heading: 4.3 FOLLOWER ACTION ABSTRACTION POLICY GRADIENT
text: These methods mentioned above are fully implemented can enhance the performance dramatically. But when facing the RL-based followers, the SMG is still hard to converge. This is because in SMG, the policies of the leader and followers are always changing depending on other agents’ performance. To guarantee convergence, the leader can only update its policy when the followers reach (or are near to) the best response policy (Fiez et al., 2019). However, when the followers are RL-based agents, there is no way to ensure the followers’ policies are (near) the best response policies in large-scale SMG and the commonly-seen idea is to provide enough training time but it is unbearable in practice due to the limitation of computing power (Mguni et al., 2019).
To accelerate the training process, inspired by the action abstraction approach which is commonlyseen in Poker (Brown & Sandholm, 2019; Tuyls et al., 2018) and action abstraction RL (Chandak et al., 2019), we collect the followers’ primitive actions sharing the same properties together as a meta policy. Then, the followers only need to select the meta action to make a decision. Therefore, the original game is converted into a meta game, which is easy to solve. Specifically, we define the policy for the k-th follower as: πkt (a k|ŝ) =∑
z π k meta(z|ŝ)πklower(a|ŝ, z), where ŝ = 〈s, ωk〉 is the augmented state for the follower (the combination of current state and the leader’s action). πkmeta(z|ŝ) is the meta policy for the k-th follower and z is the high-level (meta) action. We hypothesize that the lower-level policy (the policy to choose the primitive actions) is already known (rule-based) and deterministic, i.e., πklower(a
k|ŝ, z) = 1. For instance, given the example of the navigation task, the πkmeta can be the selection to which landmark to explore while πklower is a specific route planning algorithm (such
as Dijkstra Algorithm). Based on this assumption, we can design a novel policy gradient to train the meta policy: ∇λkJk ∝ E [ ∇λk log πkmeta(z|ŝ)Rk ] , where λk is the parameter for meta-policy πkmeta (Details can be found in Lemma 3).

heading: 4.4 LOSS FUNCTIONS
text: In this section, we discuss how to design the leader’s and followers’ loss functions.
Loss Functions for the Leaders. The basic structure for the leader is the actor-critic structure (Sutton & Barto, 1998). We find that adding regularizers can enhance the leader’s performance and we implement the maximum entropy for the leader’s policy function as well as the L2 regularization for the termination function, i.e., Lenp = − ∑ k ∑ ωk µ
k(ωk|s,h) logµk(ωk|s,h) and Lreg = β2. We also use imitation learning to learn the predicted action function pk. Following the same logic of (Shu & Tian, 2019), two baseline functions φg(ct) and φb(ct) are also introduced to further reduce the variance. Details can be found in Appendix B.
Loss Functions for the RL-Based Followers. The basic structure for each follower is also based on the actor-critic structure. We leverage the action abstraction policy gradient as we mentioned above. The learning rate between the leader and follower should satisfy the two time-scale principle (Roughly speaking, the leader learns slower than the follower(s)), similar to (Borkar, 1997). Details can be found in Appendix B and the pseudo-code can be found in Appendix C.
5 EXPERIMENTAL RESULTS
5.1 SETUP
Tasks. We evaluate the following tasks to testify the performance of our proposed method. All of these tasks are based on SMG mentioned above. (1) resource collections: each follower collects three types of resources including its preferred one and the leader can choose two bonuses levels (Shu & Tian, 2019); (2) multi-bonus resources col-
lections: based on (1), the leader can choose four bonuses levels; (3) modified navigation: followers are required to navigate some landmarks and after one of the landmarks is reached, the reached landmark disappears and new landmark will appear randomly. (4) modified predator-prey: followers are required to capture some randomly moving preys, prizes will be given after touching them. Both (3) and (4) are based on (Lowe et al., 2017) and we modify them into our SMG setting. Moreover, to increase the difficulty, in each episode, the combinations of the followers will change, i.e., in each task, there are 40 different followers and at each episode, we randomly choose some followers to play the game. More details can be found in Appendix D.
Baselines & Ablations. To evaluate our method, we compare a recently proposed method as our baseline: M3RL (Shu & Tian, 2019). We do not include other baselines because other methods
cannot be used in our problems, as justified in (Shu & Tian, 2019). For the ablations of the leader part, we choose: (1) ours: the full implementation of our method. (2) ours w/o EBPG: removing the event-based policy gradient part; (3) ours w/o Attention: replacing follower-specified attention model by the original attention model mentioned in (Shu & Tian, 2019). For the follower part, we choose (a) with rule-based follower (b) with vanilla RL-based follower, and (c) with action abstraction RL-based follower to testify the ability of our methods when facing different followers.
Hyper-Parameters. Our code is implemented in Pytorch (Paszke et al., 2017). If no special mention, the batch size is 1 (online learning). Similar to (Shu & Tian, 2019), we set the learning rate as 0.001 for the leader’s critic and followers while 0.0003 for the leader’s policy. The optimization algorithm is Adam (Kingma & Ba, 2014). Our method takes less than two days to train on a NVIDIA Geforce GTX 1080Ti GPU in each experiment.
For the loss function, we set the λ1 = 0.01 and λ2 = 0.001. The total training episode is 250, 000 for all the tasks (including both the rule-based followers and the RL-based followers). To encourage exploration, we use the ι-greedy2. For the leader, the exploration rate is set to 0.1 and slightly decreases to zero (5000 episode). For the followers, the exploration rate for each agent is always 0.3 (except for the noise experiments).
5.2 LEARNING EFFICIENCY Table 1: Robustness results in multi-bound resource collections. b% is the probability that followers randomly choose actions.
Methods 0% Noise 30% Noise 50% Noise Ours total incentive 18.32 17.63 17.28
M3RL total incentive 4.06 3.85 4.02 Ours total reward 10.06 5.36 5.30
M3RL total reward -1.58 -3.23 -8.96 The quantitative results with different tasks are shown in Figs. 3 & 4. For the rule-based followers, from Fig. 3, we find that our method outperforms the state-of-the-art method in all the tasks, showing that our method is sample efficient and fast to coverage. There is an interesting phenomenon that in the task of multi-bonus resource collections and navigation, only our method obtains a positive reward, indicating that our method can work well in complicated environments. For ablations, we can see that ours w/o attention and ours w/o EBPG are worse than ours, representing these components do enhance the performance. For the RL-based followers, from Fig. 4, we observe that when facing the RL-based method with action abstraction, our approach outperforms the baseline method in all the tasks (in predator-prey game, the reward for ours is twice as that of the state-of-the-art). We also find that without action abstraction, the reward is less than zero, revealing that the abstraction does play a crucial role in stabilizing training.

heading: 5.3 ROBUSTNESS
text: This experiment is to evaluate whether our method is robust to the noise, i.e., the follower randomly takes actions. We make this experiment by introducing noise into the follower decision. From
Table 1, we can find that our method reaches a higher total reward (more than 5) among all the environment with noise than the state-of-the-art, indicating that our method is robust to the noise. We also observe that the total reward for the baseline method becomes lower with the increase of the noise while our method is more robust to the change. Moreover, for the incentive (the total gain), we find that our method gains much more incentive than the state-of-the-art method, showing that our method coordinates have a better coordination the followers than the state-of-the-art method.

heading: 5.4 MORE EXPERIMENTS
text: We also do a substantial number of experiments. However, due to the space limitation, we can only provide some results here: (1) The total incentives: incentive can reveal the performance of successful rate interacting with the followers. Our method outperforms the state-of-the-art method, indicating that our method has a better ability to interact with the followers. (2) Sparse EBPG: we compare the performance gap between sparse EBPG and (dense) EBPG. This results show that the sparse one is worse than the dense one, supporting the assumption that the dense signal can improve
2Normally it is called the decayed -greedy. We use ι instead of to avoid notation abuse.
the sample efficiency. (3) Visualizing attention: We visualize the attention module to find what it actually learns and the result indicates that our attention mechanism does capture the followers whom the leader needs to assign bonuses to. (4) Two time-scale training: We testify whether our two timescale training scheme works and the ablation shows that this scheme does play an important role in improving the performance of both the leader and the followers. (5) The committing interval: We observe that the dynamic committing interval (our method) performs better than the one with fixed committing intervals. (6) Reward for RL-based followers: we show the reward for the followers, which can provide the situation of the followers. The result represents that our method aids the followers to gain more than the state-of-the-art method. (7) Number of RL-based followers: finally, we testify our method in cases with different number of RL-based followers. The result shows that our method always performs well. The full results can be found in Appendix D.

heading: 6 CONCLUSION REMARKS
text: This paper proposes a novel RL training scheme for Stackelberg Markov Games with single leader and multiple self-interested followers, which considers the leader’s long-term decision process and complicated interaction between followers with three contributions. 1) To consider the long-term effect of the leader’s behavior, we develop an event-based policy gradient for the leader’s policy. 2) To predict the followers’ behaviors and make accurate response to their behaviors, we exploit the leader-follower consistency to design a novel follower-aware module and follower-specific attention mechanism. 3) We propose an action abstraction-based policy gradient algorithm to accelerate the training process of followers. Experiments in resource collections, navigation, and predator-prey game reveal that our method outperforms the state-of-the-art methods dramatically.
We are willing to highlight that SMGs contribute to the RL (especially MARL) community with three key aspects: 1). As we mentioned in the Introduction, most of the existing MARL methods assume that all the agents are willing to sacrifice themselves to maximize the total rewards, which is not true in many real-world non-cooperative scenarios. On the contrary, our proposed method realistically assumes that agents are self-interested. Thus, SMGs provide a new scheme focusing more on the self-interested agents. We think this aspect is the most significant contribution to the RL community. 2). The SMGs can be regarded as the multi-agent system with different roles (the leader and the followers) (Wilson et al., 2008) and our method provides a solution to that problem. 3). Our methods also contribute to the hierarchical RL, i.e., it provides a non-cooperative training scheme between the high-level policy (the leaders) and the low-level policy (the followers), which plays an important role when the followers are self-interested. Moreover, our EBPG also propose an novel policy gradient method for the temporal abstraction structure.
There are several directions we would like to investigate to further extend our SMG model: i) we will consider multiple cooperative/competitive leaders and multiple self-interested followers, which is the case in the labor market, ii) we will consider multi-level leaders, which is the case in the hierarchical organizations and companies and iii) we will consider the adversarial attacks to our SMG model, which may induce extra cost to the leader for efficient coordination. We believe that our work is a preliminary step towards a deeper understanding of the leader-follower scheme in both research and the application to society.

heading: ACKNOWLEDGEMENTS
text: This research is supported by NRF AISG-RP-2019-0013, NSOE-TSS2019-01, MOE and NTU.
Also, this work is supported by the National Natural Science Foundation of China under Grants (U1611264, U1811261,61602530, 61772567, U1811262 and U1711262). This work is also supported by the Pearl River Nova Program of Guangzhou (201906010080).
We would like to thank Tianming Shu, Darren Chua, Suming Yu, Enrique Munoz de Cote, and Xu He for their kind suggestions and helps. We also appreciate the anonymous reviewers for their useful suggestions.

heading: A PROOFS
text: Proposition 1. The policy gradients for termination function βk (sti ,ωti) and leader’s policy function µk ( ωkti |sti ,ωti−1 ) can be written as:
∇θJ(θ) ≈ Eτ∼pτ (·) {[∑ k∈N T∑ i=0 I(eki ) ] Rτ (T ) } ; ∇ϑJ(ϑ) ≈ Eτ∼pτ (·) {[∑ k∈N T∑ i=0 I ′(eki ) ] Rτ (T ) } ;
where θ and ϑ are the parameters for the termination function βkθ and the leader’s policy µ k ϑ. I(·) and I ′(·) are the piece-wise functions:
I(eki ) =  −∇θβk(sti ,ωti−1) 1−βk(sti ,ωti−1)
eki ∈ AT ∇θβk(sti ,ωti−1) βk(sti ,ωti−1) eki /∈ AT . I ′(eki ) =
{ ∇ϑ logµk ( ωkti |sti ,ωti−1 ) eki ∈ AT
0 eki /∈ AT
Proof. First recall the utility for the leader:
J(θ) = EP (τ) [R∗(T )]
= ∑ τ ∑ m P(|AT | = m)P (s0) ∏ k∈N {[∏ eki ∈AT βk(sti ,ωti−1)µ k ( ωkti |sti ,ωti−1 ) πk ( akti |sti , ω k ti )] ×[∏
ekj 6∈AT
( 1− βk(stj ,ωtj−1) ) πk ( aktj |stj , ω k tj )]}∏T t=0 P (st+1|st,at)Rτ (T );
Where m is the number of the times taking new action, m ≤ T . If m = T , implying that the leader has taken new action at each time. P(|AT | = m) means the probability of times taking new action within an episode.
Take derivatives on both LHS and RHS, we get:
∇θJ(θ) = ∑ τ ∑ m ∇θP(|AT | = m)P (s0) ∏ k∈N { ∏ eki ∈AT βk(sti ,ωti−1)µ k ( ωkti |sti ,ωti−1 ) πk ( akti |sti , ω k ti )×  ∏ ekj 6∈AT ( 1− βk(stj ,ωtj−1) ) πk ( aktj |stj , ω k tj )} T∏ t=0 P (st+1|st,at)Rτ (T )
= ∑ τ ∑ m
∇θP(|AT | = m) ∏ k∈N ∏ eki ∈AT βk(sti ,ωti−1) ∏ ekj 6∈AT
(1− βk(stj ,ωtj−1)) P(|AT | = m) ∏ k∈N ∏ eki ∈AT βk(sti ,ωti−1) ∏ ekj 6∈AT (1− βk(stj ,ωtj−1))
× P(|AT | = m)P (s0) ∏ k∈N { ∏ eki ∈AT βk(sti ,ωti−1)µ k ( ωkti |sti ,ωti−1 ) πk ( akti |sti , ω k ti )×  ∏ ekj 6∈AT ( 1− βk(stj ,ωtj−1) ) πk ( aktj |stj , ω k tj )} T∏ t=0 P (st+1|st,at)Rτ (T )
For brevity, we useP (τ) to representP (s0) ∏ k∈N {[∏ eki ∈AT βk(sti ,ωti−1)µ k ( ωkti |sti ,ωti−1 ) πk ( akti |sti , ω k ti )] ×[∏
ekj 6∈AT
( 1− βk(stj ,ωtj−1) ) πk ( aktj |stj , ω k tj )]}∏T t=0 P (st+1|st,at), the trajectory probabil-
ity. And we use i ∈ eki and j ∈ ekj to represent i ∈ eki ∈ AT to j ∈ ekj /∈ AT with a slight abuse of
notation. Thus the equation mentioned above can be further simplified as:
= ∑ τ ∑ m P(|AT | = m)∇θ ∑ k∈N ∑ i∈eki log(1− βk (sti ,ωti−1)) + ∑ k∈N ∑ j∈ekj log βk ( stj ,ωtj−1 )P (τ)Rτ (T ) ≈ Eτ∼pτ (·) ∇θ ∑ k∈N ∑ i∈eki log(1− βk (sti ,ωti−1)) + ∑ k∈N ∑ j∈ekj log βk ( stj ,ωtj−1 )Rτ (T ) 
= Eτ∼pτ (·)  ∑ k∈N ∑ i∈eki −∇θβk (sti ,ωti−1) 1− βk (sti ,ωti−1) + ∑ k∈N ∑ j∈ekj ∇θβk ( stj ,ωtj−1 ) βk ( stj ,ωtj−1 ) Rτ (T )  The equation above is exactly the REINFORCE trick (Sutton & Barto, 1998) and the rule of derivations. The approximation indicates that one trajectory only has one AT 3. Also based on the definition of eki and e k j , the equation can be rewritten in a more compact form:
∇θJ(θ) = Eτ∼pτ (·)  ∑ k∈N ∑ i∈eki −∇θβk (sti ,ωti−1) 1− βk (sti ,ωti−1) + ∑ k∈N ∑ j∈ekj ∇θβk ( stj ,ωtj−1 ) βk ( stj ,ωtj−1 ) Rτ (T )  = Eτ∼pτ (·) {[∑ k∈N T∑ i=0 I(eki ) ] Rτ (T ) } ,
Where I(·) is the piece-wise function:
I(eki ) =  −∇θβk(sti ,ωti−1) 1−βk(sti ,ωti−1)
eki ∈ AT ∇θβk(sti ,ωti−1) βk(sti ,ωti−1) eki /∈ AT .
This is the first part of the proof (the policy gradient for the termination function). Here, we start proving the second part (the policy gradient for the leader’s action).
The proof of the second part is similar to the first part. ∇ϑJ(ϑ) = ∑ τ ∑ m P(|AT | = m)P (s0) ∏ k∈N { ∏ eki ∈AT βk(sti ,ωti−1)µ k ( ωkti |sti ,ωti−1 ) πk ( akti |sti , ω k ti )×  ∏ ekj 6∈AT ( 1− βk(stj ,ωtj−1)) )} T∏ t=0 P (st+1|st,at)Rτ (T );
= ∑ τ ∑ m
∇ϑP(|AT | = m) ∏ k∈N ∏ eki ∈AT µk ( ωkti |sti ,ωti−1 ) P (|AT | = m) ∏ k∈N ∏ eki ∈AT µk ( ωkti |sti ,ωti−1
) × P(|AT | = m)× P (s0)
∏ k∈N ∏ i∈eki βk(sti ,ωti−1)µ k ( ωkti |sti ,ωti−1 ) πk ( akti |sti , ω k ti ) × ∏
ekj 6∈AT
(1− βk(stj ,ωtj−1))πk ( aktj |stj , ω k tj ) Rτ (T )
= ∑ τ ∑ m ∑ k∈N ∑ eki ∇ϑ logµk ( ωkti |sti ,ωti−1 )P (τ)Rτ (T ) = Eτ∼pτ (·)  ∑ k∈N ∑ i∈eki ∇ϑ logµk ( ωkti |sti ,ωti−1 )Rτ (T ) 
3We find that Upadhyay et al. (2018) also implement this approximation but use different explanations.
We rewrite it to a more compact form:
∇ϑJ(ϑ) = Eτ∼pτ (·) {[∑ k∈N T∑ i=1 I ′(ekt ) ] Rτ (T ) }
I ′(eki ) =
{ ∇ϑ logµk ( ωkti |sti ,ωti−1 ) eki ∈ AT
0 eki /∈ AT
Remark 1. Some researches also focus on event-based RL but either on single-agent continuous time (Upadhyay et al., 2018) or reward representation (Gupta et al., 2018). We are the first to develop and implement the event-based policy gradient into the multi-agent system.
Remark 2. In fact, the policy gradient for the leader actions might be somewhat sparse, i.e., we only update the policy when the leader changes its actions. Notice that the leader commits to the same action when eki /∈ AT. Therefore, the probability of leader’s action P (AT) can also represented as:
P (AT) = ∏ k∈N ∏ eki ∈AT βk(sti ,ωti−1)µ k ( ωkti |sti ,ωti−1 ) × ∏
ekj 6∈AT
( 1− βk(stj ,ωtj−1)) ) µk ( ωktj = ω k tj−1|stj ,ωtj−1 ) ;
Then the policy gradient for leader’s policy ∇ϑJ(ϑ) can thus be ∇ϑJ(ϑ) ≈ Eτ∼pτ (·) {[∑ k∈N ∑T i=0 I ′(eki ) ] Rτ (T ) } , where
I ′(eki ) =
{ ∇ϑ logµk ( ωkti |sti ,ωti−1 ) eki ∈ AT
∇ϑµk ( ωkti = ω k ti−1|st,ωti−1 ) eki /∈ AT
Lemma 1. (Reward Bound) For any agent k, the corresponding reward function rk w.r.t ω is CLipschitz continuous.
|rkt ( st, a k t , a −k t ,µ ) − rkt ( st, a k t , a −k t ,µ ′) | ≤ C|µ− µ′|, st,at = [akt , a−kt ]. Where C is a constant that satisfies C > 0.4 µ′ is leader’s new policy. a−k is the joint action without agent k’s.
Proof. Based on Assumption 1, we can build a compact metric space (A × Ω × S, Rk). From the Heine-Cantor theorem we know that the compact metric space induce uniformly continuous.
That is, for every > 0, there exists a δ > 0, such that |µ− µ′| ≤ implies |rkt ( st, a k t , a −k t ,µ ) −
rkt ( st, a k t , a −k t ,µ ′) | ≤ δ. There exists at least one positive Constance C such that δ ≤ C/ . Then |rkt ( st, a k t , a −k t ,µ ) − rkt ( st, a k t , a −k t ,µ
′) | ≤ δ ≤ C ≤ C|µ− µ′|. This Lemma is similar to the assumption in (Mguni et al., 2019). We prove it rather than make it an assumption.
Lemma 2. If Assumption 1 is satisfied, the inequality is established: |Jk( [ πk, π−k ] ,µ)− Jk( [ πk, π−k ] ,µ′)| ≤ C|µ− µ′|
Proof. Expand the utility function by the bellman equation:
4The follower’s reward is originally defined based on the rkt ( st, a k t , a −k t ,ωt ∼ µ ) . Here to emphasize the
relationship between leader’s policyµ and reward, with a little abuse of the notation, we use rkt ( st, a k t , a −k t ,µ ) to represent the reward function.
|Jk( [ πk, π−k ] ,µ)− Jk( [ πk, π−k ] ,µ′)| (1)
= |max π∈Π E
[ rk ( s0, a k 0 , a −k 0 ,µ ) + γ ∑ s1∈S P (s1|s0,a0)V k1 ( [ πk1 , π −k 1 ] ,µ, s1) ]
−max π∈Π E
[ rk ( s0, a k 0 , a −k 0 ,µ ′)+ γ ∑ s1∈S P (s1|s0,a0)V k1 ( [ πk1 , π −k 1 ] ,µ′, s1) ] | (2)
≤ max π∈Π |E{rk
( s0, a k 0 , a −k 0 ,µ ) − rk ( s0, a k 0 , a −k 0 ,µ ′) + γ
∑ s1∈S P (s′|s,a) [ V k1 ( [ πk1 , π −k 1 ] ,µ, s1)− V k1 ( [ πk1 , π −k 1 ] ,µ′, s1) ] }|, (3)
= ∑ s∈S ∞∑ t=0 γtPπ (st+1|st,π) max π∈Π ∣∣E [rk (st, akt , a−kt ,µ)− rk (st, akt , a−kt ,µ′)]∣∣ (4) The first relaxation is due to the property of max: max |A(x)−B(x)| ≥ |maxA(x)−maxB(x)|, where A(x) and B(x) are both the real functions. The Eq. (4) is the result of recursive iteration.
≤(1− γ)−1 max π∈Π |Eπ
[ rk ( st, a k t , a −k t ,µ ) − rk ( st, a k t , a −k t ,µ ′)] | ≤ C|µ− µ′| (5) Where C = (1 − γ)−1C.The last equation is drawn form the Assumption 1 and the inequality of a geometric series: |(I − γpπ)−1| ≤ (1 − γ)−1. Some parts follow the same logic of (Bacon et al., 2017; Mguni et al., 2019; Kakade & Langford, 2002).
Proposition 2. (Leader-Follower Consistency.) If both Assumptions 1 and 2 are satisfied, for every > 0, k ∈ N , there exists δ > 0, such that |µ− µ′| ≤ implies ∣∣πk − π′k∣∣ ≤ δ, where µ′ and π′k are the new policies for the leader and follower k respectively.
Proof. By combining Lemma 2 and Assumption 2, we can draw that:∣∣πk − π′k∣∣ ≤ (1− γ)C|µ− µ′| If there exist |ω − ω′| < and we have:∣∣πk − π′k∣∣ ≤ (1− γ)C|µ− µ′| ≤ (1− γ)C And we set δ ≥ (1− γ)C, the consistency is established.
Lemma 3. (Action Abstraction Policy Gradient.) Under the assumption that the low-level follower policy πklower(at|st, zt) is fixed and deterministic, the policy gradient for action abstraction-based follower can be formulated as:
∇λkJk ∝ E [ ∇λk log πkmeta(z|ŝ)Rk ] ,
Where Rk is the accumulated reward for the k-th follower.
Proof. Recall the utility for the k-th follower: Jk(µ,π) = E [∑T
t=0 γtrkt (st,at,ωt) |at ∼ π (·|st) , st+1 ∼ P (·|st,at) , s0 = s,ωt ∼ µ(·|st,ωt−1)
]
For brevity, we rewrite the object as: Jk = E [∑T
t=0 γ trkt (st,at,ωt)
] with a slight abuse of
notation. The standard policy gradient can be:
∇λkJk = E[∇λk log πkt (akt |st)Rk]
When πklower(a|ŝ, z) is fixed and deterministic, the equation can be:
∇λkJk = E [ ∇λk log[
∑ zt πkmeta(zt|ŝt)πklower(at|ŝt, zt)]Rk ] = E [ ∇λk [ ∑ zt πkmeta(zt|st)πklower(at|ŝt, zt)]∑ zt πkmeta(zt|ŝt)πklower(at|ŝt, zt) Rk ]
= E
[ ∇λk [ ∑ zt πkmeta(zt|ŝt)πklower(at|ŝt, zt)]∑
zt πkmeta(zt|ŝt)πklower(at|ŝt, zt)
Rk ] = E [ [ ∑ zt ∇λkπkmeta(zt|ŝt)πklower(at|ŝt, zt)]∑ zt πkmeta(zt|ŝt)πklower(at|ŝt, zt) Rk ]
= E
[ [ ∑ zt ∇λkπkmeta(zt|ŝt)πklower(at|ŝt, zt)]
Z Rk ] ∝ E [ ∇λk log[πkmeta(zt|ŝt) ] Rk]
Where Z = ∑ z π k meta(zt|ŝt)πklower(at|ŝt, zt) = 1 is the partition function. For brevity, with a slight abuse of notation, we omit the superscript for variables a and z which represents the index of an agent.

heading: B LOSS FUNCTIONS

heading: B.1 LEADER LOSS FUNCTIONS
text: We add a baseline function to reduce the variance of the event-based policy gradient for the leader. We adopt the idea of successor representation (Rabinowitz et al., 2018; Shu & Tian, 2019) as two expected baseline functions: φ̂g(ct) and φ̂b(ct). For the gain baseline function:
φg = ∑ g∈G ∑ k∈N I ( g = gkt ) I ( skt = sg ) vg,
For the bonus-based baseline function: φb = − ∑ g∈G ∑ k∈N I ( g = gkt ) bkt ,
Two baseline neural network functions with parameters θ̂g and θ̂b are trained through minimizing the mean square error:
Lbaseline = (φ g(ct; θ̂g)− φg)2 + (φb(ct; θ̂b)− φb)2,
Where ct is the attention-based latent variable.
To this end, the gradient for the leader can be formulated as:
∇θLpolicyl = −∇θJ ′(θ) +∇ϑ(λ1Lreg); ∇ϑLpolicyl = −∇ϑJ ′(ϑ) +∇ϑ(λ2Lenp). Where
∇θJ ′(θ) ≈ Eτ∼pτ (·) {[∑ k∈N ∑ t I(ekt ) ] (Rτ (T )− φg(ct; θ̂g) + φb(ct; θ̂b)) } ,
∇ϑJ ′(ϑ) ≈ Eτ∼pτ (·) {[∑ k∈N ∑ t I1(ekt ) ] (Rτ (T )− φg(ct; θ̂g) + φb(ct; θ̂b)) } ,
are the baseline policy gradients. We also leverage the imitation learning to learn the action probability function pk ( akt |st, hkt , θ̂I ) similar to (Shu & Tian, 2019), where θI is the parameters for follower-aware module:
LIL = E [ − 1 N ∑ k∈N log pk ( akt |st, hkt ; θ̂I )] .
The history encoder, the state encoder and the attention module are updated with the leader’s policy gradient end-to-end.

heading: B.2 FOLLOWER LOSS FUNCTIONS
text: The follower policy gradient is from Lemma 3 and we find that adding the history hk can improve the performance:
∇λkJk ∝ E [ ∇λk log πkmeta(z|s, hk)Rk ] ,
and the learning rate of the follower α and the leader β satisfy∑ t≥0 αt = ∑ t≥0 βt =∞, ∑ t≥0 α2t + β 2 t <∞, lim t→∞ βt/αt = 0.
Which indicates the follower’s learning rate is much higher than the leader. β̂ is the learning rate for the leader’s critic function, which is the same as α. Followers do not have critic function.

heading: C ALGORITHM
text: Algorithm 1: EBPG Input: The initialized leader’s parameters θ, ϑ and followers’ parameters λ; Output: Well-trained leader and followers;
1 while not converge do 2 if Rollout Stage then 3 for t ≤ T do 4 leader commits to the goals g and bonuses b according to Algorithm 2; 5 send the goals and bonuses to each agent separately; 6 for k in N do 7 The k-th follower receives its own bonus bkt and goal g k t ; 8 each agent make a decision akt = π k t (a k t |ŝt) ;
9 end 10 st+1, r l t, {rkt }k∈N = step(st,at) ; // transition function. 11 end 12 store 〈st,at,ωt, st+1, rlt, {rkt }k∈N 〉 in episode buffer; 13 else if Training Stage then 14 for k ∈ N do 15 λk ← λk − α∇λkJk; // the follower’s parameters 16 end 17 θ̂g ← θ̂g − β̂∇θ̂gLbaseline, θ̂b ← θ̂b − β̂∇θ̂bLbaseline ; // the critic’s
parameters
18 θ̂I ← θ̂I − βLIL ∇θ̂ILIL; // the follower-aware module’s parameters 19 θ ← θ − β∇θLpolicyl ; // the termination function’s parameters 20 ϑ← ϑ− β∇ϑLpolicyl ; // the parameters of leader’s policy; 21 end
Algorithm 2: Action Choices for Leader Input: Leader’s policy µ Output: goals g and bonuses b;
1 for k ∈ N do 2 sample the termination function βkt ; 3 if βkt terminates previous leader’s action ωkt−1 then 4 choose new action ωkt ; 5 else 6 ωkt = ω k t−1; // maintain previous action.

heading: D EXPERIMENT DETAILS

heading: D.1 TASKS DETAILS
text: The illustration of experimental scenarios can be found in Figure 5. Here we give some details about these environments:
Resource Collections. This task is similar to (Shu & Tian, 2019), which is based on the scene that the leader and the followers collect some resources. Each follower has its own preference which might be the same (or against) to the leader’s preference. In order to make the followers obey the leader’s instruction, the leader should pay the followers bonuses. There are total 4 types of resources and for different resources each agent has different preferences. The leader owns two type of bonus (1 or 2) and 4 types of goals (each resource is a goal). The number of leader is 1 while the number of followers is 4.
Multi-Bonus Resource Collections. This task is similar to Resource Collections. Except that the leader can take 4 level bonuses (a bonus from 1 to 4) while each agent owns one skill. The number of leader is 1 while the number of followers is 4.
Modified Navigation. This task is original from (Lowe et al., 2017). We make some modifications here to make it suitable our SMG: the leader and the followers are going to navigate some landmarks. Each follower has its own preference which might be the same (or against) to the leader’s preference. When a landmark has been navigated, it disappears immediately and a new landmark will appear. There are total 6 types of landmarks and for different landmarks, each agent has different preferences. The leader owns two type of bonuses and 6 types of goals (each landmark is a goal). The number of leader is 1 while the number of followers is 8 and the number of the landmarks is 6.
Modified Predator-Prey. This task is also original from (Lowe et al., 2017). We make some modification here to make it suitable our SMG: the leader and the followers are going to catch some preys. Each follower has its own preference which might be the same (or against) to the leader’s preference. In each step, whether a prey has been caught, it randomly chooses a direction to go. Catching a prey will not make it disappear, which means that the preys can exist until the game ends. There are total 8 types of preys and for different preys, each agent has different preferences. The leader owns two type of bonuses and 8 types of goals (each prey is a goal). The followers are faster than the preys. The number of leader is 1 while the number of followers is 10 and the number of the landmarks is 8.
Reward Design. The rewards mentioned in Section 3 are the general forms. Here, we define two specified forms of the leader and followers reward function in our experiments:
Leader Reward. We define vg as the prize (utility) for finishing task g. We set the reward function for the leader at step t as: rlt = ∑ g∈G ∑ k∈N I ( g = gkt ) ( I ( skt = sg ) vg − bkt ) , formulated by the
total gain ∑ g∈G ∑ k∈N I ( g = gkt ) I ( skt = sg ) vg (the total prizes got by the leader at time t) minus
the total payment− ∑ g∈G ∑ k∈N I ( g = gkt ) bkt (the total bonuses paid to the followers). We should
emphasize that our leader reward is total different from the (Shu & Tian, 2019): in their approaches, the leader changes its mind after signing a contract will not be punished. To make it suitable to the real world, we modify the reward as the leader should pay the followers bonuses immediately after signing the contract and cannot get back if it gives up the contract.
Follower Reward. For the followers, we set the reward for the k-th follower as: rkt = ∑ g r k g,t =(
uk,g I ( skt = sg ) + I ( glt = g k t ) × bkt ) , where uk,g reveals the payoff of the k-th follower when finishing task g (the preference). Specifically, rkt indicates that the follower can either follow the leader’s instruction or just do what it prefers to. The followers will receive reward immediately after signing the contract (the leader and the followers achieve an agreement). I ( skt = sg ) means that the follower finishes the task g at step t. A penalty is added to the followers if the followers betray the leader (the followers and the leader sign the contract but the followers do not agree to work).

heading: D.2 TRAINING DETAILS

heading: D.2.1 NETWORK DESIGN
text: Our network is based on (Shu & Tian, 2019). Some do not suit our method. We do some modification here: (1) We change the vanilla attention mechanism (sum/average all the history and action of each follower together) to a follower-specified one: each follower has a weight which indicates how important the follower is at the current step. (2) The output for g and b are changed into the sequential form, i.e., we first calculate p(gkt |ckt ) to get gkt , then based on gkt , we choose p(bkt |gkt , ckt ). (3) The history information hkt is compressed by the LSTM (Gers et al., 2000).
Leader’s Network. For the history information, we are willing to stress that the history consists of two parts, one is the statistical information similar to (Shu & Tian, 2019) and the other is the past information given by the neural networks. We leverage the LSTM with 128 hidden units and two fully-connected layers to compress the past information and the statistical information and obtain hkt . Each layer contains 128 neurons.
For the state information, we encode it into a 128-dimension hidden state by a state encoder. For resource collections task, the state encoder consist of a convolutional layer with 64 channels and kernels of 1×1 and two fully connected layers with 128 neurons. For navigation and predator-prey task, we encode the state using two fully connected layers with 128 neurons.
For the follower-specific attention module, we firstly predict the âkt from the follower-aware module which contains an LSTM with 128 hidden units and two fully-connected layers with 128 neurons. Secondly, we transform âkt , h k t and the output of the state encoder into A(st, â k t , h k t ) using a fully connected layer with 128 neurons. The hidden vector is generated by concatenating all the A(st, â k t , h k t ). Then the attention weight w is calculated using two fully connected layers with softmax activation. The ct is calculated as a weighted summation of w and A(·). Finally, we obtain the k-th attention value ckt by concatenating ct and A(st, â k t , h k t ).
For sequential graph-based model, we firstly calculate p(gkt |ckt ) using two fully connected layers. Each layer consists of 128 neuron units. The input of layers is ckt and the output is g k t . Then we obtain bkt by two fully connected layers. The input of layers is the concatenation of c k t and g k t .
Follower’s Network. We construct the RL-based followers by a state encoder and two fully connected layers. Each fully connected layer contains 128 neurons. For resource collections task, the state encoder consists of a convolutional layer with 64 channels and kernels of 1×1, two fully connected layers with 128 neurons and an LSTM with 128 hidden units. For navigation and predatorprey task, we encode the state using two fully connected layers with 128 neurons and an LSTM with 128 hidden units. Notice that the network structure of action-abstraction RL-based follower and non-action-abstraction RL-based follower is similar. The only difference is that the output of the former is high-level action z and the output of the latter is low-level action akt .

heading: D.3 MORE EXPERIMENTAL RESULTS

heading: D.3.1 TOTAL INCENTIVE FUNCTION
text: Total incentive (income) functionsRin = ∑ t r l t+ ∑ t ∑ k b k t can reveal how well the leader interacts with the followers; the higher the Rin is, the more successful the coordination between the leader and the followers.
From Figure 6 and 7, comparing with the state-of-the-art method, we can see that our method far outperforms the state-of-the-art method, which reveals that our method does have a better ability to coordinate with the followers. In all of the scenarios, without the EBPG, the performance of our method is worse than ours with EBPG. Specifically, in some scenarios (e.g., multi-bonus resource collections, navigation), without the EBPG, the performance of our method is (or nearly) similar to the performance of M3RL, showing the effectiveness of our novel policy gradient. Moreover, we can notice that in navigation environment, without follower-specified attention, the performance of our method diminishes rapidly, which implies that in some scenarios, attention does play an important role.

heading: D.3.2 ROBUST OF OUR METHOD
text: In this section, we are going to testify the robust of our method. Specifically, we evaluate whether our method is robust to the noise. We make this experiment by introducing noise into the follower decision. For example, if we set the noise function as 30%, indicating that there is 30% probability that the followers will choose action randomly.

heading: D.3.3 SPARSE EVENT-BASED POLICY GRADIENT
text: This experiment testifies whether the dense event-based policy gradient increases the leader’s performance comparing with the sparse event-based policy gradient. We make ablations here: (1) Ours: the full structure of our method; (2) Sparse Event-Based Policy Gradient (sparse EBPG): the fully structure of ours except that the EBPG is replaced by sparse event-based policy gradient; (3) sparse EBPG w/o attention: replacing the follower-specified attention mechanism by averaging the input features.
From Figure 8 and 9 we can find that if the policy gradient is sparse, its performance is worse than the dense one, implying that the dense method does improve the leader’s performance. There is also an interesting phenomenon that sparse EBPG with follower-specified attention mechanism performs better than that without, revealing that the attention can stabilize training when the training signal is sparse.
D.3.4 VISUALIZING ATTENTION
Following the same logic of (Iqbal & Sha, 2019), we visualize the weight of the attention when the leader takes actions. From Figure 10, we find that the attention mechanism does learn to strongly attend to the followers that the leader needs to take actions. The followers with leader’s commitment obtain much higher attention weight than others, showing that the attention module actually learn to identify the important followers while leader committing new action. Thus, the attention mechanism does play an important role in improving the performance.

heading: D.3.5 TWO TIME-SCALE UPDATING
text: In order to evaluate the performance of our two time-scale update scheme (TTSU), we do an ablation study as shown in Fig 11. We can find that the performance where the followers’ learning rate α (1 × 10−3) is much larger than the leader’s β (3 × 10−4) is better than the performance where the leader’s learning rate is similar to the followers (1 × 10−3). Moreover, without TTSU, the reward curves of training methods become unstable, revealing that TTSU can stabilize the training process. In fact, TTSU improves the rate of convergence and play an important role in improving performance.

heading: D.3.6 COMMITTING INTERVAL
text: We evaluate the leader’s performance between static committing interval and our dynamic committing interval. As shown in Figure 12, we observe that all the different fixed committing intervals only change the rate of convergence and do not enhance the leader’s performance. All the fixed
committing intervals are much worse than our dynamic committing approach, revealing the fact that our dynamic committing approach aids a lot in improving the leader’s performance.
? The last four terms are so close to zero and we use the rewards from last episode as the final results.

heading: D.3.7 REWARD FOR RL-BASED FOLLOWERS
text: We are interesting in the reward for the RL-based follower(s). Intuitively, a well-performing leader can make the follower gain more. As shown in Figure 13, the reward for RL follower is higher than M3RL follower in all the tasks. This represents the leader can coordinate the followers better and make them gain more reward than other methods, which forms a win-win strategy.

heading: D.3.8 NUMBER OF THE FOLLOWER AGENTS
text: Finally, we evaluate the leader’s performance with different number of RL-based followers. As shown in Figure 15, we find that our method outperforms the state-of-the-art method when facing different number of RL-based workers.

heading: D.3.9 DIFFERENT COMBINATIONS STUDY
text: To further illustrate the performance of different combinations of our methods, we make an extra ablation here. We choose the resource collections as the environments. Our analysis is as follows:
For the RL based followers scenario, As shown in Table 2, we find that the action abstraction policy gradient is very important to converge. Additionally, adding different modules can improve the performance and the method with all the modules reach the highest reward than other combinations. The contribution ranking for each module is: Action abstraction policy gradient > EBPG > Leaderfollower consistency.

Review summaries:
This paper aims to improve the performance of leader agents in leader-follower Stackelberg Markov games (SMG) by introducing architectural components. an event-aware policy gradient for the temporally abstracted level of the leader, and a follower abstraction technique that mitigates the impact of follower distributional shift. The authors show significant improvements over the state of the art on several problems, and also evaluate the importance of the different components of their approach with thorough ablations.