An Efficient Transfer Learning Framework for Multiagent Reinforcement Learning
Year: 2021

Abstract:
Transfer Learning has shown great potential to enhance single-agent Reinforcement Learning (RL) efficiency. Similarly, Multiagent RL (MARL) can also be accelerated if agents can share knowledge with each other. However, it remains a problem of how an agent should learn from other agents. In this paper, we propose a novel Multiagent Policy Transfer Framework (MAPTF) to improve MARL efficiency. MAPTF learns which agent’s policy is the best to reuse for each agent and when to terminate it by modeling multiagent policy transfer as the option learning problem. Furthermore, in practice, the option module can only collect all agent’s local experiences for update due to the partial observability of the environment. While in this setting, each agent’s experience may be inconsistent with each other, which may cause the inaccuracy and oscillation of the option-value’s estimation. Therefore, we propose a novel option learning algorithm, the successor representation option learning to solve it by decoupling the environment dynamics from rewards and learning the option-value under each agent’s preference. MAPTF can be easily combined with existing deep RL and MARL approaches, and experimental results show it significantly boosts the performance of existing methods in both discrete and continuous state spaces.

heading: 1 Introduction
text: Transfer Learning has achieved expressive success of accelerating single-agent Reinforcement Learning (RL) via leveraging prior knowledge from past learned policies of relevant tasks [37, 36]. Inspired by this, transfer learning in Multiagent Reinforcement Learning (MARL) [6, 17, 4, 16, 7, 8] is also studied with two major directions: 1) transferring knowledge across different but similar tasks and 2) transferring knowledge among multiple agents in the same task. The former usually explicitly computes similarities between tasks [18, 3, 10] to transfer across similar tasks with the same number of agents, or design network structures to transfer across tasks with different numbers of agents [1, 33]. In this paper, we focus on the latter due to the following intuition: in a Multiagent System (MAS), each agent’s experience is different, so the states each agent visits (the familiarities to different regions of the environment) are also different; if we figure out some principled ways to transfer knowledge across different agents, all agents could form a big picture even without exploring the whole space of the environment, which will facilitate more efficient MARL.
∗Equal contribution. † Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
In fact, the latter direction is still investigated at an initial stage, and the assumptions and designs of some recent methods are usually simple. For example, LeCTR [27] and HMAT [19] adopt the teacher-student framework to enable each agent to learn when to advise other agents or receive advice from other agents. However, they only consider a two-agent scenario. Later, PAT [22] extends this idea to scenarios with more than two agents, and enables each agent to learn from other agents through an attentional teacher selector. However, it simply uses the difference of two unbounded value functions as the student reward which may cause instability.
DVM [32] and LTCR [35] are two methods to transfer knowledge among multiple agents through policy distillation. However, they simply decompose the training process into two stages (i.e., the learning phase and the transfer phase) by turns, which is a coarse-grained manner. Moreover, they consider the equal significance of knowledge transfer throughout the whole training process, which is counter-intuitive. A good transfer should be adaptive rather than being equally treated, e.g., the transfer should be more frequent at the beginning of the training since agents are less knowledgeable about the environment, while decay as the training process continues because agents are familiar with the environment gradually and should focus more on their own experiences.
In this paper, we propose a novel Multiagent Policy Transfer Framework (MAPTF) which models the policy transfer among multiple agents as the option learning problem. In contrast to previous teacherstudent and policy distillation frameworks, MAPTF is adaptive and applicable to scenarios consisting of more than two agents. Specifically, MAPTF adaptively selects a suitable policy for each agent to exploit, which is imitated by an agent as a complementary optimization objective. MAPTF also uses the termination probability as a performance indicator to determine whether the exploitation should be terminated to avoid negative transfer. Furthermore, due to partial observability of the environment, the update of the option-value function is based on all agent’s local experience. However, in this setting, each agent’s experience may be inconsistent, which could cause the option-value estimation to oscillate and become inaccurate. A novel option learning algorithm, the Successor Representation Option (SRO) learning is used to overcome this inconsistency by decoupling environment dynamics from rewards to learn the option-value function under each agent’s preference. MAPTF can be easily incorporated into existing Deep RL and MARL approaches. Our simulations show it significantly boosts the performance of existing approaches both in discrete and continuous state spaces.

heading: 2 Preliminaries
text: Partially Observable Stochastic Games. Stochastic Games [24] are a natural multiagent extension of Markov Decision Processes (MDPs), which model the dynamic interactions among multiple agents. Considering the fact agents may not have access to the complete environmental information, we follow previous work’s settings and model the multiagent learning problems as partially observable stochastic games [13]. A Partially Observable Stochastic Game (POSG) is defined as a tuple 〈N ,S,A1, · · · ,An, T ,R1, · · · ,Rn,O1, · · · ,On〉, whereN is the set of agents; S is the set of states; Ai is the set of actions available to agent i (the joint action space A = A1 ×A2 × · · · × An); T is the transition function that defines transition probabilities between global states: S ×A×S → [0, 1]; Ri is the reward function for agent i: S × A → R and Oi is the set of observations for agent i. A policy πi: Oi × Ai → [0, 1] specifies the probability distribution over the action space of agent i. The goal of agent i is to learn a policy πi that maximizes the expected return with a discount factor γ: J = Eπi [∑∞ t=0 γ trit ] .
The Options Framework. Sutton et al. [30] firstly formalized the idea of temporally extended action as an option. An option ω ∈ Ω is defined as a triple {Iω, πω, βω} in which Iω ⊂ S is an initiation state set, πω is an intra-option policy and βω : Iω → [0, 1] is a termination function that specifies the probability an option ω terminates at state s ∈ Iω . An MDP endowed with a set of options becomes a Semi-MDP, which has a corresponding optimal option-value function over options learned using intra-option learning. The options framework considers the call-and-return option execution model, in which an agent picks an option ω according to its option-value function Qω(s, ω), and follows the intra-option policy πω until termination, then selects a next option and repeats the procedure.
Deep Successor Representation (DSR). The successor representation (SR) [9] is a basic scheme that describes the state value function by a prediction about the future occurrence of all states under a fixed policy. SR decouples the dynamics of the environment from the rewards. Given a transition
(s, a, s′, r), SR is defined as the expected discounted future state occupancy:
M(s, s′, a) = E [ ∞∑ t=0 γt1[st = s ′]|s0 = s, a0 = a ] , (1)
where 1[.] is an indicator function with value of one when the argument is true and zero otherwise. Given the SR, the Q-value for selecting action a at state s can be formulated as the inner product of the SR and the immediate reward: Qπ(s, a) = ∑ s′∈SM(s, s
′, a)R(s′). DSR [21] extends SR by approximating it using neural networks. Specifically, each state s is represented by a feature vector φs, which is the output of the network parameterized by θ. Given φs, SR is represented as msr(φs, a|τ) parameterized by τ , a decoder gθ̄(φs) parameterized by θ̄ outputs the input reconstruction ŝ, and the immediate reward at state s is approximated as a linear function of φs: R(s) ≈ φs ·w, where w ∈ RD is the weight vector. In this way, the Q-value function can be approximated by putting these two parts together as: Qπ(s, a) ≈ msr(φs, a|τ) ·w. The stochastic gradient descent is used to update parameters (θ, τ,w, θ̄). Specifically, the loss function of τ is:
L(τ, θ) = E [ (φs + γm ′ sr(φs′ , a ′|τ ′)−msr(φs, a|τ)) 2 ] , (2)
where a′ = argmaxamsr(φ ′ s, a) ·w, and m′sr is the target SR network parameterized by τ ′ which follows DQN [26] for stable training. The reward weight w is updated by minimizing the loss function: L(w, θ) = (R(s)− φs ·w)2 . The parameter θ̄ is updated using an L2 loss: L(θ̄, θ) = (ŝ− s)2 . Thus, the loss function of DSR is the composition of the three loss functions: L(θ, τ,w, θ̄) = L(τ, θ) + L(w, θ) + L(θ̄, θ).
3 Multiagent Policy Transfer Framework (MAPTF)
3.1 Framework Overview
Agent𝑖𝑖
Actor
Critic
Critic Loss
Actor Loss
Termination Network
Option Network
Agent1 Agent2 Agent𝑛𝑛
Environment OObbss𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒊𝒊𝒊𝒊
OObbss𝒐𝒐𝒐𝒐 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊
Option Module
Replay Buffer
Reward 𝒓𝒓𝒊𝒊 Obs 𝒐𝒐𝒊𝒊
Action 𝒂𝒂𝒊𝒊
𝑎𝑎𝑖𝑖
𝑜𝑜𝑖𝑖
𝑳𝑳𝒕𝒕𝒓𝒓𝒊𝒊 = 𝒇𝒇 𝒕𝒕 𝐃𝐃𝐃𝐃𝐃𝐃(𝝅𝝅𝝎𝝎,𝝅𝝅𝒊𝒊)
𝑜𝑜𝑖𝑖′ 𝑟𝑟𝑖𝑖
𝒐𝒐𝒊𝒊 𝒂𝒂𝒊𝒊 𝒓𝒓𝒊𝒊 𝒐𝒐𝒊𝒊′ 𝒊𝒊 𝝎𝝎
𝜷𝜷𝝎𝝎
𝝎𝝎 𝝎𝝎𝒐𝒐𝒐𝒐𝒐𝒐
Choice 𝝎𝝎
Agent policy
𝜋𝜋𝑛𝑛
𝑜𝑜𝑖𝑖
…𝜋𝜋1 𝜋𝜋𝑗𝑗 …
Figure 1: Framework overview.
In this section, we describe our MAPTF in detail. Figure 1 illustrates the MAPTF, which contains two modules, the agent’s module with n agents interacting with an environment, and the option module which determines which agent’s policy is useful for each agent. The option module first initializes the option set with n options: Ω = {ω1, ω2, · · · , ωn}, each ωi is a tuple {Iωi , πωi , βωi} and πωi equals to agent i’s policy πi. At the start of each episode, for each agent, the option module selects an option ω based on the option-value function and the termination function. Each option terminates according to its termination function and then another option is selected to repeat the process. During the training phase, the option module uses experiences from all agents to update the option-value function and corresponding termination function. Each agent will exploit the knowledge from another agent πω based on the selected option ω. This is achieved through policy imitation, which serves as a complementary optimization objective (the option module is responsible for this and each agent does not know which policy it imitates and how the extra loss function is calculated). The exploitation is terminated as the selected option terminates, and then another option is selected to repeat the process. In this way, each agent efficiently exploits useful information from other agents, and as a result, the learning process of the whole system is accelerated and improved.
In the following section, all agents share the same option set based on the assumption that all agents are homogeneous and each agent’s policy may be helpful for other agents. MAPTF would be easily
established in the situation where the option module initializes different option sets for each agent, e.g., each agent only needs to imitate a small number of agents. In this case, instead of inputting states into the option-value network and outputting a fixed number of option-values, we input each state-option pair to the network and output a single option-value [23, 25, 11].

heading: 3.2 MAPTF
text: Algorithm 1 MAPTF-PPO Input: option set Ω = {ω1, ω2, · · · , ωn}, replay buffer Di, parameters of actor network ρi and critic network υi for each agent i
1: for each episode do 2: Select an option ω for each agent i 3: Select an action ai ∼ πi(oi) for each agent i 4: Perform ~a, observe ~r and new state s′ 5: Store transition (oi, ai, ri, oi ′
, ω, i) to Di 6: Select ω′ if ω terminates for each agent i 7: for each agent i do 8: Optimize the critic loss w.r.t υi (Equation 3) 9: MAPTF calculates the transfer loss Litr 10: Optimize the actor loss w.r.t ρi (Equation 4) 11: end for 12: Update the option module (Algorithm 2) 13: end for
In this section, we describe how MAPTF is applied in PPO [29], a popular single-agent RL algorithm. The way MAPTF combines with other RL and MARL algorithms is similar. The whole process of MAPTF combined with PPO is shown in Algorithm 1. With the input of n options Ω = {ω1, ω2, · · · , ωn}, for each episode, the option module selects an option ω for each agent (Line 2), and each agent selects an action ai following its policy πi (Line 3). The joint action ~a is performed, then the reward r and new state s′ is returned from the environment (Line 4). The transition is stored in each agent’s replay buffer Di (Line 5). If ω terminates, then the option module selects another option ω ′ for each agent (Line 6).
For each update step, each agent updates its critic network by minimizing the loss Lic (Line 8):
Lic = − T∑ t=1 ( ∑ t′>t γt ′−trit − Vυi(oit))2, (3)
where T is the length of the trajectory segment in PPO. Then each agent updates its actor network by minimizing the summation of the original loss and the transfer loss Litr (Line 10):
L̄ia = T∑ t=1 πi(ait|oit) πiold(a i t|oit) Ai − λKL[πiold|πi] + Litr, (4)
where Ai = ∑ t′>t γ t′−trit − Vυi(oit) is the advantage function of agent i.
To transfer useful knowledge among agents, MAPTF calculates the distance Dis(πω|πi) between each exploited policy πω and each agent’s policy πi, and transfers the loss to each agent respectively, serving as a complementary optimization objective for each agent. This means that apart from maximizing the cumulate reward, each agent also imitates another agent’s policy πω by minimizing the loss function Litr as follows:
Litr = f(t)Dis(π ω|πi), (5)
where, f(t) = 0.5 + tanh(3− µt)/2 is the discounting factor. µ is a hyper-parameter that controls the decreasing degree of the weight. This means that at the beginning of learning, each agent exploits knowledge from other agents mostly. As learning continues, knowledge from other agents becomes less useful and each agent focuses more on the current self-learned policy. We consider the MAPTF is a general framework that can be combined with any existing Deep RL and MARL algorithms.
For policy-based algorithms, the corresponding term is the cross-entropy loss: H(πω|πi) (and other choices of the distance metric are also suitable). For value-based algorithms, MAPTF measures the distance of two Q-value distributions.
The next issue is how to update the option module. To evaluate which agent’s policy is useful for each agent, the option module needs to collect all agent’s experiences for the update. What if the experience from one agent is inconsistent with others? In a POSG, each agent can only obtain the local observation and individual reward signal, which may be different for different agents even at the same state, e.g., each agent has an individual goal to achieve or has different roles, and the rewards assigned to each agent are different. If we use inconsistent experiences to update the same option-value and termination probability, the estimation of the option-value function would oscillate and become inaccurate. To this end, We propose a novel option learning algorithm, the Successor Representation Option (SRO) to address this problem, which is described in the next section.
3.3 SRO Learning
MAPTF applies a novel option learning algorithm, Successor Representation Option (SRO) learning to learn the option-value function under each agent’s preference. The SRO network architecture is shown in Figure 2, with each observation oi from each agent i as input. oi corresponding to the global state s is inputted through two fully-connected layers to generate the state embedding φoi , which is transmitted to three network sub-modules. The first sub-module contains the state reconstruction model which ensures φoi well representing oi, and the weights for the immediate reward approximation at local observation oi. The immediate reward is approximated as a linear
function of φoi : Ri(φoi) ≈ φoi ·w, where w ∈ RD is the weight vector. The second sub-module is used to approximate SR for options msr(φoi , ω|τ) which describes the expected discounted future state occupancy of executing the option ω. The last sub-module is used to update the termination probability β(φ
oi ′ , ω|$).
Givenmsr(φoi , ω|τ), the SRO-value function can be approximated as: Qω(φoi , ω) ≈ msr(φoi , ω|τ)· w. Since options are temporal abstractions [30], SRO also needs to calculate the Ũ function which is served as SRO upon arrival, indicating the expected discounted future state occupancy of executing an option ω upon entering a local observation oi ′ :
Ũ(φ oi ′ , ω|τ ′) = (1− β(φ oi ′ , ω|$))msr(φoi′ , ω|τ ′)+
β(φ oi ′ , ω|$)msr(φoi′ , ω ′|τ ′),
(6)
where ω′ = argmaxω∈Ωmsr(φoi′ , ω|τ ′) ·w.
The learning process of SRO is shown in Algorithm 2. It first initialized the network parameters for the SRO network and the target network. Then, for each update step, it samples a batch of B/N transitions from each agent’s buffer Di, which means there are B transitions in total (Line 2). SRO loss is composed of three components: the state reconstruction loss L(θ̄, θ), the loss for reward weights L(w, θ) and SR loss L(τ, θ). The state reconstruction network is updated by minimizing two losses L(θ̄, θ) and L(w, θ) (Lines 3,4):
L(θ̄, θ) = ( gθ̄(φoi)− oi )2 ,
L(w, θ) = ( ri − φoi ·w )2 .
(7)
The second sub-module, SR network approximates SRO and is updated by minimizing the standard L2 loss L(τ, θ) (Lines 5-8):
L(τ, θ) = 1
B ∑ b (yb −msr(φoi , ω|τ)) 2 , (8)
where yb = φoi +γŨ(φoi′ , ω|τ). At last, the termination probability of the selected option is updated. According to the call-and-return option execution model, the termination probability β~ω controls
Algorithm 2 SRO Learning. Input: option set Ω = {ω1, ω2, · · · , ωn}, parameters of state feature θ, reward weights w, state reconstruction θ̄, termination network $, SR network τ , SR target network τ ′; replay buffer Di for each agent i;
1: for each update step do 2: Select B/N samples (oi, ai, ri, oi ′
, ω, i) from each Di 3: Optimize L(θ̄, θ) w.r.t θ̄, θ (Equation 7) 4: Optimize L(w, θ) w.r.t w, θ (Equation 7) 5: for each ω do 6: if πω selects action ai at observation oi then 7: Calculate Ũ(φ
oi ′ , ω|τ ′) (Equation 6)
8: Optimize L(τ, θ) w.r.t τ (Equation 8) 9: Optimize the termination network w.r.t $ (Equation 9)
10: end if 11: end for 12: Copy τ to τ ′ every k steps 13: end for
when to terminate the selected option and then to select another option accordingly, which is updated w.r.t $ as follows (Line 9):
$ = $ − α$ ∂β(φ oi ′ , ω|$)
∂$
( A(φ
oi ′ , ω|τ ′) + ξ
) , (9)
where A(φ oi ′ , ω|τ ′) is the advantage function and approximated as msr(φoi′ , ω|τ ′) · w − maxω∈Ωmsr(φoi′ , ω|τ ′) · w, and ξ is a regularization term to ensure explorations [2, 36]. At last, the target network parameterized by τ ′ copies from τ every k steps (Line 12).

heading: 4 Experimental Results
text: We evaluate the performance of MAPTF combined with the popular single-agent RL algorithm (PPO [29]) and MARL algorithm (MADDPG [25] and QMIX [28]) on two representative multiagent games, Pac-Man [31] and multiagent particle environment (MPE) [25] (illustrated in the appendix). Specifically, we first combine MAPTF with PPO on Pac-Man to validate whether MAPTF successfully solves the sample inconsistency due to the partial observation. Then, we combine MAPTF with three baselines (PPO, MADDPG and QMIX) on MPE to further validate whether MAPTF is a more flexible way for knowledge transfer among agents. We also compare with DVM [32], which is a recent multiagent transfer method. All results are averaged over 10 seeds. More experimental details and parameters settings are detailed in the appendix, source code is provided on https: //github.com/tianpeiyang/MAPTF_code.

heading: 4.1 Pac-Man
text: Pac-Man [31] is a mixed cooperative-competitive maze game with one pac-man player and several ghost players. We consider three Pac-Man scenarios (OpenClassic, MediumClassic, and Original-
Classic) with the game difficulties increasing. The goal of the pac-man player is to eat as many pills as possible and avoid the pursuit of ghost players. For ghost players, they aim to capture the pac-man player as soon as possible. In our settings, MAPTF controls several ghost players to catch a pac-man player controlled by a well pre-trained PPO policy. The game ends when one ghost catches the pac-man player, or the episode exceeds 100 steps. Each ghost player receives −0.01 penalty for each step and a +5 reward for catching the pac-man player.
Figure 3 (a) presents the average rewards on the OpenClassic scenario. We can see that MAPTF performs better than other methods and achieves the average discount rewards of +3 approximately with a smaller variance. In contrast, PPO and DVM only achieve the average discount rewards of +2.5 approximately with a larger variance. This phenomenon indicates that MAPTF enables efficient knowledge transfer between ghost players, thus facilitating better performance.
Next, we consider a complex scenario with a larger layout size than the former, and it contains obstacles (walls). Figure 3 (b) shows the advantage of MAPTF is much more apparent compared with PPO and DVM. Furthermore, MAPTF performs best among all methods, which means it effectively recognizes more useful information for each agent. MAPTF performs better than DVM because MAPTF enables each agent to effectively exploit useful information from other agents, which successfully avoids negative transfer when other agents’ policies are only partially useful. However, DVM just transfers all information from other agents through policy distillation without distinction.
Finally, we consider a scenario with the largest layout size, and four ghost players catching one pac-man player. Similar results can be observed in Figure 3 (c). By comparing the results of the three scenarios, we see that the superior advantage of MAPTF increases when faced with more challenging scenarios. Intuitively, as the environmental difficulties increase, agents are harder to explore the environment and learn the optimal policy. In such a case, agents need to exploit other agents’ knowledge more efficiently, which would significantly accelerate the learning process, as demonstrated by MAPTF.
4.2 MPE
0.0 0.4 0.8 1.2 1.6 2.0 Training episodes ×104
0
100
200
300
Av er
ag e
re w
ar d
PPO MAPTF DVM
(a) Predator-prey (N = 4)
0.0 0.4 0.8 1.2 1.6 2.0 Training episodes ×104
0
800
600
400 200 A ve ra ge
r ew
ar d
PPO MAPTF DVM
(b) Predator-prey (N = 12)
0.0 0.4 0.8 1.2 1.6 2.0 Training episodes ×104
−250
−200
−150
−100
Av er
ag e
re w
ar d
PPO MAPTF DVM
(c) Cooperative navigation (N = 6)
0.0 0.4 0.8 1.2 1.6 2.0 Training episodes ×104
−250
−200
−150
−100
Av er
ag e
re w
ar d
PPO MAPTF DVM
(d) Cooperative navigation (N = 10)
MPE [25] is a multiagent particle world with continuous observation and discrete action space. We consider two scenarios of MPE: predator-prey and cooperative navigation. The predator-prey contains three (nine) agents which are slower and want to catch one (three) adversary (rewarded +10 by each hit). The adversary is faster and wants to avoid being hit by the other three (nine) agents. Obstacles block the way. The cooperative navigation contains six (ten) agents to cover six (ten) corresponding landmarks. Agents are penalized with a −1 penalty if they collide with other agents. Thus, agents have to learn to cover all the landmarks while avoiding collisions. Both games end when exceeding 100 steps. Both domains contain the sample inconsistency problem since each agent’s local observation contains the relative distance
tion, each agent is assigned a different task, i.e., approaching a different landmark from others, which means each agent may receive different rewards under the same observation.
Figure 4 (a) shows the average rewards on MPE under PPO. We can see that MAPTF achieves higher average rewards than vanilla PPO and DVM. A similar phenomenon can be found in Figure 4 (b), and the superior advantage of MAPTF is enlarged with the increase in the number of agents. This is because MAPTF successfully solves the sample inconsistency using SRO, and thus efficiently distinguishes which part of the information is useful and provides positive transfer for each agent.
Furthermore, it uses the individual termination probability to determine when to terminate the transfer process, which is more flexible, thus facilitating more efficient and effective knowledge transfer among agents.
(c) Cooperative navigation (N = 6)
Figure 4 (c) and (d) shows the average rewards on cooperative navigation game with six (ten) agents. In this game, agents are required to cover all landmarks while avoiding collisions. We can see that MAPTF performs best among all methods, which means it causes fewer collisions and keeps a shorter average distance from landmarks than other methods. The advantage of MAPTF is due to its effectiveness in identifying useful information from other agents’ policies. Therefore, each agent exploits useful knowledge of other agents and, as a result, thus leads to the least collisions and the minimum distance from landmarks.
Finally, we present the performance of MAPTF combined with MADDPG and QMIX on MPE tasks shown in Figure 5 and Figure 6. To further validate the advantage of SRO, we also provide the results of MAPTF with traditional option learning (denoted as MAPTF w/o SRO in Figure 5). MAPTF w/o SRO contains the option module which learns the option value function following single-agent option learning [2, 36]. We can observe that MAPTF performs best among all methods. Although MAPTF with traditional option learning performs better than MADDPG and QMIX learning from scratch, it cannot handle the sample inconsistency problem, thus achieving lower performance than the full MAPTF.
4.3 Ablation Study
The influence of SRO. In this section, we first provide an ablation study to investigate whether SRO selects a suitable policy for each agent, thus efficiently enabling agents to exploit useful information from others. Figure 7 presents the action movement in the environment. Each arrow is the direction of movement caused by the specific action at each location. Four figures show the direction of movement caused by the action selected from the policy of an agent at t1 = 6 × 105 steps (Figure 7(a), top left), and at t2 = 2× 106 (Figure 7(c), bottom left); the direction of movement caused by the action selected from the intra-option policies of SRO at t1 = 6× 105 steps (Figure 7(b), top right), and at t2 = 2 × 106 steps (Figure 7(d), bottom right) respectively. The preferred direction of movement
should be towards the blue circle. We can see that actions selected by the intra-option policies of SRO are more accurate than those selected from the agent’s own policy, namely, more prone to pursue the adversary (blue). This shows that the policy selected by SRO performs better than the agent itself, which means SRO successfully distinguishes useful knowledge from other agents. Therefore, the agent can learn faster and better after exploiting knowledge from this selected policy by SRO than learning from scratch.
The influence of parameter sharing (PS). Finally, we investigate the influence of PS, a common trick in multiagent learning, to validate that the superior performance of MAPTF cannot be achieved by PS only. All above MAPTF-PPO experiments do not incorporate PS (both MAPTF-MADDPG and MAPTF-QMIX use PS since we reuse the source code of previous work[25, 28].) Results of the influence of PS on the performance of MAPTF are shown in Figure 8. At the beginning of the training, comparing MAPTF w/ and w/o PS (PPO w/ and w/o PS), PS shows some acceleration since it requires a smaller number of parameters to be updated. However, it does not provide advantages as the training continues. We can see that both MAPTF w/ and w/o PS outperform PPO w/ PS, which validates that the advantage of MAPTF is significant, and its superior performance cannot be achieved by the parameter sharing technique only.

heading: 5 Related Work
text: Tranfer Learning in MultiAgent Reinforcement Learning (MARL) [6, 17, 4, 16, 7, 8] has been studied with two primary directions: one direction of works focuses on transferring knowledge across multiagent tasks to accelerate the learning process. For example, a number of works explicitly compute the similarities between states or temporal abstractions [18, 3, 10] to transfer across multiagent tasks. The other direction of works transfers knowledge among multiple agents in the same task, which is still investigated at an initial stage. For example, Omidshafiei et al. [27] proposed LeCTR to learn to teach in a multiagent environment. Later, they extended peer-to-peer teaching to a hierarchical structure [19] to improve the teacher credit assignment when faced with long-horizons and delayed rewards problems. However, both LeCTR and HMAT only consider a two-agent scenario.
There are also some works considering to share information among more than two agents, but they suffer from either of the following drawbacks. For example, Liang and Li [22] proposed an attentional teacher-student method under the teacher-student framework where each agent asks for advice from other agents through learning an attentional teacher selector. However, it merely uses the difference of two modes’ (self-learning mode and teaching mode) value functions as the reward to train the student policy, which can be very unstable since the value functions are not bounded. Wadhwania et al. [32] proposed a multiagent policy distillation framework to distill a policy from all agents’ policies and replace each agent’s policy every k steps with it. Similarly, Xue et al. [35] proposed a new algorithm called LTCR to share information among agents through model distillation. However, both DVM and LTCR simply decompose the training process into two stages (i.e., the learning phase and the transfer phase) by turns, and treat all agents without distinction, which may cause negative transfer.
The option framework was firstly proposed in [30] as a kind of temporal abstraction which is modeled as Semi-MDPs. A number of works focused on option discovery in single-agent RL
[2, 20, 14, 15]. An important example is the option-critic [2] which learns multiple source policies in the form of options from scratch, end-to-end. However, the option-critic tends to collapse to single-action primitives in later training stages. Later, several works are proposed to overcome this problem[14, 15]. There are also some works following this direction and use options in MARL settings, such as Macro-Action-based MARL[34], dynamic termination options [12] and DOC [5]. These works learn to solve Dec-PoMDP problems using options. The objective of all these works and MAPTF are orthogonal, that MAPTF transfers knowledge among agents and the rest of works learn the policy from scratch, which is not the focus of this work.

heading: 6 Discussion
text: We developed a general Multiagent Policy Transfer Framework (MAPTF) for multiagent learning. As noted, the knowledge transfer among agents is achieved through policy imitation. This complementary objective needs to calculate the distance between two agents’ policies, as well as the weighting factor f(t) = 0.5 + tanh(3− µt)/2. µ controls the decreasing degree of the weight is important for our work because an unfavorable value of µ may cause negative transfer. Currently, we empirically set the value of µ for different multiagent tasks (detailed in the appendix). How to automatically adjust this parameter to relax the restriction of our work leaves for future work.
Perhaps the biggest remaining limitation of our work is that MAPTF does not make a big contribution to multiagent coordination which is an important problem in MASs. MAPTF learns which agent’s policy is useful for each agent from the perspective of each agent’s local view, other than the global view. which may be stuck in local optimal in some cases. An effective way to achieve coordination among agents is credit assignment. So a natural extension of MAPTF is to design the option module in a standard centralized training, decentralized execution manner, i.e., learning the joint optionvalue function and then decomposing it into individual ones and updating each individual option’s termination function separately. As a result, decisions are made with respect to both local and global perspectives. The precise description and formulation of this extension, as well as the training and testing, are left for future work.

heading: 7 Conclusion and Future Work
text: In this paper, we propose a novel Multiagent Policy Transfer Framework (MAPTF) for efficient MARL by taking advantage of knowledge transfer among agents. MAPTF models the knowledge transfer among agents as the option learning problem to determine which agent’s policy is useful for each agent, and when to terminate it. Furthermore, to resolve the sample inconsistency problem, we propose the Successor Representation Option learning, which decouples the environment dynamics from rewards to learn the option-value function under each agent’s preference. MAPTF can be easily combined with existing DRL and MARL approaches to significantly boost their performance, as shown by the experimental results. How to achieve multiagent coordination in more complex multiagent problems leaves our future work.

heading: Acknowledgements
text: The work is supported by the National Natural Science Foundation of China (Grant No.: U1836214) and the new Generation of Artificial Intelligence Science and Technology Major Project of Tianjin under grant: 19ZXZNGX00010.

Review summaries:
The authors propose a Multiagent Policy Transfer Framework (MAPTF) that learns which agent policy is appropriate for reuse and when to terminate it. Consequently, MAPTF is seen as an option learning approach to policy transfer; it uses the Successor Representation Option (SRO) algorithm proposed by the authors to learn the option-value function under each agent's preference.  The originality of the proposal consists of extending the problem of option learning, integrating it with the problem of policy transfer and reuse. Thus, the idea is to help the learning of a particular agent in the MARL context with the execution of other agents' options. MAPTF learns, for each agent, the option-value function and when to terminate the execution of the option. Experiments performed in Pac-Man and in the Multiagent Particle environment (MPE) showed the effectiveness of the proposal.

The paper proposes a multi-agent algorithm which combines options with a successor representation and imitation learning. My understanding of the algorithm is that it initializes n options (one for each agent), and a meta-policy which selects which option should be used by each agent during each “episode” (see algorithm 1). So agents can use the most relevant option, rather than relying solely on their own option. Individual options are learned with PPO. Successor Representations are used for the meta-policy (I believe) by using a shared feature representation but individual preference vectors for each agent. Finally, the options for all agents are forced to be similar to each other through an imitation learning loss, which is scaled down over the course of training. 

The authors propose a method for accelerating learning in collaborative multiagent RL domains. The acceleration is achieved by the sharing of information between the agents controlled by a framework that also involves learning multiple options. The authors show clear performance improvements in two different domains against "regular" learning and another transfer learning-related method. However, my main concern with this paper is that it is very difficult to read and understand even at a high level. Therefore, I am not sure I would be able to reproduce what is reported here.