An Efficient Transfer Learning Framework for Multiagent Reinforcement Learning
Authors: Tianpei Yang
Weixun Wang
Hongyao Tang
Jianye HAO
Zhaopeng Meng
Hangyu Mao
Dong Li
Wulong Liu
Yingfeng Chen
Yujing Hu
Changjie Fan
Chengwei Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: multiagent reinforcement learning
transfer learning
reinforcement learning
option
succesor representation

Abstract: Transfer Learning has shown great potential to enhance single-agent Reinforcement Learning (RL) efficiency. Similarly, Multiagent RL (MARL) can also be accelerated if agents can share knowledge with each other. However, it remains a problem of how an agent should learn from other agents. In this paper, we propose a novel Multiagent Policy Transfer Framework (MAPTF) to improve MARL efficiency. MAPTF learns which agent's policy is the best to reuse for each agent and when to terminate it by modeling multiagent policy transfer as the option learning problem. Furthermore, in practice, the option module can only collect all agent's local experiences for update due to the partial observability of the environment. While in this setting, each agent's experience may be inconsistent with each other, which may cause the inaccuracy and oscillation of the option-value's estimation. Therefore, we propose a novel option learning algorithm, the successor representation option learning to solve it by decoupling the environment dynamics from rewards and learning the option-value under each agent's preference. MAPTF can be easily combined with existing deep RL and MARL approaches, and experimental results show it significantly boosts the performance of existing methods in both discrete and continuous state spaces.

An Efficient Transfer Learning Framework for Multiagent Reinforcement Learning
Tianpei Yang', ; Weixun : Tang'; Jianye Hao Zhaopeng Meng', Hangyu Mao?, Dong Li", Wulong Liu', Chengwei Zhang', Yujing Hu²,
Yingfeng Chen", Changjie Fan*
College of Intelligence and Computing, Tianjin University
booyang..xwang.iiee www.ann.bueccontr..iiaaa. hiane..a..aaaa..a.ooo.o hao..eggzpp @tju.edu.cn
Noah's Ark Lab, Huawei, ( Imaohangyul I, ,lidong liwwllngg @ huaweii.com
DDlian Maritime University, chenvy @ cttmu.edd.co
NetFase Fuxi Al Lab, (huyujing, chenyingfengl, fanchangjie ) @corp.netease.com
Abstract
Transfer Learning has shown great potential to enhance single-agent Reinforcement Learning (RL.) efficiency. Similarly, Multiagent RL (MARL) can also be acceler- ated if agents can share knowledge with each other. However, it remains a problem of how agent should learn from other agents. In this paper, we propose novel Multiagent Policy Transfer Framework (MAPTF) to improve MARL. efficiency. MAPTF learns which agent's policy the best to reuse for each agent and when terminate it by modeling multiagent policy transfer as the option learning problem. Furthermore, in practice, the option module can only collect all agent's local expe- riences for update due to the partial observability of the environment. While in this setting, each agent's experience may be inconsistent with each other, which may cause the inaccuracy and oscillation of the option-valuee 's estimation. Therefore, we propose a novel option learning algorithm, the successor representation option learning to solve it by decoupling the environment dynamics from rewards and learning the option-value under each agent's 's preference. MAPTF can be easily combined with existing deep RL and MARL approaches, and experimental results show significantly boosts the performance of existing methods in both discrete and continuous state spaces.
Introduction
Transfer Learning has achieved expressive success of accelerating single-agent Reinforcement Learning (RL) via leveraging prior knowledge from past learned policies of relevant tasks 37 36 Inspired by this, transfer learning Multiagent Reinforcement Learning (MARL) 6 is also studied with two major directions: 1) transferring knowledge across different but similar tasks and 2) transferring knowledge among multiple agents in the same task. The former usually explicitly computes similarities between tasks [18 3 [0] to transfer across similar tasks with the
same number of agents, or design network structures to transfer across tasks with different numbers of agents 1 3. In this paper, we focus on the latter due to the following intuition: in Multiagent System (MAS), each agent's experience is different, so the states each agent visits (the familiarities to different regions of the environment) are also different: if we figure out some principled ways to transfer knowledge across different agents, all agents could form big picture even without exploring the whole space of the environment, which will facilitate more efficient MARL.
Equul contribution. Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In fact, the latter direction is still investigated at an initial stage, and the assumptions and designs of some recent methods are usually simple. For example, LeCTR (27] and HMAT [9] adopt the teacher-student framework to enable each agent to learn when to advise other agents or receive advice from other agents. However, they only consider a two-agent scenario. Later, PAT [2] extends this idea to scenarios with more than two agents, and enables each agent to learn from other agents through an attentional teacher selector. However, it simply uses the difference of two unbounded value functions as the student reward which may cause instability.
DVM [32 and LTCR [35 are two methods to transfer knowledge among multiple agents through policy distillation. However, they simply decompose the training process into two stages (i.e., the learning phase and the transfer phase) by turns, which is coarse- -ranneeg manner. Moreover, they consider the equal significance of knowledge transfer throughout the whole training process, which counter-intuitive. A good transfer should be adaptive rather than being equally treated, e.g., the transfer should be more frequent at the beginning of the training since agents are less know ledgeedee about the environment, while decay as the training process continues because agents are familiar with the environment gradually and should focus more on their own experiences.
In this paper, we propose a novel Multiagent Policy Transfer Framework (MAPTF) which models the policy transfer among multiple agents as the option learning problem. In contrast to previous teacher- student and policy distillation frameworks, MAPTF adaptive and applicable to scenarios consisting of more than two agents. Specifically, MAPTF adaptively selects a suitable policy for each agent to exploit, which imitated by an agent as a complementary optimization objective. MAPTF also uses the termination probability as a performance indicator to determine whether the exploitation should be terminated to avoid negative transfer. Furthermore, due to partial observability of the environment, the update of the option-value function is based on all agent's local experience. However, in this setting, each agent's experience may be inconsistent, which could cause the option-value estimation to oscillate and become inaccurate. A novel option learning algorithm, the Successor Representation Option (SRO) learning is used to overcome this inconsistency by decoupling environment dynamics from rewards to learn the option-value function under each agent's preference. MAPTF can be easily incorporated into existing Deep RL and MARL approaches. Our simulations show it significantly boosts the performance of existing approaches both in discrete and continuous state spaces.
2 Preliminaries
Partially Observable Stochastic Games. Stochastic Games [24 are natural multiagent extension of Markov Decision Processes (MDPs), which model the dynamic interactions among multiple agents. Considering the fact agents may not have access to the complete environmental information, we follow previous work's 's settings and model the multiagent learning problems as partially observable stochastic games [3]. A Partially Observable Stochastic Game (POSG) is defined
tuple (N, S. A", T.R...... ........R""" OO.................... OO"), where A is the set of agents; S is the set of states; A' is the set of actions available to agent (the joint action space A = A x A² x x A"); T is the transition function that defines transition probabilities between global states: S x A x S + 10, 1]; R' is the reward function for agent i: S A IR and O' is the set of observations for agent i. A policy T²: O' x A' + [0, 1] specifies the probability distribution over the action space of agent i. The goal of agent i is to learn a policy x' that maximizes the expected return with a discount factor y: J J=H E.
[Eroo]] [r:]]
The Options Framework. Sutton et al. 30
firstly formalized the idea of temporally extended action as an option. An option w E Q is defined as a triple {I", T ,,83 in which I c S is an initiation state set, n" is an intra-option policy and 8= I + [0. 1] is termination function that specifies the probability an option w terminates at state s E I An MDP endowed with a set of options becomes Semi-MDP, which has corresponding optimal option-value function over options learned using intra-option learning. The options framework considers the call-and-rette unndrrturn option execution model, in which an agent picks option w according to its option-value function (u. w). and follows the intra-option policy x" until termination, then selects a next option and repeats the procedure.
Deep Successor Representation (DSR). The successor representation (SR) 9 is basic scheme that describes the state value function by a prediction about the future occurrence of all states under a fixed policy. SR decouples the dynamics of the environment from the rewards. Given transition

---

(s, s', SR is defined as the expected discounted future state occupancy:
sc M(s, .a) E Σ 711(s,
= =
where [[]] is an indicator function with value of one when the argument is true and zero otherwise. Given the SR. the Q-value for selecting action at state be formulated as the inner product of the SR and the immediate reward: (=(ssa) Eees (s, a)((s').
DSR 21 extends SR by approximating it using neural networks. Specifically, each state is represented by feature vector ds, which is the output of the network parameterized by A. Given Φx. SR is represented as mar(ós, a|r) parameterized by 7, decoder (0)) parameterized by 0 outputs the input reconstruction 8, and the immediate reward at state is approximated as a linear function of Φ.: R(s) R Φs where E RD is the weight vector. In this way, the Q-value function can be approximated by putting these two parts together as: Q= a) (ás, alT) w. The stochastic gradient descent is used to update parameters (0, 0). Specifically, the loss function of is:
L E (Ø. ymm (dor, a (T') m.r(0. ()).)))))) alr))²]
where a' = argmax, or (6',, a) w, and m'r is the target SR network parameterized by r' which follows DQN (26) for stable training. The reward weight w is updated by minimizing the loss function: L(w, 0) (R(s) o. w)². The parameter 0 is updated using an L2 loss: L(8,0)
s)²
Thus, the loss function of DSR is the composition of the three loss functions: L r. w, L(T, 0) + [[w, + O).
3 Multiagent Policy Transfer Framework (MAPTF)
3.1 Framework Overview
Option Module
Transfer Less
Lr ((t)Dis(r" xx)
Agent
Crilie Lowe
of af pft D
Termination Network
P..
Replay Butfer
Agene
Agent
Agent"
Critic
oe
Actor
Option Network
pl
[
wlld
Actor
Environment
Actional of
Agent policy
Figure 1: Framework overview.
In this section, we describe our MAPTF in detail. Figure I iillutrates the MAPTF, which contains two modules, the agent's 's module with n agents interacting with an environment, and the option module which determines which agent's policy is useful for each agent. The option module first initializes the option set with n options: Ω fw! w ........ ,w,), each w is a tuple {Tw.. Twa f and T equals to agent i's policy n'. At the start of each episode, for each agent, the option module selects
an option w based on the option-value function and the termination function. Each option terminates according to its termination function and then another option is selected to repeat the process. During the training phase, the option module uses experiences from all agents to update the option-valus function and corresponding termination function. Each agent will exploit the knowledge from another agent nh based on the selected option w. This achieved through policy imitation, which serves as a complementary optimization objective (the option module is responsible for this and each agent does not know which policy it imitates and how the extra loss function calculated). The exploitation is terminated as the selected option terminates, and then another option is selected repeat the process. In this way, each agent efficiently exploits useful information from other agents, and as a result, the learning process of the whole system is accelerated and improved.
In the following section, all agents share the same option set based on the assumption that all agents are homogencous and each agent's 's policy may be helpful for other agents. MAPTF would be easily
3

---

established in the situation where the option module initializes different option sets for each agent, e.g.. each agent only needs imitate small number of agents. In this case, instead of inputting states into the option-value network and outputting a fixed number of option-values, we input each state-option pair the network and output a single option-value 23
3.2 MAPTF
Algorithm MAPTF-PPO
Input: option set - (wr w² wn) replay buffer Di, parameters of actor network p and critic network v² for each agent
1: for each episode do
2: Select an option w for each agent
3: Select action af T²' (ot) for each agent i
4: Perform ã, observe F and new state s'
5: Store transition (o', a', r²,0" ,w, D
6: Select w' if w terminates for each agent i
7: for each agent i do
8: Optimize the critic loss w.r.t of (Equation
9: MAPTF calculates the transfer loss L
10: Optimize the actor loss w.r.t p¹ (Equation
end for
12: Update the option module (Algorithm
end for
In this section, we describe how MAPTF is applied in PPO 22 popular single-agent RL algo- rithm. The way MAPTF combines with other RL and MARL algorithms is similar. The whole process of MAPTF combined with PPO shown in Algorithm | With the input of options Ω = [w=,_ _ ..... w"), }, for each episode, the option module selects an option w for each agent (Line 2), and each agent selects an action a' following its policy x' (Line 3). The joint action is performed, then the reward and new state s' is returned from the environment (Line 4). The transition is stored in each agent's 's replay buffer DD (Line 5). If ws terminates, then the option module selects another option for each agent (Line 6).
For each update step, each agent updates its critic network by minimizing the loss L (Line 8);
Σ
Vv: (o))))
r't
where T is the length of the trajectory segment in PPO. Then each agent updates its actor network by minimizing the summation of the original loss and the transfer loss L (Line 10):
(aalo?) of
E Σ
A [KL(T Iir.
(4)
where 1' -
-'=Erse Erstr' - Vor (of) is the advantage function of agent v.
To transfer useful knowledge among agents, MAPTF calculates the distance Dis(**Ir*) between each exploited policy x" and each agent's 's policy T', and transfers the loss each agent respectively, serving a complementary optimization objective for each agent. This means that apart from maximizing the cumulate reward, each agent also imitates another agent's policy T² by minimizing the loss function Li, as follows:
Di |(()Dis(**(=*'
where, f(t) = 0.5 + tanh(3 - /2t is the discounting factor. is hyper- hyper- parameter that controls the decreasing degree of the weight. This means that at the beginning of learning, each agent exploits knowledge from other agents mostly. As learning continues, knowledge from other agents becomes less useful and each agent focuses more on the current self-learned policy. We consider the MAPTF is general framework that can be combined with any existing Deep RL and MARL algorithms.

---

For policy-based algorithms, the corresponding term is the cross-entropy loss: H(x="IT* (and other choices of the distance metric are also suitable). For value-based algorithms. MAPTF measures the distance of two Q-value distributions.
The next issue is how to update the option module. To evaluate which agent's policy is useful for each agent, the option module needs to collect all agent's ' experiences for the update. What if the experience from one agent is inconsistent with others? In a POSG, each agent can only obtain the local observation and individual reward signal, which may be different for different agents even the same state, e.g., each agent has an individual goal achieve or has different roles, and the rewards assigned to each agent are different. If we use inconsistent experiences to update the same option-vallen value and termination probability, the estimation of the option- -aalue function would oscillate and become inaccurate. To this end, We propose a novel option learning algorithm, the Successor Representation Option (SRO) to address this problem, which is described in the next section.
3.3 SRO Learning
MАPTF applies a novel option learning algorithm, Suc- cessor Representation Option (SRO) learning to learn the option-value function under each agent's preference. The SRO network architecture is shown in Figure 2 with each observation o' from each agent i as input. o' cor- responding to the global state is inputted through fully-comnected layers to generate the state embedding Oa. which is transmitted to three network modules.. The first mobdmoduee contains the state reconstruction model which ensures o,- well representing o', and the weights for the immediate reward approximation at local observa- tion o'. The immediate reward is approximated as a linear function of Φet: R*((,)) de w, where E IR. is the weight vector. The second sub-module is used to approximate SR for options msr (Mo', w/) which describes the expected discounted future state occupancy of executing the option w. The last sub-module is used to update the termination probability (()=",
Input Reconstruction
Reward Welghts
Option-value function
(at) mi (o') (ofy
Qalo' moy) Qu(o',asz)
H
(at,
-P(o and)
H
Bio' ww))
Option termination probabilities
Lecol ann)
Figure The SRO architecture.
Given mar (for, w|)), the SRO-value function can be approximated as: (wor w) R. (doi w|T)- w. Since options are temporal abstractions [30], SRO also needs calculate the U function which served as SRO upon arrival, indicating the expected discounted future state occupancy of executing
an option w upon entering a local observation o²::
Ü ) w(7') (1 B(b ( w(('t) ((O , wl m (0 (0 w w'1r'),
(6)
where w/ = argmax m (er w|T')
The learning process of SRO shown in Algorithm 2 first initialized the network parameters for the SRO network and the target network. Then, for each update step, it samples a batch of B/N transitions from each agent's 's buffer Dí, which means there are B transitions in total (Line 2). SRO loss is composed of three components: the state reconstruction loss L(θ, 0), the loss for reward weights L(w, 0) and SR loss (T,,)). The state reconstruction network is updated by minimizing two losses L(O,0) 0) and L(w, 0) (Lines 3,4):
L(0,0) (g0( - o²)²
L(w, 0) (r' Qo. w)²
The second sub-module, SR network approximates SRO and is updated by minimizing the standard 1.2 loss L(r,0) (Lines 5-8):
1 - E B
L(T,0)
mn (vn-mar w()))²
(8)
where y/- = oor + ( w/T.. At last, the termination probability of the selected option updated. According to the call-and-returm option execution model, the termination probability B controls
5

---

Algorithm 2 SRO Learning.
Input: option set Ω (w",w,... w"), parameters of state feature 0, reward weights w, state
reconstruction 0, termination network w, SR network т, SR target network r'; replay buffer De for each agent i;
1: for each update step do
2: Select B/N samples (o', a², r
w. from each D
3: Optimize L(O, 0) w.r.t 0, 0 (Equation 7
4: Optimize L(w, 0) W.n.t w, (Equation
5: for each do
6: nh selects action at at observation Ø² then
7:
( Calculate ( ger, w((') (Equation 6
8:
Optimize [T, 0) w.r.t v (Fquation
9: Optimize the termination network w.r. wo (Equation 9
10: end if
11: end for
12: Copy to 7' every is steps
13:
end for
when terminate the selected option and then select another option accordingly, which is updated w.r. xo as follows (Line 9):
gg
(A(o,r +
(9)
dw
where A(o, A(o,", w/TT) is the advantage function and approximated as Mer (Mrr"" w') w
max mar @ " w/T w, and regularization term ensure explorations [ 36). At
last, the target network parameterized by 7' copies from 7 every steps (Line 12).
4 Experimental Results
We evaluate the performance of MAPTF combined with the popular single-agent RL algorithm (PPO [29] and MARL algorithm (MADDPG (25 and QMIX (8
representative multiagent games, Pac-Man (1 and multiagent particle enviromment (MPE) (25 (illustrated in the appendix). Specifically, first combine MAPTF with PPO Pac-Man to validate whether MAPTF successfully solves the sample inconsistency due to the partial observation. Then, we combine MAPTF with three baselines (PPO, MADDPG and QMIX) on MPE further validate whether MAPTF more flexible way for knowledge transfer among agents. We also compare with DVM B22, which is a recent multiagent transfer method. All results are averaged over 10 seeds. More experimental details and parameters settings are detailed in the appendix, source code is provided on ht //github. wwwpe////aaaaaaao////aoaooa watp/M/TT_acooMooo/
4.1 Pac-Man
HAPT DnH
Aoergg reward
Aowrage reward
-2
0.0
0.4
0.8
1.2
1.6
2.0
0.0
0.4
0.8
1.6
2.0
0.4
0.8
1.6
2.0
(a) OpenClassic
(b) MediunClassic
(c) OriginalClassic
Figure The performance on Pac-Man under PPO.
Pac-Man 31 is a mixed cooperative-competiive maze game with one pac-man player and several ghost players. We consider three Pac-Man scenarios (OpenClassic. MediumClassie,, and Original-

---

Classic) with the game difficulties increasing. The goal of the pac-man player is to eat as many pills as possible and avoid the pursuit of ghost players. For ghost players, they aim to capture the pac-man player as soon possible. In our settings, MAPTF controls several ghost players to catch pac-man player controlled by well pre-trained PPO policy. The game ends when one ghost catches the pac-man player, or the episode exceeds 100 steps. Each ghost player receives -0.01 penalty for each step and a +5 reward for catching the pac-man player.
Figure (a) presents the average rewards on the OpenClassic scenario. We can see that MAPTF performs better than other methods and achieves the average discount rewards of +3 approximately with a smaller variance. In contrast, PPO and DVM only achieve the average discount rewards of +2.5 approximately with larger variance. This phenomenon indicates that MAPTF enables efficient knowledge transfer between ghost players, thus facilitating better performance.
Next, we consider complex scenario with larger layout size than the former, and it contains obstacles (walls). F (b) shows the advantage of MAPTF is much more apparent compared with PPO and DVM. Furthermore, MAPTF performs best among all methods, which means effectively recognizes more useful information for each agent. MAPTF performs better than DVM because MAPTF enables each agent to effectively exploit useful information from other agents, which successfully avoids negative transfer when other agents' policies are only partially useful. However, DVM just transfers all information from other agents through policy distillation without distinction.
Finally, we consider a scenario with the largest layout size, and four ghost players catching one pac-man player. Similar results can be observed in Figure 3 (c). By comparing the results of the three scenarios, we see that the superior advantage of MAPTF increases when faced with more challenging scenarios. Intuitively, the environmental difficulties increase, agents explore the environment and learn the optimal policy. In such a case, agents need to exploit other agents' knowledge more efficiently, which would significantly accelerate the learning process, as demonstrated by MAPTE.
harder to
4.2 MPE
MPE [25] is a multiagent particle world with continuous observation and discrete action space. We consider two scenarios of MPE: predator-prey and cooperative igation. The predator-pree contains three (nine) agents which are slower and want to catch one (three) adversary (rewarded +10 by each hit). The adversary is faster and wants to avoid being hit by the other three (nine) agents. Obstacles block the way. The cooperative navigation contains six (ten) agents to cover six (ten) corre- sponding landmarks. Agents are penalized with 1 penalty if they collide with other agents. Thus, agents have to learn to cover all the landmarks while avoiding collisions. Both games end when exceeding 100 steps. Both domains contain the sample incon- sistency problem since each agent's 's local observation contains the relative distance between other agents, obstacles, and land- marks. Moreover, in cooperative naviga- tion, each agent is assigned diffferent task, i.e., approaching different landmark from others, which means each agent may receive different rewards under the same observation.
Trakiig epiantes
30
Toisin opirades
10
Predator-prey (N- 4)
(b) Predator-pre preyoor-preyycoo (N- 12)
-250
-200
60
0.5
(c) Cooperative navigation
(d) Cooperative navigation
(N = 6)
(N
10)
Figure 4: The performance on MPE under PPO.
Figure 4 (a) shows the average rewards on MPE under PPO. We can see that MAPTF achieves higher average rewards than vanilla PPO and DVM. A similar phenomenon can be found in Figure and the superior advantage of MAPTF is enlarged with the increase in the number of agents. This because MAPTF successfully solves the sample inconsistency using SRO, and thus efficiently distinguishes which part of the information is useful and provides positive transfer for each agent.
7

---

Furthermore, it uses the individual termination probability to determine when to terminate the transfer process, which is more flexible. thus facilitating more efficient and effective knowledge transfer among agents.
Figure 4 (c) and (d) shows the average wards on cooperative navigation game with six (ten) agents. In this game, agents are re- quired to cover all landmarks while avoid- ing collisions. We can see that MAPTF performs best among all methods, which means causes fewer collisions and keeps shorter average distance from landmarks than other methods. The advantage of MAPTF is due to its effectiveness in identi- fying useful information from other agents' policies. Therefore, each agent exploits useful knowledge of other agents and, as a result, thus leads to the least collisions and the minimum distance from landmarks.
400
Traning opriodiss
Thissn
(a) Predator-prey (N = 4)
(b) Predator-pre (N 12)
-300
1100
100
HULENI HAFTT
Finally, we present the performance of MAPTF combined with MADDPG and QMIX on MPE tasks shown in Figure 5 To further validate the advan- tage of SRO, we also provide the results of MAPTF with traditional option learning (denoted as MAPTF w/o SRO in Figure 5j. MAPTF w/o SRO contains the option module which learns the option value func- tion following single-agent option learning 2 36). We observe that MAPTF per- forms best among all methods. Although МАPTF with traditional option learning performs better than MADDPG and QMIX learning from scratch, cannot handle the sample inconsistency problem, thus achiev- ing lower performance than the full MAPTT.
2-6
000
02
Trotding
Cooperative navigation
(d) Cooperative navigation
(N
10)
and Figure 6
Figure 5: The performance MPE under MADDPG.
MAUTEE-ONNI
600
Vanila GOEX
120
2000
20
0.2
002
2.
1.
100
(a) Predator-prey (N 4)
(b) Predator-prey (N 12)
Figure 6: The performance on MPE under QMIX.
4.3 Ablation Study
(a) PPO..
SRO,;
(c) PPO
(d) SRO
Figure 7: Movements following agent I's policy and SRO's policy at different timesteps.
The inffuence of SRO. In this section, we first provide an ablation study to investigate whether SRO selects suitable policy for each agent, thus efficiently enabling agents to exploit useful information from others. Figuse(Zsresents the action movement in the environment. Each arrow the direction of movement caused by the specific action at each location. Four figures show the direction of movement caused by the action selected from the policy of an agent at left), and at l2 2 10 (rigure bottom left); the direction of movement caused by the action selected from the intra-option policies of SRO at 4 = 6 10 steps (Figure 7b), top right), and at = 2 x 10t steps (Figure d), bottom right) respectively. The preferred direction of movement
10a steps (Figure a), top

---

should be towards the blue circle. We can see that actions selected by the intra- intra-option policies of SRO are more accurate than those selected from the agent's own policy, namely, more prone to pursue the adversary (blue). This shows that the policy selected by SRO performs better than the agent itself, which means SRO successfully distinguishes useful knowledge from other agents. Therefore, the agent can learn faster and better after exploiting knowledge from this selected policy by SRO than learning from scratch.
The influence of parameter sharing (PS). Finally, we investigate the influence of PS, a common trick in multiagent learning. to validate that the superior performance of MAPTF cannot be achieved by PS only. All above MAPTF-PPO experiments do not incorporate PS (both MAPTF-MADDPG and MAPTF- FEFE-QIIX use PS since we reuse the source code of previous work (25 (8).) Results of the influence of PS on the performance of MAPTF are shown in Figure g At the beginning of the training, comparing MAPTF w/ and w/o PS (PPO w/ and w/o PS). PS shows some acceleration since it requires smaller number of parameters to be updated. However, it does not provide advantages as the training continues. We can see that both MAPTF w/ and w/o PS outperform PPO w/ PS, which validates that the advantage of MAPTF is significant, and its superior performance cannot be achieved by the parameter sharing technique only.
300
80D
Average reward
1o0mor wwwarr
600
200
MAPTY
MAPIF w/ PS
100
PPO w/ PS
200
PPD w/ PS
MAPTF PS
PPO wio PS
PPO weo PS
2.0
0.0
1.2 Training epiisuee
2.0
Teeining episodes
(a) Predator-prey (N
(b) Predator-prey (N 12)
Figure& 8: The influence of PS on the performance of MAPTF in predator-prey.
5
Related Work
7 8 has been studied with two primary directions: one direction of works focuses on transferring knowledge across multiagent tasks accelerate the learning process. For example, number of works explicitly com- pute the similarities between states or temporal abstractions [18 10 to transfer across multiagent tasks. The other direction of works transfers knowledge among multiple agents in the same task, which is still investigated at an initial stage. For example, Omidshafici et al. [27) proposed LcCTR to learn to teach in multiagent environment. Later, they extended veer-to-peer teaching a hierarchical structure 199 to improve the teacher credit assignment when faced with long- horizons and delayed rewards problems. However, both LeCTR and HMAT only consider two-agent scenario.
Tranfer Learning in MultiAgent Reinforcement Learning (MARL) 6 17
16
There are also some works considering to share information among more than two agents, but they suffer from either of the following drawbacks. For example, Liang and Li 22 proposed an attentional teacher-student method under the teacher-student framework where each agent asks for advice from other agents through learning an attentional teacher selector. However, it merely uses the difference of two modes' (self-learning mode and teaching mode) value functions as the reward to train the student policy, which can be very unstable since the value functions are not bounded. Wadhwania et al. [32] proposed multiagent policy distillation framework to distill a policy from all agents' policies and replace each agent's 's policy every & steps with it. Similarly, Xue et al. (35) proposed new algorithm called LTCR to share information among agents through model distillation. However, both DVM and LTCR simply decompose the training process into two stages (i.e., the learning phase and the transfer phase) by turns, and treat all agents without distinction, which may cause negative transfer.
The option framework was firstly proposed in 30 as a kind of temporal abstraction which modeled as Semi-MDPs. A number of works focused on option discovery in single-agent RL

---

An important example is the option-critic [2] which learns multiple source policies the form of options from scratch, end-to-end. However, the option-critic tends to collapse to single-action primitives in later training stages. Later, several works are proposed to overcome this problem 14 [5]. There are also some works following this direction and use options in MARL settings, such as Macro-Action- based MARL (4), dynamic termination options [2] and DOC [5]. These works learn to solve Dec-PoMDP problems using options. The objective of all these works and MAPTF are orthogonal, that MAPTF transfers knowledge among agents and the rest of works learn the policy from scratch, which is not the focus this work.
20 14 15
Discussion
We developed general Multiagent Policy Transfer Framework (MAPTF) for multiagent learning. As noted, the knowledge transfer among agents is achieved through policy imitation. This complementary objective needs to calculate the distance between two agents' policies, as well as the weighting factor f(1) 0.5 ta /t) J controls the decreasing degree of the weight is important for our work because an unfavorable value of l may cause negative transfer. Currently, we empirically set the value of ju for different multiagent tasks (detailed in the appendix). How to automatically adjust this parameter to relax the restriction of our work leaves for future work.
Perhaps the biggest remaining limitation of work is that MAPTF does not make big contribution to multiagent coordination which is an important problem in MASS. MAPTF learns which agent's policy is useful for each agent from the perspective of each agent's local view, other than the global view. which may be stuck in local optimal in some cases. An effective way to achieve coordination among agents is credit assignment. So a natural extension of MAPTF is to design the option module in a standard centralized training, decentralized execution manner, i.e., learning the joint option- value function and then decomposing it into individual ones and updating each individual option's
termination function separately. As a result, decisions are made with respect to both local and global perspectives. The precise description and formulation of this extension, as well as the training and testing, are left for future work.
7 Conclusion and Future Work
In this paper, we propose a novel Multiagent Policy Transfer Framework (MAPTF) for efficient MARL by taking advantage of knowledge transfer among agents. MAPTF models the knowledge transfer among agents the option learning problem to determine which agent's policy is useful for each agent, and when to terminate it. Furthermore, to resolve the sample inconsistency problem, we propose the Successor Representation Option learning, which decouples the environment dynamics from rewards to learn the option-value function under each agent's ' preference. MAPTF can be easily combined with existing DRL and MARL approaches to significantly boost their performance, as shown by the experimental results. How to achieve multiagent coordination in more complex multiagent problems leaves our future work.
Acknowledgements
The work supported by the National Natural Science Foundation of China (Grant No.: U1836214) and the new Generation of Artificial Intelligence Science and Technology Major Project of Tianjin under grant: 19ZXZNGX0000..
References
[1] Akshat Agarwal, Sumit Kumar, and Katia P. Sycara. Learning transferable cooperative behavior in multi-agent teams. CoRR, abs/1906.0 01202, 2019.
[2] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of AAAL, pages 1726 1726-1734, 1734, 2017.
[3] Georgios Boutsioukis, Ioannis Partalas, and loannis P. Vlahavas. Transfer learning in multi- agent reinforcement learning domains. In Recent Advances Reinforcement Learning 9th European Workshop, pages 249-260, 2011.
10

---

[4] Lucian Bu, Robert Babu, Bart De Schutter, et al. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, 3862):156-172. 2008.
[5] Jhelum Chakravorty, Patrick Nadeem Ward, Julien Roy, Maxime (hevalier-Boiswer Sumana Basu, Andrei Lupu, and Doina Precup. Option-critic in cooperative multi- systems. CoRR, abs/1911.12880 2019.
[6] Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. In Proceedings of AAAI/IAAI, pages 746-752, 1998.
[7] Felipe Leno da Silva and Anna Helena Reali Costa. A survey on transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research, 64:6455703, 2019.
[8] Felipe Leno da Silva, Garrett Warnell, Anna Helena Reali Costa, and Peter Stone. Agents teaching agents: a survey inter-agent transfer learning. Auton. Agents Multi Agent Syst., 34(1):9, 2020.
[9] Peter Dayan. Improving generalization for temporal difference learning: The successor repre- sentation. Neural Computation, 5(4):613-624, 1993.
[10] Sabre Didi and Geoff Nitschke. Multi-agent behavior-based policy transfer. In Proceedings of European Conference on the Applications of Evolutionary Computation, pages 181-197, 2016. 2016
[11] Shixiang Gu, Timothy P. Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q- learning with model-based acceleration. In Proceedings of the 33nd International Conference on Machine Learning, pages 2829-28338 2016.
[12] Dongge Han, Wendelin Boehmer, Michael J. Wooldridge, and Alex Rogers. Multi-agent hierarchical reinforcement learning with dynamic termination. In Proceedings of the 18th International Conference Autonomous Agents and MultiAgent Systems, pages 2006-2008. 2019.
[13] Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In Proceedings of AAAI, volume 4, pages 709-715, 2004.
[14] Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option: Learning options with a deliberation cost. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 3165-3172, 2018.
[15] Anna Harutyunyan. Will Dabney, Diana Borsa, Nicolas Heess, Rémi Munos, and Doina Precup. The termination critic. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, pages 2231-2240, 2019.
[16] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Mulli-Agent Systems, (3(6)-550-777, 2019.
[17] Junling Hu and Michael P. Wellman. Multiagent reinforcement learning: Theoretical framework and an algorithm. Proceedings of ICML. pages 242-250, 1998.
[18] Yujing Hu, Yang Gao, and Bo An. Accelerating multiagent reinforcement learning by equilib- rium transfer. IEEE Trans. Cybernetics, 45(7): 2015.
[19] Dong-Ki Kim, Miao Liu, Shayegan Omidshafiei, Sebastian Lopez-Cot, Matthew Riemer, Golnaz Habibi, Gerald Tesauro, Sami Mourad, Murray Campbell, and Jonathan P. How. Learning hierarchical teaching policies for cooperative agents. In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, pages 620-628, 2020.
[20] Martin Klissarov, Pierre-Lue Bacon, Jean Harb, and Doina Precup. Learnings options end-to-end for continuous action tasks. CoRR, abs/1712.00000 2017.
[21] Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning, arXiv preprint 1iXi:1666600399 2016.
11

---

[22] Yongyuan Liang and Bangwei Li. Parallel knowledge transfer in multi-agent reinforcement learning. CoRR, 00ss2003.130885 2020.
[23]
Timothy P. Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra, Continuous control with deep reinforcement learning. In Proceedings of the 4th International Conference on Learning Representations, 2016.
[24] Michael L.. Littman. Markov games as a framework for multi-agent reinforcement learning. In Proceedings of ICML, pages 157-163, 1994.
[25] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. In Proceedings of NeurIPS, pages 6379-6390, 2017.
[26] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 5 518(754)):529, 2015.
[27] Shayegan Omidshafici, Dong-Ku Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Camphell, and Jonathan P. How. Learning teach in cooperative multiagent reinforcement learning. In Proceedings of AAAI, pages 6128-6136, 01886,63,2220 2019.
[28] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi- agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, pages 4292-4301, 2018.
[29] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.063:0 2017.
[30] Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDP A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181
-21,, 1999.
[31] Tycho der Ouderaa. Deep reinforcement learning in pac-man. 2016.
[32] Samir Wadhwania, Dong-Ki Kim, Shayegan Omidshafiei, and Jonathan How. Policy distilla- tion and value matching in multiagent reinforcement learning. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019, Macau, SAR, China, November 3-8, 8200, 2019.
2019, pages
[33] Weixun Wang. Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few to more: large-scale dynamic multiagent curriculum learning. In Proceedings of AAAI, 2020.
[34] Yuchen Xiao, Joshua Hoffman, and Christopher Amato. Macro iere-aacctioobbeeee deep multi-agent reinforcement learning. In Proceedings of the 3rd Annual Conference on Robot Learning, pages 1146-1161.
[35] Zeyue Xue, Shuang Luo, Chao Wu, Pan Zhou, Kaigui Bian, and Wei Du. Transfer hetero- geneous knowledge among peer-to-peer teammates: A model distillation approach. CoRR, abs/2002.02202, 2020.
[36] Tianpei Yang, Jianye Hao, Zhaopeng Meng, Zongzhang Zhang, Yujing Hu, Yingfeng Chen, Changjie Fan, Weixun Wang, Wulong Liu, Zhaodong Wang, and Jiajie Peng. Efficient deep reinforcement learning via adaptive policy transfer. Proceedings of the Twenty-Ninth Interna- tional Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3094-3100 2020.
[37] Haiyan Yin and Sinno Jialin Pan. Knowledge transfer for deep reinforcement learning with hierarchical experience replay. In Proceedings of AAAI, pages 1640-1646, 2017.
12


Reviews:
[{"review_id": "vDkVA6fLWG", "reviewer": "Reviewer_G6LK", "summary": "The authors propose a method for accelerating learning in collaborative multiagent RL domains. The acceleration is achieved by the sharing of information between the agents controlled by a framework that also involves learning multiple options. The authors show clear performance improvements in two different domains against \"regular\" learning and another transfer learning-related method. However, my main concern with this paper is that it is very difficult to read and understand even at a high level. Therefore, I am not sure I would be able to reproduce what is reported here.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the authors present clear performance improvements in the experimental evaluation, there are multiple points to improve in regard to clarity and the description of the method.\n\nThe first point would be the title, which is overly generic. The authors should make sure that the title reflects what is novel in this paper and the main characteristics of the proposed method. While it is true that it is a transfer learning method, dozens of such methods exist, and this title doesn't help a reader screening for papers.\n\nThe manuscript itself is also very hard to read. Why are the options a critical point in the knowledge-sharing mechanism? If all options are learned from scratch, how I can expect that they should be particularly useful for other agents? ALso, it is not clear at all exactly which information is communicated to the other agents and how often communication is performed.\n\nReading your paper, I feel like it could be useful to choose which option to be applied in each step through an attention layer, similarly as it is done in DECAF:\n\nGlatt, Ruben, et al. \"DECAF: deep Case-based Policy Inference for knowledge transfer in Reinforcement Learning.\" Expert Systems with Applications 156 (2020): 113420.\n\nThe DVM  algorithm is used for comparisons in the experimental evaluation, but you never explained why this method is most similar to yours, and how they differ (in fact, I think not even the acronym is explained). It would also be very relevant to show your comparisons how often the agents communicate. Communication is a critical aspect in any multiagent application, and it would be good to show the trade-off performance improvement x communication used.\n\nI don't see any reason to show the performance for PPO and a multiagent base method for the same domain. One should be fine.\n\nFinally, I am concerned that both of the domains chosen in your evaluation do not require coordination (i.e., the agents can probably solve the task by simply chasing pac-man/prey independently). It would be good to include a domain where coordination is required.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No short-term societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "7EITnA6PHd", "reviewer": "Reviewer_RZaH", "summary": "The paper proposes a multi-agent algorithm which combines options with a successor representation and imitation learning. My understanding of the algorithm is that it initializes n options (one for each agent), and a meta-policy which selects which option should be used by each agent during each “episode” (see algorithm 1). So agents can use the most relevant option, rather than relying solely on their own option. Individual options are learned with PPO. Successor Representations are used for the meta-policy (I believe) by using a shared feature representation but individual preference vectors for each agent. Finally, the options for all agents are forced to be similar to each other through an imitation learning loss, which is scaled down over the course of training. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Clarity:**\n- The paper has significant issues with clarity which make it difficult to assess the proposed algorithm. I had to read the methods section several times to parse what is being done. \n- For example, consider the following: “The option module select an option ω for each agent. During the training phase, the option module uses experience from all agents to update the option-value function and corresponding termination probabilities.” -> this indicates that there is a single option pool being shared among all agents. This would imply that actually each agent is running the identical policy, which is composed of the same set of options. The paper then goes on to say, “Each agent will exploit the knowledge from another agent πω based on the selected option ω, and this is achieved through policy imitation, which serves as a complementary optimization objective”. This is very confusing, because it is not clear how agents actually have their own policies if they are all using the same option pool to act. How can they imitate each others’ policies if they share the same policy? Only after re-reading this section to determine that there are n options, did I realize that the intention is that each option is meant in some sense to correspond to one agent, although this does not appear to be enforced.\n- The second problem is that how the SR is used is not made clear by the paper. A careful reading determines it is used for the option-value function, i.e. the meta-policy. This only became apparent to me after reading that the individual agent policies were learned with PPO, but it took several passes. This should be stated more clearly and earlier in the paper (and repeated).\n- A further suggestion to help clarify the paper would be to state explicitly which parameters are unique to each agent. The options are actually not unique to the agents, because they are shared freely among the agents. My current understand is that the only agent-specific parameters are the preference vectors w. \n- The motivation given in lines 26-30 is compelling. \n- Using \\tau for parameters is not a great use of notation, since \\tau is often using to represent trajectories in RL.\n- The explanation of successor features (lines 87-98) could be made more clear.\n- The transfer loss L_{tr} does not appear to be introduced before it is used in Eq. 4\n- There are typos and grammar mistakes throughout e.g. “a regularization term to ensure explorations” in line 176\n\n**Novelty/Originality:**\n- The novelty of the proposed algorithm is limited. The idea of using options for multi-agent learning has been explored in [33,12,5] as cited in this paper. The idea of using successor representations with a shared feature representation and individual preference vectors for each agent was proposed in https://arxiv.org/abs/2102.12560, which is not cited by this paper. \n- The idea of using imitation learning to make the agents’ policies more similar appears to be novel, but it is not well motivated. Why use options at all in this case, rather than allowing agents to share parameters for more efficient learning? This should be better motivated. It would also be good to include an ablation study assessing the importance of this imitation learning loss, and to benchmark against multi-agent techniques which instead rely on parameter sharing among agents. \n\n**Quality:** \n- MADDPG is not a reasonable multi-agent baseline. It is several years old (from 2017), and many more performant algorithms have been proposed since then (e.g. QMIX, etc, or https://arxiv.org/pdf/2106.02195.pdf for a thorough collection of SOTA baselines). It is difficult to assess the contributions of this paper unless it is properly benchmarked against recent techniques. \n- The paper claims that DVM, published in 2019, is “the most recent multiagent transfer method”. This is not true; see e.g. https://arxiv.org/abs/2102.12560\n- The ablation studies do not assess the contribution of the options framework, or the imitation learning approach. \n- A positive aspect of the paper is that it uses 10 random seeds per experiment, indicating the results of the experimental evaluations are more reliable and more likely to replicate. Good work!\n- The use of the state reconstruction loss to help refine the successor representation is a nice touch.\n\n**Significance:**\n- Overall the paper has promise, but it is difficult to assess the significance of the contributions due to the clarity issues and the lack of appropriate benchmarking against recent techniques. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper provides a nice limitations section that analyzes drawbacks of the method. It does not appear to address societal impact. My suggestion would be to include a discussion of possible beneficial applications of effective multi-agent learning as motivation for why the problem is important.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "2VC1MXz2gC", "reviewer": "Reviewer_q2HS", "summary": "The authors propose a Multiagent Policy Transfer Framework (MAPTF) that learns which agent policy is appropriate for reuse and when to terminate it. Consequently, MAPTF is seen as an option learning approach to policy transfer; it uses the Successor Representation Option (SRO) algorithm proposed by the authors to learn the option-value function under each agent's preference.\n\nThe originality of the proposal consists of extending the problem of option learning, integrating it with the problem of policy transfer and reuse. Thus, the idea is to help the learning of a particular agent in the MARL context with the execution of other agents' options. MAPTF learns, for each agent, the option-value function and when to terminate the execution of the option. Experiments performed in Pac-Man and in the Multiagent Particle environment (MPE) showed the effectiveness of the proposal.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "MAPTF belongs to a class of novel combinations of well-known techniques. MAPTF extends the option framework to a MARL context, allowing the reuse of other agents' options in learning. Likewise, the SRO algorithm is an adaptation of previous algorithms – SRO learns the option-value function under the preference of each agent, as well as the termination probability of each option.\n\nThe text is a little confusing, thus making it difficult to fully understand the proposal. The evolution of the content given in the Preliminaries section is not perfectly extended to the TL-in-MARL scenario. The notation gets quite confusing.\n\nA question that remains is whether the authors, so as to deal with POSG, simply considered the observation of each agent as state information (i.e, assumed that o = s). Another question is how MARL came about: are they simply agents learning their own policy and completely ignoring the other agents, as if they were completely independent learning? If so, how did one deals with the non-stationarity of the MARL environment? In section 6, the authors comment that they did not address the problem of coordination between agents, but ignoring the non-stationarity of a MARL is more consequential than that. Although the authors claim broad generality for their proposal, I am not convinced that it would work for heterogeneous agents.\n\nThe figures could be improved and should be in perfect harmony with both their description in the text and the algorithms presented.\n\nMinor issues:\nAlgorithm 1, step 2 and 6: How is an option selected? Step 4: How is the new state s’ observed? Step 9: is Eq. 9 used here? How is Dis(.,.) calculated?\nAlgorithm 2: Is it run for each agent? Make this clear in step 1 of the algorithm.\n\nGeneral question: at the beginning of learning in a MARL, agents still don't know much and exhibit erratic behavior. However, eq. 9 indicates that it uses the policies of other agents the most at the beginning of learning. As everyone is learning at the same time, would the transfer be helpful?\n\nThe experimental results presented are very promising. However, as there were many doubts about the modeling carried out, it is not possible to be sure that the contribution is in fact very significant. While the basic ideas are promising, development must be improved.\n\nI would like to reinforce that I have read the authors' rebuttal and thank them for all the clarifications and improvements to the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "about 4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewer team agrees that this paper has some clear strengths and weaknesses:
(+) It proposes a new way of combining two known techniques, the option framework and successor representation, for MARL.
(+) The proposed algorithm achieves good performance empirically.
(+) The additional experimental results provided in the author response phase addressed some of the reviewers' concerns, especially the comparison with QMIX and the additional ablation study. These results improve the quality of the paper and the significance of the work.
(-) The applicability of the proposed technique is limited by some level of homogeneity or similarity among the agents. The authors discussed how the technique can be used for heterogeneous agents in the response phase but the benefit of the framework clearly relies on a (sub)set of agents who have similarities in performing the tasks and do not need to carefully coordinate with each other.
(-) While the authors claim that they have made efforts on improving the clarity of the paper since the last submission to an earlier conference, the reviewer team finds that the paper still lacks clarity in a number of places, which makes it hard for the reader to understand the paper. For example, it was not very clear which parameters are shared and which ones are agent-specific. Alg 1 lacks clear explanations in some of the important steps. Some of the related work is mentioned without explaining the differences from this work clearly. The authors' response clarified some of these points but the paper's clarity needs to be improved. In addition, there are typos and grammar mistakes throughout the paper.

Overall, the reviewer team views this paper as a borderline paper mainly due to the lack of clarity. The novel combination of the option framework and successor representation may inspire further research and the significantly improved empirical performance over baselines is promising. In the next version of the paper, I would recommend the authors include the additional experimental results presented in the response phase and improve the clarity of the paper based on reviewers' comments and suggestions.
Decision: Accept (Poster)