Transfer among Agents: An Efficient Multiagent Transfer Learning Framework
Authors: Tianpei Yang
Jianye HAO
Weixun Wang
Hongyao Tang
Zhaopeng Meng
Hangyu Mao
Dong Li
Wulong Liu
Yujing Hu
Yingfeng Chen
Changjie Fan
Venue:  (2021)
Keywords: Multiagent learning
transfer learning
reinforcement learning

Abstract: Transfer Learning has shown great potential to enhance the single-agent Reinforcement Learning (RL) efficiency, by sharing learned policies of previous tasks. Similarly, in multiagent settings, the learning performance can also be promoted if agents can share knowledge between each other. However, it remains an open question of how an agent should learn from other agents' knowledge. In this paper, we propose a novel multiagent option-based policy transfer (MAOPT) framework to improve multiagent learning efficiency. Our framework learns what advice to give to each agent and when to terminate it by modeling multiagent policy transfer as the option learning problem. MAOPT provides different kinds of variants which can be classified into two types in terms of the experience used during training. One type is the MAOPT with the Global Option Advisor which has the access to the global information of the environment. However, in many realistic scenarios, we can only obtain each agent's local information due to the partial observation. The other type contains MAOPT with the Local Option Advisor and MAOPT with the Successor Representation Option (SRO) which are suitable for this setting and collect each agent's local experience for the update. In many cases, each agent's experience is inconsistent with each other which causes the option-value estimation to oscillate and to become inaccurate. SRO is used to handle the experience inconsistency by decoupling the dynamics of the environment from the rewards to learn the option-value function under each agent's preference. MAOPT can be easily combined with existing deep RL approaches. Experimental results show it significantly boosts the performance of existing deep RL methods in both discrete and continuous state spaces.

Under review as a conference paper at ICLR 2021
TRANSFER AMONG AGENTS: AN EFFICIENT MULTIA- GENT TRANSFER LEARNING FRAMEWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Transfer Learning has shown great potential enhance the single-agent Rein- forcement Learning (RL) efficiency, by sharing learned policies of previous tasks. Similarly, in multiagent settings, the learning performance can also be promoted if agents can share knowledge with each other. However, it remains open ques- tion of how an agent should learn from other agents' knowledge. In this paper, we propose novel multiagent option-based policy transfer (MAOPT) framework to improve multiagent learning efficiency. Our framework learns what advice to give to each agent and when to terminate it by modeling multiagent policy transfer the option learning problem. MAOPT provides different kinds of variants which can be classified into two types in terms of the experience used during training. One type is the МАOPT with the Global Option Advisor which has the access to the global information of the environment. However, in many realistic scenarios, we can only obtain each agent's local information due to partial observation. The other type contains MAOPT with the Local Option Advisor and MAOPT with the Successor Representation Option (SRO) which are suitable for this setting and collect each agent's local experience for the update. In many cases, each agent's s experience is inconsistent with each other which causes the option-value estima- tion to oscillate and to become inaccurate. SRO used to handle the experience inconsistency by decoupling the dynamics of the environment from the rewards to learn the option-vales function under each agent's ' preference. MAOPT can be easily combined with existing deep RL approaches. Experimental results show it significantly boosts the performance of existing deep RL. methods in both discrete and continuous state spaces.
INTRODUCTION
Transfer Learning has shown great potential to accelerate single-agent RL via leveraging prior knowledge from past learned policies of relevant tasks (Yin & Pan 2017 Yang et al.) 2020). In-
spired by this, transfer learning in multiagent reinforcement learning (MARL) (Claus & Boutilier
1998 Hu & Wellman [998]] Bu et al., 2008 Hernandez-Leal et al. 2019 da Silva & Costal 2019)
also studied with two major directions: 1) transferring knowledge across different but similar MARL tasks and 2) transferring knowledge among multiple agents in the same MARL task. For the former, several works explicitly compute similarities between states or temporal abstractions (Hu) et al. 2015; Bouusioukis et al.] [201 Didi & Nitschke (0166) to transfer across similar tasks with the same number of agents, or design new network structures to transfer across tasks with different
numbers of agents (Agarwal et al. 2019 Wang et al.) 20200. In this paper, we focus on the latter multiagent system (MAS), each agent's experience is different, so the states each agent encounters (the degree of familiarity to the different regions of the whole environment) are also different; if we figure out some principled ways to transfer knowledge across different agents, all agents could form big picture about the MAS even without exploring the whole space of the environment, and this will definitely facilitate more efficient MARL (da Silva al. 2020).
direction due the tollowing intuition:
Transferring knowledge among multiple agents is still investigated at an initial stage, and the as- sumptions and designs of some recent methods usually simple. For example, LeCTR (Omid- shafiei et al., 2019) and HMAT (Kim et al.] (2020) adopted the teacher-student framework to learn to teach by assigning each agent roles (i.e., the teacher and the student), so the agent could learn

---

Under review as a conference paper at ICLR 2021
when and what to advise other agents or receive advice from other agents. However, both LeCTR and HMAT only consider two-agent scenarios. Liang & Li (2020) proposed a method under the succher-studett framework where each agent asks for advice from other agents through learning an attentional teacher selector. However, they simply used the difference of two unbounded value func- tions as the reward signal which may cause instability. DVM (Wadhwania al. 2019) and LTCR Xue et al. (2020) are two proposed multiagent policy distillation methods to transfer knowledge a- mong more than two agents. However, both methods decompose the solution into several stages in a coarse- gorrseer manner. Moreover, they consider the distillation equally throughout the whole train- ing process, which is counter-intuitive. A good transfer should be adaptive rather than being equally treated, e.g., the transfer should be more frequent at the beginning of the training since agents are less knowledgeable about the environment, while decay as the training process continues because agents are familiar with the environment gradually and should focus more on their own knowledge.
In this paper, we propose a novel MultiAgent Option-based based Policy Transfer (MAOPT) framework which models the policy transfer among multiple agents an option learning problem. In contrast to the previous teacher-student framework and policy distillation framework, MAOPT is adaptive and applicable to scenarios consisting of more than two agents. Specifically, MAOPT adaptively selects a suitable policy for each agent as the advised policy, which is used as a complementary optimization objective of each agent. MAOPT also uses the termination probability as a performance indicator to determine whether the advice should he terminated to avoid negative transfer. Furthermore, to facilitate the scalability and robustness, MAOPT contains two types: one type is MAOPT with the global option advisor (MAOPT-GOA), the other type consists of MAOPT with the local option advisor (MAOPT-LOA) and MAOPT with the successor representation option advisor (MAOPT- SRO). Ideally, we can obtain the global information to estimate the aption-value function, where MAOPT-GOA is used to select joint policy set, in which each policy is advised to each agent. However, in many realistic scenarios, we can only obtain each agents' local experience, where we adopt MAOPT-LOA and MAOPT-SRO. Each agent's experience may be inconsistent due to partial observations, which may cause the inaccuracy in option-value's 's estimation. MAOPT-SRO is used to overcome the inconsistency in multiple agents' experience by decoupling the dynamics of the environment from the rewards to learn the option-value function under each agent's preference. MАOPT can be easily incorporated into existing DRL approaches and experimental results show that it significantly boosts the performance of existing DRL approaches both in discrete and continuous state spaces.
2 PRELIMINARIES
Stochastic Games (Littman 1994) are natural multiagent extension of Markov Decision Pro- cesses (MDPs), which model the dynamic interactions among multiple agents. Considering the fact agents may not have access to the complete environmental information, we follow previ- ous work's 's settings and model the multiagent learning problems as partially observable stochastic games (Hansen et al. 2004). A Partially Observable Stochastic Game (POSG) is defined as a tuple (N,S,A',
A", RI. RR" 0² O²), where N is the set of agents; S is the set of s- tates; A is the set of actions available to agent i (the joint action space A = A' x A x Ax"); is the transition function that defines transition probabilities between global states: Sx AxS - [0, 1]; R' is the reward function for agent i: S x A + I and O' is the set of observations for agent i. A policy n²: O' A + [0, 1] specifies the probability distribution over the action space of agent The goal of agent is to learn a policy x' that maximizes the expected return with a discount factor y: J E, [E >rr]].
The Options Framework. Sutton et al. (1999) firstly formalized the idea of temporally extended action as an option. An option w 52 is deñied as a triple {Iu:, Tw. Bw)) in which I_ C S is an initiation state set, The is an intra-option policy and Ba I_ - [0. 1] is termination function that specifies the probability an option w terminates at state s E I_-. An MDP endowed with a set of options becomes a Semi-Markov Decision Process (Semi-MDP), which has a corresponding opti- mal option-valus -aauue function over options learned using intra-option learning. The options framework considers the and-rretur option execution model, in which agent picks option according to its option-value function (,,ss, w) and follows the intra-option policy Th until termination, then selects next option and repeats the procedure.
2

---

Under review as a conference paper at ICLR 2021
Deep Successor Representation (DSR). The successor representation (SR) (Dayan 1993) is basic scheme that describes the state value function by a prediction about the future occurrence of all states under fixed policy, SR decouples the dynamics of the environment from the rewards. Given a transition (s, a, 8 r)) ), SR is defined as the expected discounted future state occupancy:
8 M E Σ
s'[[s0 = 5, ag 7'1[s
(1)
where [[]] is an indicator function with value of one when the argument is true and zero otherwise. Given the SR, the Q-value for selecting action at state a can be formulated as the inner product of the SR and the immediate reward: Q (s) = Eees M(s, s', Rs'(
DSR (Kulkarni et al. 2016) extends SR by approximating it using neural networks. Specifical- ly, each state is represented by D-dimensional feature vector Φss which is the output of the network parameterized by A. Given d.. SR is represented as msr(os, alT) parameterized by т, a decoder ga(#x) parameterized by 9 outputs the input reconstruction 8, and the immediate reward at state s is approximated as a linear function of Φ,: R(s) = 6, w, where w E RD is the weight vector. In this way, the Q-value function can be approximated by putting these two parts togeth- er as: (" a) x Mar(( (Øs. alr) w. The stochastic gradient descent is used to update parameters (6, T, w. Specifically, the loss function of is:
L(T, 0) E
(10= [ + (bs
- mar (mr()))
where a' = arg max, Mar (B',, (5',a) a) w, and m is the target SR network parameterized by 7' which follows DQN (Mnih al.) 2013) for stable training. The reward weight is updated by minimizing
the loss function: L(w, 0) = (R(s) d.- w)² The parameter updated using an L2 loss:
L(0.0) 0) = ($ (5-s) s)² Thus, the loss function of DSR is the composition of the three loss functions:
L(O, T, W. () = L(T, L(w, 0) + L(O,
3
MULTIAGENT OPTION-BASED POLICY TRANSFER (MAOPT)
3.1 FRAMEWORK OVERVIEW
In this section, we describe our MAOPT framework in detail. Fig- ure Iillustrates the MAOPT framework which contains n agents interacting with the environment and corresponding option advi- sors. At each step, each agent obtains its own observation o', selects an action a' following its policy π', and receives its re- ward r², Each option advisor initializes the option set, and selects an option for each agent. During the training phase, the option advisor uses samples from all agents to update the option-value function and corresponding termination probabilities. Each agent advised by an option advisor, and the advice is to exploit this complemen-
Option Advibors
Replay
Butter
advice,
advico,
adivicaa
Agenn.
Agenty
Aggent
Obt Rowad
Environment
advised policy through imitation, which serves tary optimization objective (each agent does not know which pol- icy it imitates and how the extra loss function is calculated The exploitation of this advised policy is terminated as the selected option terminates and then another option is selected. In this way, each agent efficiently exploits useful information from other agents and as a result, the learning process of the whole system is accelerated and improved. Note that in the following section we assume the agents using the option advisor are homogeneous, i.e., agents share the same option set. While our MAOPT can also support the situation where each agent is initialized with different numbers of options, e.g., each agent only needs to imitate its neighbours. To achieve this, instead of input states into the ootion-value network, we just input the pair of states and options to the network and output a single option-value.
Figure 1: Framework overview.
Our proposed MAOPT can be classified into two types in terms of the experience used during train- ing. One type is MAOPT with the global option advisor (MAOPT-GOA) which has the access the
We provide the theoretical analysis to show this objective ensures to converge to an improved policy and will affect the convergence of the original RI. algorithm.
3

---

Under review as a conference paper at ICLR 2021
global information (i.e., (s, a, r, s'), where Ei= r") of the environment. Thus, MAOPT-GOA selects a joint option as the advice set given the global observation of the environment and then eval- uates the performance of the selected joint option. Selecting joint option means that each advice given to each agent begins and ends simultaneously. However, in many realistic scenarios, we can only obtain each agent's local information due the partial observation. Moreover, the degree of familiarity to the environment of each agent is different, then some agents may need to imitate their teachers for a longer time. Therefore, more flexible way to control each advice when to terminate individually is necessary. The other type contains MAOPT with the local option advisor (MAOPT- LOA), and MAOPT with the successor representation option advisor (MAOPT-SRO) which collects each agent's local experience for the update. In many cases, each agent's 's experience is inconsistent with each other, e.g., each agent has an individual goal to achieve or has different roles, and the rewards assigned to each agent are different. If we simply use all experiences for the update, the option-value estimation would oscillate and become inaccurate. MAOPT-SRO is used to handle the experience inconsistency by decoupling the dynamics of the environment from the rewards to learn the option-value function under each agent's 's preference.
3.2 MAOPT-GOA
In cases where we have access the global information of the environment, the global option advisor is used to advise each agent. The procedure of MAOPT-GOA is described as follows (pseudo-code is included in the appendix). First, MAOPT-GOA initializes the joint option set
Ω1 x x...x x 12" (where Q = fw f.r _.w")) }). Each option w corresponds to agent i's policy x'. The joint option-value function defined as (s [.] ()) parameterized by vi which evaluates the performance of each joint option a. The corresponding target network is parameterized by e which copies from v every steps. The termination network parameterized by w outputs the termination probability (s',, ()w) of the joint option W.
The update of the joint option-value network update follows previous work (Sutton et al. 1999, Bacon et al. 2017). We first samples B transitions uniformly from the global replay buffer, for each sample (s, 0, r, s'). we calculate the joint U function, the joint option-value function upon arrival:
Uls (1 B(s ت x)w) (s aWW') + B(s', a w) max Q=(s', ('l,,,)'1))).
(3)
deef
Then, the option-valus value network minimizes the following loss:
Lg
1
Σ (rb + 7U(UU Qu Hk)))
(4)
B
where rb =E,
According to the call-and-return option execution model, the termination probability Ba controls when terminate the selected joint option and then to select another joint option accordingly, which is updated w.r.t w as follows (Bacon et al. 2017):
03(8' ,.)=
a=
A(s', (lw)) + +
(5)
dw
where, A(s', (5/l)) is the advantage function which can be approximated as Q= W (/)) ma (= 5f (v'), and & is regularization term to ensure explorations (Bacon et al. 2017 Then, given each state 8, MAOPT-GOA selects a joint option W following the e-greedy strategy over the joint option-value function. Then MAOPT-GOA calculates the cross-entropy H (T. /m²) be- tween each intra-option policy Tw and each agent's 's policy x", and gives it to each agent respectively, serving as a complementary optimization objective of each agent, which means that apart from max- imizing the cumulate reward, the agent also imitates the intra-option policy Tw by minimizing the loss function Lt, The imitation for the intra-option policy is terminated as the option terminated, and then another option is selected to provide advice for the agent. The formula of the loss function Li is given as follows:
T.J. = S(1) (Tw))'))
where, f(t) 0.5 tanh - pat) is the weighting factor of H(T.(T²). μ is hyper-paramett parameter that controls the decrease degree of the weight. This means that at the beginning of learning, each agent exploits knowledge from other agents mostly. As learning continues, knowledge from other
4

---

Under review as a conference paper at ICLR 2021
Inpat Reconstruction
MON In')
Advico
Reward W Weights
Agent!
Grato Lose
Option
Option-value function
Co's
Qu (o(.oog))
Critic
al
mir (o") lp-
Qu eb2)
or
Q..(o'. aug)
0'
Actor
Blo'
Parkyy
A(o', an))
Option termination probabilities
a
Blol ab))
Figure 2: MAOPT-LOA.
Figure 3: The SRO architecture.
agents becomes less useful and each agent focuses more the current self-learned policy. Note that we cannot calculate the cross-entropy between the discrete action policies directly. To remedy this, apply the softmax function with some temperature the discrete action vectors to transform the actions into discrete categorical distributions.
In MAOPT-GOA, each advice given each agent begins and ends simultaneously. While for each agent, when to terminate the reusing of other agents' knowledge should be decided asynchronously and individually since the degree of familiarity to the environment of each agent probably not identical. Moreover, in many realistic scenarios, we can only obtain each agent's local information due to partial observation. Therefore, more flexible way advise each agent necessary. In the following section, we describe the second type of MAOPT in detail.
3.3 MAOPT-LOA
MAOPT-LOA equips each agent an option advisor, and each advisor uses local information from all agents for estimation. How MAOPT-LOA is applied in actor-critic methods illustrated in Figure (pseudo-code is included in the appendix). Firstly, MAOPT-LOA initializes options w _ w²) }. Each option w corresponds to agent i's policy n². The input of option network parameterized by z6 and termination network parameterized by w is the local observation of each agent i. The option- value function Qu (o w (b) and termination probability 3(o', w are used to evaluate the performance of each option w' E Ω.
The update of the option-value function and the termination probability is similar to that in MAOPA- GOA. For the update of each agent MAOPT-LOA first selects an option from fwl _ ..... " following c- greedy strategy over the option-valle -aluee function. Then MAOPT-LOA calculates the cross- entropy H (m))=²²) between each intra-option policy Tu and each agent's policy æ and gives it to each agent respectively, contributing to a complementary loss function Li,, for each agent.
Note that the option-value network and termination network collect experience from all agents for the update. What if the experience from one agent is inconsistent with others? In a POSG, each agent can only obtain the local observation and individual reward signal, which may be different for different agents even at the same state. If we use inconsistent experiences to update one shared option-value network and termination network, the estimation of the option-valus function would oscillate and become inaccurate. We propose MAOPT-SRO to address this problem. MAOPT-SRO decouples the dynamics of the environment from the rewards to learn the option-value function under each agent's 's preference. In this way, MAOPT-SRO can address such sample inconsistency and learn the aation-valus and the corresponding termination probabilities under each agent's 's preference which is described in the next section.
3.4 MAOPT-SRO
MAOPT-SRO applies novel option learning algorithm, Successor Representation Option (SRO) learning to learn the option-value function under each agent's 's preference. The SRO network archi- tecture is shown in Figure 3 with each observation o¹ from each agent as input. of is input through two fully-connected layers to generate the state embedding 6,,, which is transmitted to three net- work sub-modules: The first sub-module contains the state reconstruction model which ensures 6,t well representing of, and the weights for the immediate reward approximation at local obser- vation of. The immediate reward is approximated linear function of Φoi: R2 (tho)) Φo w,
5

---

Under review as a conference paper at ICLR 2021
where RR is the weight vector. The second sub-module is used to approximate SR for options mar( (0 T) which describes the expected discounted future state occupancy of executing the op- tion The corresponding target network is parameterized by 7' which copies from every l steps. The last sub-module is used to update the termination probability I(0 w/w), which is similar to that in MAOPT-LOA described in Section 3.3
Algorithm MAOPT-SRO.
Initialize: option set (w w 1.... .w"). state feature parameters 0, reward weights w, state reconstruction network parameters 8, termination network parameters w, SR network parame- ters T, SR target network parameters r', batch size T for PPO, replay buffer D', actor network parameters p², and critic network parameters of for each agent
2: for each episode do
3: Start from state
4: for each agent i do
5: Select option
6: Select an action o² ~ ('O)'
7: end for
8: Perform the joint action ā = {a² )"}
9: Observe reward F (rl.... , and new state s
rn)
10: for each agent i do
11: Store transition (o', at, of ,w to replay buffer D
12: Select another option wn w terminates
end for
14:
for every I' steps do
for each agent do
16:
Set T n'
17:
Calculate the advantage A = Ee 14 - Vo-(o))
18:
Optimize the critic loss === = Ei=l -1=11 2t=1(en (Ersi pp V (o)))2
19:
The option advisor calculates the transfer loss Lí = f(t)
20:
Optimize the actor loss Σ: = end for
-I 11=1 (aala())))
(a;1of) A$ AKL(T + Dip w.r.t p²
21:
22:
end for
23:
Sample batch of B /N transitions (o a', 1r,00
of .)) from each D
Optimize [. 0) (95(90)) - )²)² w.r.t 0,0
25:
Optimize T.(w, ()) (r* Φot w)² w.r.t w,
26:
for each w do if' Th selects action a" observation of then
28:
Calculate Ü(Ø oe' w(TT)
Set y f 6a ( (()"")
30:
Optimize the following loss w.r. T: L(T, 0) = Es (vb
msr (0or, ( w|)))
08(0 , [1)
Optimize the termination network w.r.t w w a=
Ald A(0_,,0077))
32:
end if
end for
34:
Copy to every k steps
35:
end for
Given (or w)) the SRO-value function be approximated mar (#or, (T) w. Since options are temporal abstractions (Sutton et al. 1999). SRO also needs to calculate the Ù function which is served as SR upon arrival, indicating the expected discount- ed future state occupancy of executing an option w upon entering a state. Given the transition (or a', r', we firstly introduce the SR upon arrival U as follows:
Qu(orr se
Ü(b,r,w/7') = (1 - B(0,r', ((o,r,,/=)))m (m) w/7") + 3(0,r i () w(()) where w' arg max msr (0(7" W.
(7)

---

Under review as a conference paper at ICLR 2021
We consider MAOPT-SRO combing with PPO (Schulman et al. 2017 popular single-agent policy-based RL. The way MAOPT-SRO combines with other policy- based RL algorithms is sim- ilar. Algorithm T illustrates the whole procedure of MAOPT-SRO. First, we initialize the network parameters for the state embedding network, reward prediction network, state reconstruction net- work, termination network, SR network, SR target network, and the actor and critic networks of each agent (Line 1). For each episode, each agent first obtains its local observation of which cor- responds to the current state s (Line 3). Then, MAOPT-SRO selects an option w for each agent i (Line 5), and each agent selects an action a' following its policy x' (Line 6). The joint action ā is performed, then the reward r and new state s' is returned from the environment (Lines 8,9). The transition is stored in the replay buffer D' (Line 11). If w terminates, MAOPT-SRO selects another option for agent (Line 12). For each update step, each agent updates its critic network by minimiz- ing the loss L.g (Line 18), where T is the length of the trajectory segment. Then each agent updates its actor network by minimizing the summation of the original loss and the transfer loss Li,- (Line 20). For the update of SRO, it first samples batch of B/N transitions from each agent's 's buffer D', which means there are B transitions total for the update (Line 23). SRO loss is composed of three components: the state reconstruction loss L(8, 8), the loss for reward weights L(w, 0) and SR loss L(T,0) 0). The state reconstruction network is updated by minimizing two losses L(0, 0) (Line 24) and L(w,0) 0) (Line 25). The second sub-module, SR network approximates SR for options and is updated by minimizing the standard L2 loss L(T, 0) (Lines 26-30). At last, the termination prob- ability of the selection option is updated (Line 31), where Af w('") ) is the advantage function and approximated as (a " (/WWTT) (0) max mar ( w, and is a regularization term to ensure explorations.
4 EXPERIMENTAL RESULTS
In this section, we evaluate the performance of MAOPT combined with the common single-agent RL algorithm (PPO (Schulman et al.. 2017) and MARL algorithm (MADDPG (Lowe et al. (017)). The test domains include two representative multiagent games. Pac-Man and multiagent particle environment (MPE) (illustrated in the ap- pendix). Specifically, we first combine MAOPT-LOA and MAOPT- SRO with PPO on Pac-Man to validate whether MAOPT-SRO suc- cessfully solves the sample inconsistency due to the partial ob- servation. Then, we combine MAOPT-GOA, MAOPT-LOA, and MAOPT-SRO with two baselines (MADDPG and PPO) on MPE to further validate whether MAOPT-SRO is more flexible way for knowledge transfer among agents and enhances the advantage of our framework. We also compare with DVM (Wadhwania al. 2019) which is the most recent multiagent transfer method
Beeeggeeeee reword
IPO MACIFALDA
0.8
K10
(a) OpenClassic
Nowrags reward
4.1 PAC-MAN
mo
MAcer 400
DXN
a mixed cooperative- competitive maze game with one pac-man player and two ghost players. The goal of the pac-man player is to eat as many pills as possible and avoid the pursuit of ghost players. For ghost players, they aim to capture the pac-man player as soon as possible. In our settings, we aim to control the two ghost players and the pac-man player as the opponent is controlled by well pre-traing trained PPO poli- cy. The game ends when one ghost catches the pac-man player the episode exceeds 100 steps. Each player receives -0.00 -0.01 penalty each step and +5 reward for catching the pac-man player.
Pac-Man (van der Ouderaa 2016)
Training eprsodes
x10
(b) MediumClassic
Figure 4: The performance on Pac-Man.
We consider two Pac-Man scenarios (OpenClassic and MediumClassic) with the game difficulties increasing. Figure (a) presents the average rewards on the OpenClassic scenario. We can see that both MAOPT-LOA and MAOPT-SRO perform better than other methods and achieve the average
"hhe details of neural network structures and parameter settings are in the appendix, and we share network parameters among all homogeneous agents (Gupta et al. 2017 Rashid et al. 2018).
7

---

Under review as a conference paper at ICLR 2021
discount rewards of +3 approximately with smaller variance. In contrast, PPO and DVM only achieve the average discount rewards of +2.5 approximately with larger variance. This phenomenon indicates that both MAOPT-LOA and MAOPT-SRO enable efficient knowledge transfer between two ghosts, thus facilitating better performance.
Next, we consider more complex Pac-Man game scenario, where the layout size is larger than the former and it contains obstacles (walls). From Figure (b) we can observe that the advantage of MAOPT-LOA and MAOPT-SRO is much more obvious compared with PPO and DVM. Further- more, MAOPT-SRO performs best among all methods, which means that MAOPT-SRO effectively selects more suitable advice for each agent. The reason that MAOPT performs better than DVM is that MAOPT enables each agent to effectively exploit useful information from other agents through the option-based call-and-returi mechanism, which successfully avoids negative transfer when other agents' policies are only partially useful. However, DVM just transfers all information from other agents through policy distillation. By comparing the results of the two scenarios, we see that the superior advantage of MAOPT-SRO increases when faced with more challenging scenarios. Intu- itively, as the environmental difficulties increase, agents are harder to explore the environment and to learn the optimal policy. In such a case, agents need to exploit the knowledge of other agents more efficiently, which would greatly accelerate the learning process as demonstrated by MAOPT-LOA and MAOPT-SRO.
4.2
MPE
MPE (Lowe et al. 2017) is simple multiagent particle world with continuous observation and discrete action space. We evaluate the performance of MAOPT on two scenarios: predator and prey, and cooperative navigation. The predator and prey contains three a- gents which are slower and want catch one adversary (rewarded +10 by each hit). The adversary is faster and wants to avoid be- ing hit by the other three agents. Obstacles block the way. The cooperative navigation contains four agents, and four correspond- ing landmarks. Agents are penalized with a reward of collide with other agents. Thus, agents have to learn to cover al- I the landmarks while avoiding collisions. Both games end when exceeding 100 steps.
300
MAGFZLOA
MACPTIING
200
MAGEFIGDA
100
they
Training episodes
XID
(a) PPO
400
100
MACETCCAA
Both domains contain the sample inconsistency problem since each agent's local observation contains the relative distance between other agents, obstacles, and landmarks. Moreover, in cooperative navigation, each agent is assigned different task, i.e., approach- ing a different landmark from others, which means each agent may receive different rewards under the same observation. Therefore, we cannot directly use all experience to update one shared option- value network. In such case, design individual option learning module for each agent in MAOPT-LOA, which only col- lects one agent's experience update the option-value function.
300
100
0.0
Training episodes
xTm
(b) MADDPG
Figure 5: The performance on predator and prey.
Figure 5 (a) shows the average rewards on predator and prey. We can see that all our proposed MAOPT-GOA, MAOPT-LOA, and MAOPT-SRO (combined with PPO) achieve higher average rewards than PPO and DVM. Figure 5 (b) demonstrates similar phenomenon that both MAOPT-GOA and MAOPT-SRO (combined with MADDPG) perform bet- ter than vanilla MADDPG, and MAOPT-SRO performs best among all methods. This is because MAOPT-SRO uses all agents' experience for the update and efficiently distinguishes which part of the information is useful and provides positive advice for each agent. Furthermore, it uses the indi- vidual termination probability to determine when to terminate each agent's 's advice, which a more flexible manner, thus facilitating more efficient and effective knowledge transfer among agents.
Table shows the average distance between each agent and its nearest landmark (line 1), and the average collision frequencies of agents (line 2) in cooperative navigation. In this game, agents are required to cover all landmarks while avoiding collisions. Therefore, a better result means to get a closer average distance between agents and landmarks, and less collision frequencies among agents.
8

---

Under review as a conference paper at ICLR 2021
Table Average of collisions and average distance from a landmark in cooperative navigation.
PPO
DVM
MADDPG
MAOPT- GOA
MАOPT-
MAOPT-
LOA
SRO
Avg. dist.
1.802
1.685
1.767
1,446
1.476
1.366
Collisions
0.163
0.144
0.209
0.153
0.124
0.122
We can see that MAOPT-GOA, MAOPT-LOA, and MAOPT-SRO achieve the less collisions and the shorter average distance from landmarks than other methods. Furthermore, MAOPT-SRO performs best among all methods. The superior advantage of MAOPT is due to the effectiveness in identifying the useful information from other agents' policies. Therefore, each agent exploits useful knowledge of other agents and as a result, thus leading to the least collisions and the minimum distance from landmarks.
Finally, we provide an ablation study to investigate whether MAOPT-SRO selects a suitable policy for each agent, thus efficiently enabling agents to exploit useful information from others. Figure g presents the action movement in the environment, each arrow is the direc- tion of movement caused by the specific action at each location. Four figures show the direction of movement caused by the action selected from the policy of an agen- I at 6
10a steps (Figure d a). top left), and at to = 2 10th (Figure cc), bottom left); the direction of movement caused by the action selected from the intra- option policies of MAOPT-SRO at 4 = 6 x 10 steps (Figure b), top right), and at l/2 = x 10 steps (Fig- ure o d), bottom right) respectively. The preferred direc- tion of movement should be towards the blue circle. We can see that actions selected by the intra-option policies of MAOPT-SRO are more accurate than those selected from the agent's 's own policy, namely, more prone pur- sue the adversary (blue). This indicates that the advised policy selected by MAOPT-SRO performs better than the agent itself, which means MAOPT-SRO successfully dis- tinguishes useful knowledge from other agents. There- fore, the agent can learn faster and better after exploiting knowledge from this advised policy than learning from scratch.
(a) PPO.,
(b) MAOPT-SRO,
(c) PPO
(d) MAOPT-SRO
Figure 6: Analysis of agent I's policy and MAOPT-SRO's policy.
CONCLUSION AND FUTURE WORK
In this paper, we propose a novel MultiAgent Option-baser basion Policy Transfer (MAOPT) framework for efficient multiagent learning by taking advantage of option- policy transfer. Our framework learns what advice to give to each agent and when to terminate it by modeling multiagent transfer as the option learning problem. Furthermore, to facilitate the robustness of our framework, we provide two types: one type is MAOPT-GOA, which is adopted in fully cooperative settings (with access to global state and reward). The other type contains MAOPT-LOA and MAOPT-SRO, which are proposed for mixed settings (only access to local state with inconsistency and also individual re- wards). MAOPT-SRO is proposed to solve the sample inconsistency due to the partial observation, by decoupling the dynamics of the environment from the rewards to learn the option-value function under each agent's preference. MAOPT can be easily combined with existing DRL approaches. Experimental results show it significantly boosts the performance of existing DRL methods. As for future work, it is worth investigating how to achieve coordination among agents by designing MAOPT-GOA in centralized training, decentralized exccution manner. For example, it is worth in- vestigating how to decompose the joint option-valie value function into individual option-vaude valuue functions and update each termination probability separately.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Akshat Agarwal, Sumit Kumar, and Katia P. Sycara. Learning transferable cooperative behavior in multi-agent teams. CoRR, abs/1906.012000 2019.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of AAAi pp. 1734-1734, 2017.
Georgios Boutsioukis, Ioannis Partalas, and Ioannis P. Vlahavas. Transfer learning in multi-agent reinforcement learning domains. In Recent Advances in Reinforcement Learning 9th European Workshop, pp. 249-260, 2011.
Lucian Bu, Robert Babu, Bart De Schutter, et al. A comprehensive survey of multiagent reinforce- ment learning. IEEE Transactions on Systems, Man, and Cybernetics, 38(2):156- 172-1,, 2008.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multia- gent systems. In Proceedings of AAAI/IAA,, pp. 746-752, 1998.
Felipe Leno da Silva and Anna Helena Reali Costa. survey transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research, 64:645-703, 2019.
Felipe Leno da Silva, Garrett Warnell, Anna Helena Reali Costa, and Peter Stone. Agents teaching agents: survey on inter-agent transfer learning. Auton. Agents Multi Agent Syst. 34(1):9 2020.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5(4):613-624, 1993.
Sabre Didi and Geoff Nitschke. Multi-agent behavior-based policy transfer. In Proceedings of European Conference on the Applications of Evolutionary Computation, pp. 181-197, 2016.
Jayesh K. Gupta, Maxim Egorov, and Mykel J. Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. Autonomous Agents and Multiagent Systems AAMAS 2017 Workshops, pp. 66-83, 2017.
Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In Proceedings of AAAI, volume 4, pp. 709-715, 2004.
Pablo Hernandez- Bilal Kartal, and Matthew E. Taylor. survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent
2019.
Junling Hu and Michael Wellman. Multiagent reinforcement learning: Theoretical framework and an algorithm. In Proceedings of ICML, pp. 242-250, 1998.
Yujing Hu, Yang Gao, and Bo An. Accelerating multiagent reinforcement learning by equilibrium transfer. IEEE Trans. Cybernetics, 45(7):1289-1322 2015.
Dong-Ki Kim, Miao Liu, Shayegan Omidshafiei. Sebastian Lopez-Cot, Matthew Riemer, Golnaz Habibi, Gerald Tesauro, Sami Mourad, Murray Campbell, and Jonathan P. How. Learning hierar- chical teaching policies for cooperative agents. In Proceedings of the 19th International Confer- ence on Autonomous Agents and Multiagent Systems, pp. 620-628, 2020.
Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02550 2016.
Yongyuan Liang and Bangwei Li. Parallel knowledge transfer in multi-agent reinforcement learning. CoRR, abs/2003. 13085, 2020.
Michael L. Littman. Markov games framework for multi-agent reinforcement learning. In Proceedings of ICMI, pp. 157-163, 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compettive environments. In Proceedings of NewrIPS, pp. 6379- 6390, 2017.
10

---

Under review as a conference paper at ICLR 2021
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioamnis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Shayegan Omidshafiei, Dong-Ki Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, and Jonathan P. How. Learning to teach in cooperative multiagent reinforcement learning. Proceedings of AAAI, pp. 6128-6136, 2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foer- ster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. Proceedings of the 35th International Conference on Machine Learn- ing. pp. 4292-4301, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Leonid P Shil' nikov. Methods of qualitative theory in nonlinear dynamics, volume 5. World Scien- tific, 2001.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A frame- work for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181 211, 1999. ISSN 0004-3702.
Tycho van der Ouderaa. Deep reinforcement learning in pac-man. 2016.
Samir Wadhwania, Dong-Ki Ki Kim, Shayegan Omidshafiei, and Jonathan P. How. Policy distillation and value matching in multiagent reinforcement learning. In 2019 IEFF/RSJ International Con- ference Intelligent Robots and Systems, IROS 2019, Macau, SAR, China, November 3-8, 2019, pp. 8193-8200, 2019.
Weixun Wang. Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few more: large-scale dynamic multiagent curriculum learning. Proceedings of AAAI, 2020.
Zeyue Xue, Shuang Luo, Chao Wu, Pan Zhou, Kaigui Bian, and Wei Du. Transfer heteroge- neous knowledge among peer-to-peer teammates: model distillation approach. CoRR, ab- s/2002.02202, 2020.
Tianpei Yang, Jianye Hao, Zhaopeng Meng, Zongzhang Zhang, Yujing Hu, Yingfeng Chen, Changjie Fan, Weixun Wang, Wulong Liu, Zhaodong Wang, and Jiajie Peng. Efficient deep rein- forcement learning via adaptive policy transfer. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3094-3100, 2020.
Haiyan Yin and Sinno Jialin Pan. Knowledge transfer for deep reinforcement learning with hierar- chical experience replay. In Proceedings of AAAI, pp. 1640-1646, 2017.

---

Under review as a conference paper at ICLR 2021
A PSEUDO-CODE FOR MAOPT-GOA AND MAOPT-IOA
Algorithm MAOPT-GOA.
1: Initialize: the joint option set Si x ... xΩn,, each 92 = {wi, w2, ... Wn ). joint option-value network parameters 1/ joint option-value target network parameters v/' termination network parameters w. replay buffer D, actor network parameters pt for each agent i, critic network parameters v² for each agent batch size T for PPO
2: for each episode do
3: Start from state
4: Select joint option a
5: for each agent i do
6: Select action a' ~ (o')
7: end for
8: Perform the joint action ă {a' )²)
9: Observe reward r {r'.... rd) and new state
10: Store transition (s, c. r, a3, 8') replay buffer D
Select another joint option " if / terminates
12:
for every T steps do for each agent do
14:
Set T = n¹
15:
Calculate the advantage A Evze Evser Vor(of)
16:
Optimize the critic loss L = 1==
(Ersi ' Ei=r(Er
Vy (oi))²
17:
The option advisor calculates the transfer loss L = [(1) (T (x (x))
18:
Optimize the actor loss L: = 1t=1 (a end for
<'aajlo/) AB [ (T0) + Diy w.r.t p
20:
end for
21:
Sample batch of transitions from D
22:
for each sample (s, a, r, 8') do
23:
for each 3 do
24:
if Twe selects action a² for all E a then
25:
Calculate (s',, a )
26:
Set + r (U( (s',)l1)) (lb"")
Optimize the following loss w.t th: Lg BB E. (yo (sb, 03 ))
28:
Optimize the termination loss w.n.t Ok
0 A(s', a3 (b))
Dw
29: end if
30:
end for
31:
end for
Copy & to e every l steps
33:
end for
Algorithm illustrates the whole procedure of MAOPT-GOA. First, we initialize the network pa- rameters for the joint option-value network, termination network, joint option target network, and the actor and critic networks of each agent For each episode, each agent first obtains its local observation of which corresponds to the current state (Line 3). Then, MAOPT-GOA selects a joint option @ for all agents (Line 4), and each agent selects an action a' following its policy n' (Lines 5-7). The joint action ă is performed, then the reward and new state s' is returned from the envi- ronment (Lines 8, 9). The transition stored the replay buffer D (Line 10). If / terminates, then MAOPT-IOA selects another joint option J' (Line 11). For each update step, each agent updates its critic network by minimizing the loss L (Line 16), and updates its actor network by minimizing the summation of the original loss L' and the transfer loss L;, (Line 18).
For the update of GOA. it first samples batch of B transitions from the replay buffer D (Line 21). Then GOA updates the joint option-vall value network by minimizing the standard L2 loss ( (Lines 22-27). At last, the termination probability of the selection joint option is updated (Line 28).
12

---

Under review as a conference paper at ICLR 2021
Algorithm 3 MAOPT-IOA.
1: Initialize: the option set fwi,".. w. option-value network parameters 16, option- target network parameters v., termination network parameters w. replay buffer D' for each agent i, actor network parameters p² for each agent critic network parameters o² for each agent batch size T for PPO
2: for each episode do
3: Start from state
4: for each agent i do
5: Select option w
6: Select an action a² (o')
7: end for
8: Perform the joint action {at am))
9: Observe reward = (.".... r")
and new state s'
10: for each agent i do
Store transition (o', a', r², of ,w, replay buffer D
12:
Select another option w' w terminates
13:
end for
14:
for every 7 steps do
15:
for each agent do
16:
Set T =
17:
Calculate the advantage A = Evze 2 (o
18:
Optimize the critic loss L = Et= (Errs 7t 'pi Vu. (o)))²
19:
The option advisor calculates the transfer loss Lir = (Tx)/(F
20:
Optimize the actor loss Li Dt=1 x (o end for
Af d + Li w.r.t p
21:
22:
end for
23:
Sample batch of B/N transitions from each D
24:
for each sample (o', a", (o',a', of do for each w do if nw selects at o; then
25:
26:
27:
Calculate U(o", ww(e)")
Set f- r¹ + >U (o' ,w/x²") Optimize the option-value network by minimizing the following loss w.r.t T:
29:
La E. (3b- wl)))
09 w||=) A(o ,w/(b" + E
30:
Optimize the termination network W.E. Ok end if end for
dw
31:
32:
33:
end for
Copy & to e every l steps
35:
end for
AlgorithmB Billustrates the whole procedure MAOPT-IOA. First, we initialize the network param- eters for the option-value network, termination network, option target network, and the actor and critic networks of each agent For each episode, each agent first obtains its local observation of which corresponds to the current state (Line 3). Then, MAOPT-IOA selects an option w for each agent i (Line 5), and each agent selects an action a² following its policy x' (Line 6). The joint action a performed, then the reward and new state s' is returned from the environment (Lines 8, 9). The transition is stored to each agent's 's replay buffer D' (Line 11). If w terminates, then MAOPT-IOA se- lects another option for agent i (Line 12). For each update step, each agent updates its critic network by minimizing the loss L' (Line 18), and updates its actor network by minimizing the summation of the original loss L and the transfer loss I (Line 20).
13

---

Under review as a conference paper at ICLR 2021
For the update of IOA, it first samples batch of B/N transitions from each agent's buffer D' (Line
23). Then MAOPT-IOA updates the option- value network by minimizing the standard L22 loss L(w) (Lines 24-29). At last, the termination probability of the selection option is updated (Line 30).
B ENVIRONMENT ILLUSTRATIONS AND DESCRIPTIONS
Pac-Man (van der Ouderaa 2016) is a mixed cooperative-compeitty maze game with one pac-man player and two ghost players (Figure The most complex scenario is shown in Figure To with four ghost players and one pac-man player. The goal of the pac-man player is to cat as many pills (denoted as white circles in the grids) possible and avoid the pursuit of ghost players. For ghost players, they aim to capture the pac-man player soon possible. In our settings, we aim to control the two ghost players and the pac-man player as the opponent controlled by well pre-trained PPO policy. The game ends when one ghost catches the pac-man player the episode exceeds 100 steps. Each player receives -0.01 penalty each step and +55 reward for catching the pac-man player.
MPE Lowe et al. 2017) is simple multiagent particle world with continuous observation and discrete action space. We evaluate the performance of MAOPT on two scenarios: predator and prey (Figure g with four agents and Figure with twelve agents), and cooperative navigation (Figure with four agents and Figure with ten agents). The predator and prey contains three (nine in adversary (blue (rewarded +10 by each hit). The adversary is faster and wants to avoid being hit by the other three agents. Obstacles (grey) block the way. The cooperative navigation contains four (ten in Figure [1] agents (green), and four corresponding landmarks (cross). Agents are penalized with a reward of if they collide with other agents. Thus, agents have learn cover all the landmarks while avoiding collisions. At each step, each agent receives reward of the negative value of the distance between the nearest landmark and itself. Both games end when exceeding 100 steps.
Figure [1] agents (green) which slower and want to catch
(a) Open Classic
(b) Medium Classic
Figure Pac-Man.
Figure 8:
Predator and prey Figure 9: Cooperative naviga- tion (N 4).
(N 4).
Figure 10: OriginalClassic
Figure 11: Predator and prey Figure 12: Cooperative naviga- (N = 12).
tion (N = 10).
State Description
Pac-Man The layout size two scenarios are 25 x 9 (OpenClassic)). 20 x 11 (MediumClassia) and 28 27 (OriginalClassic) respectively. The observation of each ghost player contains its position,
14

---

Under review as a conference paper at ICLR 2021
the position of its teammate, walls, pills, and the pac-man, which is encoded as one-hot vector. The input of the network is a 68-dimension in OpenClassic, 62-dimension in MediumClassic and dimension in OriginalClassic.
MPE The observation of each agent contains its velocity, position, and the relative distance between landmarks, blocks, and other agents, which is composed of 18-dimension in predator and prey with four agents (36-dimension with twelve agents), 24-dimension cooperative navigation with four agents (60-dimension with agents) as the network input.
C ADDITIONAL RESULTS AND ANALYSIS
We here summarize the components and properties of our framework, and list the suitable scenarios of each option advisor. MAÒPT-GOA contains agent models and model for learning the joint- option value function and termination probabilities. MAOPT-GOA is used when we can obtain the global information. While in practice, only partial observations available in some environments. Therefore, we also provide MAOPT-LOA to enable knowledge transfer among agents. MАOPT- GOA contains n agent models and 1 model for learning the individual option value function and termination probabilities. The option model adopts the parameter sharing similar to common MARL training. However, each agent only obtains the local observation and individual reward signals, which may be different for different agents even at the same state. If we use inconsistent experiences to update the option-value network and termination network, the estimation of the option- option-value function would oscillate and become inaccurate. Due to partial observability and reward conflict, we design a novel option learning based on successor features. MAOPT-SRO contains n agent models and 1 model for learning the individual SRO value function and termination probabilities. The SRO model adopts the parameter sharing similar to common MARL training.
Table 2: Aspects of MAOPT with three kinds of option adivsors.
MAOPT-GOA
MAOPT-LOA
MAOPT-SRO
agent's models
n agent's models SRO model
agent's models 1 joint option model
Components
option model
(parameter sharing)
(parameter sharing)
Partial observation
Reward conflicts
The degree of flexibility
weak
strong
strong
Figures 1 and 14 are the enlarged version corresponding to Figures 5
(a) and (b). Table 3 and
presents the average return achieved in predator and prey under two basclines: PPO and MADDPG.
Table 3: Average return with standard deviation in predator and prey (N 4). First maximum value bolded.
PPO
DVM
MAOPT-GOA 285.27 + 24.66
MAOPT-LOA 268.67 + 23.18
MAOPT-SRO
370
241.87 + 12.38
296.88+18.75
Table 4: Average return with standard deviation in predator and prey (N = 4). First maximum value bolded.
MADDPG
MAOPT-GOA
MAOPT-SRO
229.68
301.48 + 36.22
338.8 + 8.78
Experimental results games shown in Figure 10 12 and 11 are shown in the following Figure L5
We can observe that MAOPT-SRO outperforms PPO and DVM, scales well with the increase in the number of agents.
15

---

Under review as a conference paper at ICLR 2021
PPO
300
MAOPT-LOA MAOPT-SRO
Average reward
DVM
MAOPT-GOA
200
100
0.0
0.4
0.8
1.2
1.6
2.0
Training episodes
x104
Figure 13: The performance predator and prey (N 4).
400
MADDPG
MAOPT-SRO
MAOPT-GOA
Average reward
300
200
100
0.0
0.4
0.8
1.2
1.6
2.0 x104
Training episodes
Figure 14: The performance on predator and prey (N = 4).
200
220
SUNDITI 1010
-100
DVN
19-3
Average reward
-125
0.0
0.4
Tiwiniing spisodes
0.8
1.2
1.6
x10
2.0
-50
Treining ephondex
x1o²
-205
0.0
0.4
Truning episodas
0.8
1.2
16
2.0
OriginalClassic
(b) Predator and prey (N = 12)
(c)
Cooperative navigation (N
10)
Figure 15: The performance on various games.
16

---

Under review as a conference paper at ICLR 2021
D THEORETICAL ANALYSIS
We first explain that the advised policy is better than the agent's own policy. none of other agents' policies is better than the agent's s own policy, then the advised policy is the agent's own policy, which means there is no need to imitate.
The intuitive explanation of such transfer among agents is based on mutual imitation among agents. If an agent imitates a policy which is better than its own policy, it can achieve higher performance.
Which policy should be imitated by which agent decided by our option advisor. The option-valus value function estimates the performance of each option, as well as the intra-option policy, therefore we select the option with the maximum option-value for each agent imitate the intra-option policy of this option. The convergence of option-value learning has been proved and verified (Sutton et all
1999 Bacon et al. 2017 timestep.
Therefore, the advised policy is the best among all policies at current
Then we provide the theoretical analysis to show the agent's policy will finally converge to the imitated policy through imitation. Give agents, options, for each agent the option advisors selects an option w arg max (w, w) and w contains the policy ((,((p)) with the maximum expected return (s). Each agent imitates the advised policy (pt) to minimizes the difference between two policies (pt) and (p.( A P: = (s(p)) (p - p.). If we set I2 P P.. then calculate the difference of I; follows: 24 -as (1), (s)
Then we have
an (vi, - m.((slpt))
-a (12, (spp()))))
0
4 0
0
0
0
On Cn (slpt) - no (s(pp))
=AØ
Note that (s)-n.)(ss) 2 then -a, (s) (s)) S O, the main diagonal of the diagonal ma- trix A only contains non-positive values. Therefore, the real part of all eigenvalues is non non-positive. By means of Lyapunov's stability theorem (Shil nikov 200T
proved that is globally and asymptotically stable. The extreme of each 2; approaches O: Tim;- 2; = 0 for E {1, ... .n). Therefore, each policy would converge to the advised policy through imitation.
To conclude, we show that for each agent, the advised policy is better than the policy of the agent itself, and each policy would converge to the advised policy through imitation. Thus, each agent's policy will converge to improved policy through imitation, and this will not affect the convergence of the vanilla RL algorithm.
E NETWORK STRUCTURE AND PARAMETER SETTINGS
Network Structure Here we provide the network structure for PPO and MAOPT-SRO shown in Figure 16 (a) and (b) respectively.
Parameter Settings
Here provide the hyperparameters for MAOPT, DVM well two baselines, PPO and MAD- DPG shown in Table 5 respectivey.
17

---

Under review as a conference paper at ICLR 2021
Actor
Softmax
Critic
| hiddenunits
hidden units
64 hidden units
64 hidden units
64 hidden units
hidden units
of
of
(a) PPO
Input Reconstruction
Dense
Reward Weights
SR embedding
Wix
64 hidden units
32 hidden units
m m3
Option value function
(o',aa
m (ot)
Ca (12)
Qu (mn)
((o',aa))
$(0 wz) :
Option termination probabilities
((05,an))
(b) SRO
Figure 16: Network structures.
18

---

Under review as a conference paper at ICLR 2021
Table 5: Hyperparameters for all methods based on PPO.
Hyperparameter
Value
Learning rate
3e
Length of trajectory segment T
32
Gradient norm clip À
0.2
Optimizer
Adam
Batch size B of the option advisor
32
Replay memory size
1e5
Learning rate
1e -
Action-selector
6-greedy
6-start
1.0
e-finish anneal time
0.05
5e4 step
target-updaat--nttrvall
1000
distillation-interaal for DVM
2e5 step
distillation-iteraiion for DVM
2048 step
Table 6: Hyperparameters for all methods based on MADDPG.
Hyperparameter
Value
Learning rate
1e
Batch size
1024
Optimizer
Adam
Batch size B of the option advisor
32
Replay memory size
1e5
Learning rate
le - 5
Action-selector e-start
e-greedy
1.0
e-finish
0.05
anneal time
5e4 step
turget-update-nntee
1000
19


Reviews:
[{"review_id": "sgwbozLk09N", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---------------------\nPost-rebuttal\n---------------------\nI am improving my grade a bit. I recommend the authors to dedicate some time further improving the paper clarity, especially in the matters related to my review and the other reviewers'\n\n---------------------\n\nThe authors propose a transfer learning framework for transferring knowledge in multiagent tasks. Their method consists of learning a centralized option-based advisor, that will extract advice to provide to all agents in the system based on the options learned. This advice will be used to compute an auxiliary cost function that will ideally guide all agents towards learning faster.\n\n---------------------\nPros\n+ The proposal is novel as far as I can tell\n+ The experimental evaluation shows very good performance in all evaluated scenarios\n+ Timely and relevant research\n\n----------------------------\nCons\n- The manuscript needs polishing. Some English review is needed (paper is understandable but has many small mistakes)\n- The assumptions of the method are not very clearly discussed.\n- The novel parts of the paper are kind of intertwined with equations and ideas proposed in other works, which makes it harder to quickly see what was proposed in this specific work.\n- I feel like LeCTR (Omidshafiei, 2019), cited in the paper, should have been included in the experimental evaluation\n\n----------------------\n Suggestions\n----------------------\n- I am missing a clear and comprehensive discussion about the assumptions regarding the proposal. Specifically, which communication channels should be available to the method? Does the option advisor need to be able to communicate with all agents all the time? Anything changes regarding the needed infrastructure if we are using GOA, LOA, or SRO?\n\n- Section 3 presents newly proposed concepts together with equations that have been introduced in the option-critic paper, for example. Section 3 should explain only new concepts and equations, making it clear what is a new proposal and what has been proposed before (that ideally should be described in the background section). Also, this section is a little verbose, you can make it more objective to improve readability.\n\n- Why is DVM worse than the regular PPO in Figure 4 (a). It seems to me the algorithm is incorrectly configured or that the algorithm performs poorly in this specific scenario - in which case you should also evaluate another algorithm that performs better in this scenario. In general, I also think that you should add LeCTR at least in the evaluation in the particle environment.\n\n- Add a table at the beginning of section 3 outlining the scenarios in which SRO, LOA, and GOA perform best, and which aspects affect in the algorithm choice.\n-----------\nminor\n----------\n- don't let titles without text (3. Approach)\n- the paper needs an English review\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mAAEj-TfUql", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an option-based framework for multiple agents to share knowledge with each other in the same MARL task. For scalability and robustness, two variants of the framework are designed, including 1) a global option advisor, which has the access to the global information of the environment; 2) local option advisor combined with successor representation option to enable more accurate option-value estimation. Experimental results demonstrate the proposed method is able to improve the performance of existing deep RL approaches for multiagent domains.\n\n\nPros:\n\n1. This work models the policy transfer among agents as an option learning problem. This is an interesting idea for knowledge transfer in MARL tasks.\n\n2. Comparing to previous teacher-student framework and policy distillation framework, a key feature is that the proposed framework is more adaptive (in what sense?) and applicable to scenarios consisting of more than two agents. \n\n\nCons: \n\n1. Some details in are not clear (see Questions and Comments)\n\n2. This framework based on centralized information, which may not always be available for multiagent problems, especially for noncooperative settings.\n\nQuestions and Comments:\nThe authors mention that “selecting a joint option means each advice given to each agent begins and ends simultaneously”. Does this mean the options (advice given) all have the same length? Especially for MAOPT-GOA, the termination function is over joint options. But this seems to be a very restricted assumption, given there are a lot of uncertainty in real world and different sub tasks may take different amount of time. \n\nAdvice is given in option level, level action selection are based on the intra-option policy, which is learned through imitation learning, does that mean the student need to collected the demonstration data from an expert (teacher?), how do you obtain the expert policy?\n\nAre the agents homogenous? Meaning do they share the same option sets? How many options are available for each agent for the tasks in the experiment?\n\nIt is mentioned that “each option \\omega^i contains an intra-option policy corresponding to an agent’s policy \\pi^i”. It is unclear why the authors make this statement. Since in general \\pi^i may not corresponds to the intra-option policy of \\omega^i, unless there is only one option.\n\n=======================After the rebuttal ========================\nAfter rebuttal, I think the author have addressed most my questions. However, I agree with other reviewers’ opinions that the paper need further polishing and clarification, especially regarding the cooperative settings of the problems (homogeneous team, number of options etc as admitted by the authors) and associated theoretical and practical issues. Given the novelty of the idea and the current status of paper, I maintain my score.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ul2KYY3NuSG", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe authors propose a MARL solution based on the idea of \"Local Option Advisor\" and \"Option-based Policy Transfer\". The a\n\n\nStrenghts\n\n- This is timely area, in which it is good to see progress and alternative solutions to the existing ones.\n\n- The problem of transfer learning might enable to apply MARL in different environments and application scenarios.\n\nWeaknesses\n\n- The significance of the proposed solution is unclear given the fact that the additional complexity of the approach proposed by the authors is not fully justified. Why is basic transfer learning not sufficient?\n\n- The evaluation of the method is based on experiments, but the plots provided by the authors do not show definite evidence of the superior performance (the confidence interval are apparently overalapping).\n\n- The aspects related to transfer learning are not fully evaluated in the paper.\n\nIn general, the actual contribution of this work with respect to the state of the art in MARL/transfer learning is not completely clear. The authors propose a rather complex solution, but its actual motivation is not apparent. In particular, it is unclear to see which particular design needs they are trying to address with these additional mechanisms. In other words, the advantages in adopting these mechanisms are not proven by the authors in my opinion.\n\nMoreover, the method is evaluated experimentally, but its actual superiority is unclear. In fact, if you examine Figures 4 and 5, you see that the performance are comparable (the confidence appear to overlap).\n\nQuestions\n\n- What are the situations where the proposed solution provides a real advantage? There is also a question about how the proposed solution generalizes to other problems: are there any underlying assumptions of the proposed additional mechanisms to a basic transfer learning model?\n\n- Is there any specific theoretical evidence that shows that the addition of these mechanisms might lead to improved performance?\n\n- What is the tradeoff of the proposed solution with respect to the added complexity? Is this complexity justified? Are there situations/problems that can be tackled only through these added mechanisms?\n\n- Could you addditional explanations to the results in Table 1? How should we interpret them.\n\n- Figure 6 is not completely \"readable\": how should we interpret the results presented in that figure?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8OhNug8FXy8", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the multi-agent Reinforcement Learning setting, and proposes a scalable method for the agents to help each other learn, by transferring their policies to each other. The main idea presented of the paper is that an agent i may optimize its policy $\\pi_i$ using a combined loss, that not only optimizes the agent returns obtained by the agent, but also imitates the policy of some other agent j. Which agent should imitate what agent is also learned, using an approach inspired from the Options framework, and Option learning algorithms such as the Option-Critic architecture. Several variants of the proposed approach are introduced, depending on how much information the agents can communicate in a particular setting.\n\nThis paper is very interesting, as it proposes a method that combines two domains of Reinforcement Learning, namely Multi-Agent systems and Options, that are not often combined. The empirical results are encouraging, and the method seems to scale to a significant amount of agents.\n\nHowever, the paper is, in my opinion, very difficult to fully understand. I'm an well versed in Options, and knowledgable in multi-agent systems, but the difficulty of understanding the paper does not come from a lack of background information on these two domains (successor features are also very well presented). In my opinion, small but important details are missing. For instance, a precise and explicit description of what the agents exchange as information (policy parameters, actions?), and when, would have helped. Figures 1 and 2, and most of the formulas of the paper, focus on the mathematical aspect of the proposed architecture. They allow to see that all the pieces of information fit together into sound formulas. But it is very difficult to see how to implement the algorithm in practice. I would suggest that the authors add pseudocode sections, for the acting loops of the agents, their training loops, how agents decide which agent to imitate, and how the option and termination functions are learned.\n\nBecause the ideas proposed in the paper are interesting, and seem original, I would recommend accepting this paper, but only if the authors manage to make the paper clearer for a first reader.\n\nAuthor response: discussion with the authors allowed to clarify what the agent see and how they exchange information. I still find the paper a bit difficult to understand, most probably due to its novelty, but I consider that the current version is acceptable. As such, I recommend accepting this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8AgE93PUZjW", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper proposes a new option-based policy transfer framework for multi-agent reinforcement learning (MARL) called MAOPT. By framing multi-agent transfer as an option learning problem, MAOPT methods are able to learn when to give advice to agents and when to stop it. Authors provide a version of MOAPT for fully cooperative setting based on global state and reward, as well as two versions for mixed settings based on local states and per-agent rewards. The paper presents experimental results on two environments that show performance gains over existing RL methods.\n\n##########################################################################\n\nReasons for score:\n\nI vote for accepting the paper. I like the idea of utilising the option-based transfer learning approach applied in MARL. My concerns regarding the paper are mainly about the experiments. Hopefully, these would be addressed during the rebuttal period.\n\n##########################################################################\n\nPros:\n\n1. The proposed option-based method offers a new prospective for policy transfer in MARL. The design of the MAOPT is interesting and reasonable for many MARL problems.\n2. The authors propose two different variants of the MAOPT that correspond to different MARL scenarios, namely fully-cooperative and mixed settings. The mixed setting is also studied by two different models, namely MAOPT-LOA and MAOPT-SRO, the latter of which is specifically designed to handle the experience inconsistencies of agents.\n2. This paper provides results of several experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework.\n\n##########################################################################\n\nCons:\n\n1. It is interesting to see how the methods scale with the increase of the number of agents. To this end, it would have been great to see experimental results including more than a handful of agents. Even if the methods don't scale outstandingly well, it would be worth adding a discussing regarding this.\n2. The authors mention two existing transfer learning methods, namely DVM and LTCR, but compare their method only with the former. It isn't clear to me why the authors haven't used LTCR as a baseline.\n\n##########################################################################\n\nQuestions during rebuttal period:\n\nPlease address and clarify the cons above.\n\nAlso, I'm not really sure about this statement in the introduction: \"in a multiagent system (MAS), the exploration strategy of each agent is different...\". For instance, there is a MARL method called MAVEN that coordinates the exploration of all agents by conditioning their behaviour on joint variable from a latent space (http://papers.nips.cc/paper/8978-maven-multi-agent-variational-exploration). Maybe you could rephrase your statement to make it a bit more clear?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method on multi-agent options-based policy transfer where agents help each other learn by exchanging policies.

The core idea behind the paper is novel, as it addresses a new and emerging topic of social learning, and of interest to ICLR community. The authors significantly improved the paper with additional experiments and theoretical analysis during the rebuttal process, resulting in a compelling case for the significance of the method. 

Unfortunately, the paper requires addressing the clarity, and a careful proofreading pass, making it unsuitable to ICLR in its current form,

Decision: Reject