Transfer among Agents: An Efficient Multiagent Transfer Learning Framework

1 INTRODUCTION . Transfer Learning has shown great potential to accelerate single-agent RL via leveraging prior knowledge from past learned policies of relevant tasks ( Yin & Pan , 2017 ; Yang et al. , 2020 ) . Inspired by this , transfer learning in multiagent reinforcement learning ( MARL ) ( Claus & Boutilier , 1998 ; Hu & Wellman , 1998 ; Bu et al. , 2008 ; Hernandez-Leal et al. , 2019 ; da Silva & Costa , 2019 ) is also studied with two major directions : 1 ) transferring knowledge across different but similar MARL tasks and 2 ) transferring knowledge among multiple agents in the same MARL task . For the former , several works explicitly compute similarities between states or temporal abstractions ( Hu et al. , 2015 ; Boutsioukis et al. , 2011 ; Didi & Nitschke , 2016 ) to transfer across similar tasks with the same number of agents , or design new network structures to transfer across tasks with different numbers of agents ( Agarwal et al. , 2019 ; Wang et al. , 2020 ) . In this paper , we focus on the latter direction due to the following intuition : in a multiagent system ( MAS ) , each agent ’ s experience is different , so the states each agent encounters ( the degree of familiarity to the different regions of the whole environment ) are also different ; if we figure out some principled ways to transfer knowledge across different agents , all agents could form a big picture about the MAS even without exploring the whole space of the environment , and this will definitely facilitate more efficient MARL ( da Silva et al. , 2020 ) . Transferring knowledge among multiple agents is still investigated at an initial stage , and the assumptions and designs of some recent methods are usually simple . For example , LeCTR ( Omidshafiei et al. , 2019 ) and HMAT ( Kim et al. , 2020 ) adopted the teacher-student framework to learn to teach by assigning each agent two roles ( i.e. , the teacher and the student ) , so the agent could learn when and what to advise other agents or receive advice from other agents . However , both LeCTR and HMAT only consider two-agent scenarios . Liang & Li ( 2020 ) proposed a method under the teacher-student framework where each agent asks for advice from other agents through learning an attentional teacher selector . However , they simply used the difference of two unbounded value functions as the reward signal which may cause instability . DVM ( Wadhwania et al. , 2019 ) and LTCR Xue et al . ( 2020 ) are two proposed multiagent policy distillation methods to transfer knowledge among more than two agents . However , both methods decompose the solution into several stages in a coarse-grained manner . Moreover , they consider the distillation equally throughout the whole training process , which is counter-intuitive . A good transfer should be adaptive rather than being equally treated , e.g. , the transfer should be more frequent at the beginning of the training since agents are less knowledgeable about the environment , while decay as the training process continues because agents are familiar with the environment gradually and should focus more on their own knowledge . In this paper , we propose a novel MultiAgent Option-based Policy Transfer ( MAOPT ) framework which models the policy transfer among multiple agents as an option learning problem . In contrast to the previous teacher-student framework and policy distillation framework , MAOPT is adaptive and applicable to scenarios consisting of more than two agents . Specifically , MAOPT adaptively selects a suitable policy for each agent as the advised policy , which is used as a complementary optimization objective of each agent . MAOPT also uses the termination probability as a performance indicator to determine whether the advice should be terminated to avoid negative transfer . Furthermore , to facilitate the scalability and robustness , MAOPT contains two types : one type is MAOPT with the global option advisor ( MAOPT-GOA ) , the other type consists of MAOPT with the local option advisor ( MAOPT-LOA ) and MAOPT with the successor representation option advisor ( MAOPTSRO ) . Ideally , we can obtain the global information to estimate the option-value function , where MAOPT-GOA is used to select a joint policy set , in which each policy is advised to each agent . However , in many realistic scenarios , we can only obtain each agents ’ local experience , where we adopt MAOPT-LOA and MAOPT-SRO . Each agent ’ s experience may be inconsistent due to partial observations , which may cause the inaccuracy in option-value ’ s estimation . MAOPT-SRO is used to overcome the inconsistency in multiple agents ’ experience by decoupling the dynamics of the environment from the rewards to learn the option-value function under each agent ’ s preference . MAOPT can be easily incorporated into existing DRL approaches and experimental results show that it significantly boosts the performance of existing DRL approaches both in discrete and continuous state spaces . 2 PRELIMINARIES . Stochastic Games ( Littman , 1994 ) are a natural multiagent extension of Markov Decision Processes ( MDPs ) , which model the dynamic interactions among multiple agents . Considering the fact agents may not have access to the complete environmental information , we follow previous work ’ s settings and model the multiagent learning problems as partially observable stochastic games ( Hansen et al. , 2004 ) . A Partially Observable Stochastic Game ( POSG ) is defined as a tuple 〈N , S , A1 , · · · , An , T , R1 , · · · , Rn , O1 , · · · , On〉 , where N is the set of agents ; S is the set of states ; Ai is the set of actions available to agent i ( the joint action spaceA = A1×A2×· · ·×An ) ; T is the transition function that defines transition probabilities between global states : S×A×S → [ 0 , 1 ] ; Ri is the reward function for agent i : S × A → R and Oi is the set of observations for agent i . A policy πi : Oi × Ai → [ 0 , 1 ] specifies the probability distribution over the action space of agent i . The goal of agent i is to learn a policy πi that maximizes the expected return with a discount factor γ : J = Eπi [ ∑∞ t=0 γ trit ] . The Options Framework . Sutton et al . ( 1999 ) firstly formalized the idea of temporally extended action as an option . An option ω ∈ Ω is defined as a triple { Iω , πω , βω } in which Iω ⊂ S is an initiation state set , πω is an intra-option policy and βω : Iω → [ 0 , 1 ] is a termination function that specifies the probability an option ω terminates at state s ∈ Iω . An MDP endowed with a set of options becomes a Semi-Markov Decision Process ( Semi-MDP ) , which has a corresponding optimal option-value function over options learned using intra-option learning . The options framework considers the call-and-return option execution model , in which an agent picks an option o according to its option-value function Qω ( s , ω ) , and follows the intra-option policy πω until termination , then selects a next option and repeats the procedure . Deep Successor Representation ( DSR ) . The successor representation ( SR ) ( Dayan , 1993 ) is a basic scheme that describes the state value function by a prediction about the future occurrence of all states under a fixed policy . SR decouples the dynamics of the environment from the rewards . Given a transition ( s , a , s′ , r ) , SR is defined as the expected discounted future state occupancy : M ( s , s′ , a ) = E [ ∞∑ t=0 γt1 [ st = s ′ ] |s0 = s , a0 = a ] , ( 1 ) where 1 [ . ] is an indicator function with value of one when the argument is true and zero otherwise . Given the SR , the Q-value for selecting action a at state s can be formulated as the inner product of the SR and the immediate reward : Qπ ( s , a ) = ∑ s′∈SM ( s , s ′ , a ) R ( s′ ) . DSR ( Kulkarni et al. , 2016 ) extends SR by approximating it using neural networks . Specifically , each state s is represented by a D-dimensional feature vector φs , which is the output of the network parameterized by θ . Given φs , SR is represented as msr ( φs , a|τ ) parameterized by τ , a decoder gθ̄ ( φs ) parameterized by θ̄ outputs the input reconstruction ŝ , and the immediate reward at state s is approximated as a linear function of φs : R ( s ) ≈ φs · w , where w ∈ RD is the weight vector . In this way , the Q-value function can be approximated by putting these two parts together as : Qπ ( s , a ) ≈ msr ( φs , a|τ ) · w. The stochastic gradient descent is used to update parameters ( θ , τ , w , θ̄ ) . Specifically , the loss function of τ is : L ( τ , θ ) = E [ ( φs + γm ′ sr ( φs′ , a ′|τ ′ ) −msr ( φs , a|τ ) ) 2 ] , ( 2 ) where a′ = arg maxamsr ( φ ′ s , a ) ·w , and m′sr is the target SR network parameterized by τ ′ which follows DQN ( Mnih et al. , 2015 ) for stable training . The reward weight w is updated by minimizing the loss function : L ( w , θ ) = ( R ( s ) − φs ·w ) 2 . The parameter θ̄ is updated using an L2 loss : L ( θ̄ , θ ) = ( ŝ− s ) 2 . Thus , the loss function of DSR is the composition of the three loss functions : L ( θ , τ , w , θ̄ ) = L ( τ , θ ) + L ( w , θ ) + L ( θ̄ , θ ) . 3 MULTIAGENT OPTION-BASED POLICY TRANSFER ( MAOPT ) 3.1 FRAMEWORK OVERVIEW In this section , we describe our MAOPT framework in detail . Figure 1 illustrates the MAOPT framework which contains n agents interacting with the environment and corresponding option advisors . At each step , each agent i obtains its own observation oi , selects an action ai following its policy πi , and receives its reward ri . Each option advisor initializes the option set , and selects an option for each agent . During the training phase , the option advisor uses samples from all agents to update the option-value function and corresponding termination probabilities . Each agent is advised by an option advisor , and the advice is to exploit this advised policy through imitation , which serves as a complementary optimization objective ( each agent does not know which policy it imitates and how the extra loss function is calculated ) ∗ . The exploitation of this advised policy is terminated as the selected option terminates and then another option is selected . In this way , each agent efficiently exploits useful information from other agents and as a result , the learning process of the whole system is accelerated and improved . Note that in the following section we assume the agents using the option advisor are homogeneous , i.e. , agents share the same option set . While our MAOPT can also support the situation where each agent is initialized with different numbers of options , e.g. , each agent only needs to imitate its neighbours . To achieve this , instead of input states into the option-value network , we just input the pair of states and options to the network and output a single option-value . Our proposed MAOPT can be classified into two types in terms of the experience used during training . One type is MAOPT with the global option advisor ( MAOPT-GOA ) which has the access to the ∗We provide the theoretical analysis to show this objective ensures to converge to an improved policy and will not affect the convergence of the original RL algorithm . global information ( i.e. , ( s , ~a , r , s′ ) , where r = ∑n i=1 r i ) of the environment . Thus , MAOPT-GOA selects a joint option as the advice set given the global observation of the environment and then evaluates the performance of the selected joint option . Selecting a joint option means that each advice given to each agent begins and ends simultaneously . However , in many realistic scenarios , we can only obtain each agent ’ s local information due to the partial observation . Moreover , the degree of familiarity to the environment of each agent is different , then some agents may need to imitate their teachers for a longer time . Therefore , a more flexible way to control each advice when to terminate individually is necessary . The other type contains MAOPT with the local option advisor ( MAOPTLOA ) , and MAOPT with the successor representation option advisor ( MAOPT-SRO ) which collects each agent ’ s local experience for the update . In many cases , each agent ’ s experience is inconsistent with each other , e.g. , each agent has an individual goal to achieve or has different roles , and the rewards assigned to each agent are different . If we simply use all experiences for the update , the option-value estimation would oscillate and become inaccurate . MAOPT-SRO is used to handle the experience inconsistency by decoupling the dynamics of the environment from the rewards to learn the option-value function under each agent ’ s preference .

Summary: The paper proposes a new option-based policy transfer framework for multi-agent reinforcement learning (MARL) called MAOPT. By framing multi-agent transfer as an option learning problem, MAOPT methods are able to learn when to give advice to agents and when to stop it. Authors provide a version of MOAPT for fully cooperative setting based on global state and reward, as well as two versions for mixed settings based on local states and per-agent rewards. The paper presents experimental results on two environments that show performance gains over existing RL methods.