An Efficient Transfer Learning Framework for Multiagent Reinforcement Learning

1 Introduction . Transfer Learning has achieved expressive success of accelerating single-agent Reinforcement Learning ( RL ) via leveraging prior knowledge from past learned policies of relevant tasks [ 37 , 36 ] . Inspired by this , transfer learning in Multiagent Reinforcement Learning ( MARL ) [ 6 , 17 , 4 , 16 , 7 , 8 ] is also studied with two major directions : 1 ) transferring knowledge across different but similar tasks and 2 ) transferring knowledge among multiple agents in the same task . The former usually explicitly computes similarities between tasks [ 18 , 3 , 10 ] to transfer across similar tasks with the same number of agents , or design network structures to transfer across tasks with different numbers of agents [ 1 , 33 ] . In this paper , we focus on the latter due to the following intuition : in a Multiagent System ( MAS ) , each agent ’ s experience is different , so the states each agent visits ( the familiarities to different regions of the environment ) are also different ; if we figure out some principled ways to transfer knowledge across different agents , all agents could form a big picture even without exploring the whole space of the environment , which will facilitate more efficient MARL . ∗Equal contribution . † Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In fact , the latter direction is still investigated at an initial stage , and the assumptions and designs of some recent methods are usually simple . For example , LeCTR [ 27 ] and HMAT [ 19 ] adopt the teacher-student framework to enable each agent to learn when to advise other agents or receive advice from other agents . However , they only consider a two-agent scenario . Later , PAT [ 22 ] extends this idea to scenarios with more than two agents , and enables each agent to learn from other agents through an attentional teacher selector . However , it simply uses the difference of two unbounded value functions as the student reward which may cause instability . DVM [ 32 ] and LTCR [ 35 ] are two methods to transfer knowledge among multiple agents through policy distillation . However , they simply decompose the training process into two stages ( i.e. , the learning phase and the transfer phase ) by turns , which is a coarse-grained manner . Moreover , they consider the equal significance of knowledge transfer throughout the whole training process , which is counter-intuitive . A good transfer should be adaptive rather than being equally treated , e.g. , the transfer should be more frequent at the beginning of the training since agents are less knowledgeable about the environment , while decay as the training process continues because agents are familiar with the environment gradually and should focus more on their own experiences . In this paper , we propose a novel Multiagent Policy Transfer Framework ( MAPTF ) which models the policy transfer among multiple agents as the option learning problem . In contrast to previous teacherstudent and policy distillation frameworks , MAPTF is adaptive and applicable to scenarios consisting of more than two agents . Specifically , MAPTF adaptively selects a suitable policy for each agent to exploit , which is imitated by an agent as a complementary optimization objective . MAPTF also uses the termination probability as a performance indicator to determine whether the exploitation should be terminated to avoid negative transfer . Furthermore , due to partial observability of the environment , the update of the option-value function is based on all agent ’ s local experience . However , in this setting , each agent ’ s experience may be inconsistent , which could cause the option-value estimation to oscillate and become inaccurate . A novel option learning algorithm , the Successor Representation Option ( SRO ) learning is used to overcome this inconsistency by decoupling environment dynamics from rewards to learn the option-value function under each agent ’ s preference . MAPTF can be easily incorporated into existing Deep RL and MARL approaches . Our simulations show it significantly boosts the performance of existing approaches both in discrete and continuous state spaces . 2 Preliminaries . Partially Observable Stochastic Games . Stochastic Games [ 24 ] are a natural multiagent extension of Markov Decision Processes ( MDPs ) , which model the dynamic interactions among multiple agents . Considering the fact agents may not have access to the complete environmental information , we follow previous work ’ s settings and model the multiagent learning problems as partially observable stochastic games [ 13 ] . A Partially Observable Stochastic Game ( POSG ) is defined as a tuple 〈N , S , A1 , · · · , An , T , R1 , · · · , Rn , O1 , · · · , On〉 , whereN is the set of agents ; S is the set of states ; Ai is the set of actions available to agent i ( the joint action space A = A1 ×A2 × · · · × An ) ; T is the transition function that defines transition probabilities between global states : S ×A×S → [ 0 , 1 ] ; Ri is the reward function for agent i : S × A → R and Oi is the set of observations for agent i . A policy πi : Oi × Ai → [ 0 , 1 ] specifies the probability distribution over the action space of agent i . The goal of agent i is to learn a policy πi that maximizes the expected return with a discount factor γ : J = Eπi [ ∑∞ t=0 γ trit ] . The Options Framework . Sutton et al . [ 30 ] firstly formalized the idea of temporally extended action as an option . An option ω ∈ Ω is defined as a triple { Iω , πω , βω } in which Iω ⊂ S is an initiation state set , πω is an intra-option policy and βω : Iω → [ 0 , 1 ] is a termination function that specifies the probability an option ω terminates at state s ∈ Iω . An MDP endowed with a set of options becomes a Semi-MDP , which has a corresponding optimal option-value function over options learned using intra-option learning . The options framework considers the call-and-return option execution model , in which an agent picks an option ω according to its option-value function Qω ( s , ω ) , and follows the intra-option policy πω until termination , then selects a next option and repeats the procedure . Deep Successor Representation ( DSR ) . The successor representation ( SR ) [ 9 ] is a basic scheme that describes the state value function by a prediction about the future occurrence of all states under a fixed policy . SR decouples the dynamics of the environment from the rewards . Given a transition ( s , a , s′ , r ) , SR is defined as the expected discounted future state occupancy : M ( s , s′ , a ) = E [ ∞∑ t=0 γt1 [ st = s ′ ] |s0 = s , a0 = a ] , ( 1 ) where 1 [ . ] is an indicator function with value of one when the argument is true and zero otherwise . Given the SR , the Q-value for selecting action a at state s can be formulated as the inner product of the SR and the immediate reward : Qπ ( s , a ) = ∑ s′∈SM ( s , s ′ , a ) R ( s′ ) . DSR [ 21 ] extends SR by approximating it using neural networks . Specifically , each state s is represented by a feature vector φs , which is the output of the network parameterized by θ . Given φs , SR is represented as msr ( φs , a|τ ) parameterized by τ , a decoder gθ̄ ( φs ) parameterized by θ̄ outputs the input reconstruction ŝ , and the immediate reward at state s is approximated as a linear function of φs : R ( s ) ≈ φs ·w , where w ∈ RD is the weight vector . In this way , the Q-value function can be approximated by putting these two parts together as : Qπ ( s , a ) ≈ msr ( φs , a|τ ) ·w . The stochastic gradient descent is used to update parameters ( θ , τ , w , θ̄ ) . Specifically , the loss function of τ is : L ( τ , θ ) = E [ ( φs + γm ′ sr ( φs′ , a ′|τ ′ ) −msr ( φs , a|τ ) ) 2 ] , ( 2 ) where a′ = argmaxamsr ( φ ′ s , a ) ·w , and m′sr is the target SR network parameterized by τ ′ which follows DQN [ 26 ] for stable training . The reward weight w is updated by minimizing the loss function : L ( w , θ ) = ( R ( s ) − φs ·w ) 2 . The parameter θ̄ is updated using an L2 loss : L ( θ̄ , θ ) = ( ŝ− s ) 2 . Thus , the loss function of DSR is the composition of the three loss functions : L ( θ , τ , w , θ̄ ) = L ( τ , θ ) + L ( w , θ ) + L ( θ̄ , θ ) . 3 Multiagent Policy Transfer Framework ( MAPTF ) 3.1 Framework Overview Agent𝑖𝑖 Actor Critic Critic Loss Actor Loss Termination Network Option Network Agent1 Agent2 Agent𝑛𝑛 Environment OObbss𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒐𝒊𝒊𝒊𝒊 OObbss𝒐𝒐𝒐𝒐 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 Option Module Replay Buffer Reward 𝒓𝒓𝒊𝒊 Obs 𝒐𝒐𝒊𝒊 Action 𝒂𝒂𝒊𝒊 𝑎𝑎𝑖𝑖 𝑜𝑜𝑖𝑖 𝑳𝑳𝒕𝒕𝒓𝒓𝒊𝒊 = 𝒇𝒇 𝒕𝒕 𝐃𝐃𝐃𝐃𝐃𝐃 ( 𝝅𝝅𝝎𝝎 , 𝝅𝝅𝒊𝒊 ) 𝑜𝑜𝑖𝑖′ 𝑟𝑟𝑖𝑖 𝒐𝒐𝒊𝒊 𝒂𝒂𝒊𝒊 𝒓𝒓𝒊𝒊 𝒐𝒐𝒊𝒊′ 𝒊𝒊 𝝎𝝎 𝜷𝜷𝝎𝝎 𝝎𝝎 𝝎𝝎𝒐𝒐𝒐𝒐𝒐𝒐 Choice 𝝎𝝎 Agent policy 𝜋𝜋𝑛𝑛 𝑜𝑜𝑖𝑖 …𝜋𝜋1 𝜋𝜋𝑗𝑗 … Figure 1 : Framework overview . In this section , we describe our MAPTF in detail . Figure 1 illustrates the MAPTF , which contains two modules , the agent ’ s module with n agents interacting with an environment , and the option module which determines which agent ’ s policy is useful for each agent . The option module first initializes the option set with n options : Ω = { ω1 , ω2 , · · · , ωn } , each ωi is a tuple { Iωi , πωi , βωi } and πωi equals to agent i ’ s policy πi . At the start of each episode , for each agent , the option module selects an option ω based on the option-value function and the termination function . Each option terminates according to its termination function and then another option is selected to repeat the process . During the training phase , the option module uses experiences from all agents to update the option-value function and corresponding termination function . Each agent will exploit the knowledge from another agent πω based on the selected option ω . This is achieved through policy imitation , which serves as a complementary optimization objective ( the option module is responsible for this and each agent does not know which policy it imitates and how the extra loss function is calculated ) . The exploitation is terminated as the selected option terminates , and then another option is selected to repeat the process . In this way , each agent efficiently exploits useful information from other agents , and as a result , the learning process of the whole system is accelerated and improved . In the following section , all agents share the same option set based on the assumption that all agents are homogeneous and each agent ’ s policy may be helpful for other agents . MAPTF would be easily established in the situation where the option module initializes different option sets for each agent , e.g. , each agent only needs to imitate a small number of agents . In this case , instead of inputting states into the option-value network and outputting a fixed number of option-values , we input each state-option pair to the network and output a single option-value [ 23 , 25 , 11 ] .

Summary: The paper proposes a multi-agent algorithm which combines options with a successor representation and imitation learning. My understanding of the algorithm is that it initializes n options (one for each agent), and a meta-policy which selects which option should be used by each agent during each “episode” (see algorithm 1). So agents can use the most relevant option, rather than relying solely on their own option. Individual options are learned with PPO. Successor Representations are used for the meta-policy (I believe) by using a shared feature representation but individual preference vectors for each agent. Finally, the options for all agents are forced to be similar to each other through an imitation learning loss, which is scaled down over the course of training.