UPDeT: Universal Multi-agent RL via Policy Decoupling with Transformers

1 INTRODUCTION . Reinforcement Learning ( RL ) provides a framework for decision-making problems in an interactive environment , with applications including robotics control ( Hester et al . ( 2010 ) ) , video gaming ( Mnih et al . ( 2015 ) ) , auto-driving ( Bojarski et al . ( 2016 ) ) , person search ( Chang et al . ( 2018 ) ) and visionlanguage navigation ( Zhu et al . ( 2020 ) ) . Cooperative multi-agent reinforcement learning ( MARL ) , a long-standing problem in the RL context , involves organizing multiple agents to achieve a goal , and is thus a key tool used to address many real-world problems , such as mastering multi-player video games ( Peng et al . ( 2017 ) ) and studying population dynamics ( Yang et al . ( 2017 ) ) . A number of methods have been proposed that exploit an action-value function to learn a multiagent model ( Sunehag et al . ( 2017 ) , Rashid et al . ( 2018 ) , Du et al . ( 2019 ) , Mahajan et al . ( 2019 ) , Hostallero et al . ( 2019 ) , Zhou et al . ( 2020 ) , Yang et al . ( 2020 ) ) . However , current methods have poor representation learning ability and fail to exploit the common structure underlying the tasks this is because they tend to treat observation from different entities in the environment as an integral part of the whole . Accordingly , they give tacit support to the assumption that neural networks are able to automatically decouple the observation to find the best mapping between the whole observation and policy . Adopting this approach means that they treat all information from other agents or different parts of the environment in the same way . The most commonly used method involves concatenating ∗Corresponding author . the observations from each entity in to a vector that is used as input ( Rashid et al . ( 2018 ) , Du et al . ( 2019 ) , Zhou et al . ( 2020 ) ) . In addition , current methods ignore the rich physical meanings behind each action . Multi-agent tasks feature a close relationship between the observation and output . If the model does not decouple the observation from the different agents , individual functions maybe misguided and impede the centralized value function . Worse yet , conventional models require the input and the output dimensions to be fixed ( Shao et al . ( 2018 ) , Wang et al . ( 2020 ) ) , which makes zero-shot transfer impossible . Thus , the application of current methods is limited in real-world applications . Our solution to these problems is to develop a multi-agent reinforcement learning ( MARL ) framework with no limitation on input or output dimension . Moreover , this model should be general enough to be applicable to any existing MARL methods . More importantly , the model should be explainable and capable of providing further improvement for both the final performance on singletask scenarios and transfer capability on multi-task scenarios . Inspired by the self-attention mechanism ( Vaswani et al . ( 2017 ) ) , we propose a transformer-based MARL framework , named Universal Policy Decoupling Transformer ( UPDeT ) . There are four key advantages of this approach : 1 ) Once trained , it can be universally deployed ; 2 ) it provide more robust representation with a policy decoupling strategy ; 3 ) it is more explainable ; 4 ) it is general enough to be applied on any MARL model . We further design a transformer-based function to handle various observation sizes by treating individual observations as ” observation-entities ” . We match the related observation-entity with action-groups by separating the action space into several action-groups with reference to the corresponding observation-entity , allowing us to get matched observation-entity — action-group pairs set . We further use a self-attention mechanism to learn the relationship between the matched observation-entity and other observation-entities . Through the use of self-attention map and the embedding of each observation-entity , UPDeT can optimize the policy at an action-group level . We refer to this strategy as Policy Decoupling . By combining the transformer and policy decoupling strategies , UPDeT significantly outperforms conventional RNNbased models . In UPDeT , there is no need to introduce any new parameters for new tasks . We also prove that it is only with decoupled policy and matched observation-entity — action-group pairs that UPDeT can learn a strong representation with high transfer capability . Finally , our proposed UPDeT can be plugged into any existing method with almost no changes to the framework architecture required , while still bringing significant improvements to the final performance , especially in hard and complex multi-agent tasks . The main contributions of this work are as follows : First , our UPDeT-based MARL framework outperforms RNN-based frameworks by a large margin in terms of final performance on state-of- the-art centralized functions . Second , our model has strong transfer capability and can handle a number of different tasks at a time . Third , our model accelerates the transfer learning speed ( total steps cost ) to make it roughly 10 times faster compared to RNN-based models in most scenarios . 2 RELATED WORK . Attention mechanisms have become an integral part of models that capture global dependencies . In particular , self-attention ( Parikh et al . ( 2016 ) ) calculates the response at a specific position in a sequence by attending to all positions within this sequence . Vaswani et al . ( 2017 ) demonstrated that machine translation models can achieve state-of-the-art results solely by using a self-attention model . Parmar et al . ( 2018 ) proposed an Image Transformer model that applies self-attention to image generation . Wang et al . ( 2018 ) formalized self-attention as a non-local operation in order to model the spatial-temporal dependencies in video sequences . In spite of this , self-attention mechanisms have not yet been fully explored in multi-agent reinforcement learning . Another line of research is multi-agent reinforcement learning ( MARL ) . Existing work in MARL focuses primarily on building a centralized function to guide the training of individual value function ( Lowe et al . ( 2017 ) , Sunehag et al . ( 2017 ) , Rashid et al . ( 2018 ) , Mahajan et al . ( 2019 ) , Hostallero et al . ( 2019 ) , Yang et al . ( 2020 ) , Zhou et al . ( 2020 ) ) . Few works have opted to form a better individual functions with strong representation and transfer capability . In standard reinforcement learning , this generalization has been fully studied ( Taylor & Stone ( 2009 ) , Ammar et al . ( 2012 ) , Parisotto et al . ( 2015 ) , Gupta et al . ( 2017 ) , Da Silva & Costa ( 2019 ) ) . While multi-agent transfer learning has been proven to be more difficult than the single-agent scenario ( Boutsioukis et al . ( 2011 ) , Shao et al . ( 2018 ) , Vinyals et al . ( 2019 ) ) . However , the transfer capability of a multi-agent system is of greater significance due to the various number of agents , observations sizes and policy distributions . To the best of our knowledge , we are the first to develop a multi-agent framework capable of handling multiple task at a time . Moreover , we provide a policy decoupling strategy to further improve the model performance and facilitate the multi-agent transfer learning , which is a significant step towards real world multi-agent applications . 3 METHOD . We begin by introducing the notations and basic task settings necessary for our approach . We then describe a transformer-based individual function and policy decoupling strategy under MARL . Finally , we introduce different temporal units and assimilate our Universal Policy Decoupling Transformer ( UPDeT ) into Dec-POMDP . 3.1 NOTATIONS AND TASK SETTINGS . Multi-agent Reinforcement Learning A cooperative multi-agent task is a decentralized partially observable Markov decision process ( Oliehoek et al . ( 2016 ) ) with a tuple G = 〈S , A , U , P , r , Z , O , n , γ〉 . Let S denote the global state of the environment , while A represents the set of n agents and U is the action space . At each time step t , agent a ∈ A ≡ { 1 , ... , n } selects an action u ∈ U , forming a joint action u ∈ U ≡ Un , which in turn causes a transition in the environment represented by the state transition function P ( s′|s , u ) : S × U × S → [ 0 , 1 ] . All agents share the same reward function r ( s , u ) : S ×U → R , while γ ∈ [ 0 , 1 ) is a discount factor . We consider a partially observable scenario in which each agent makes individual observations z ∈ Z according to the observation function O ( s , a ) : S × A → Z . Each agent has an actionobservation history that conditions a stochastic policy πt , creating the following joint action value : Qπ ( st , ut ) = Est+1 : ∞ , ut+1 : ∞ [ Rt|st , ut ] , where Rt = ∑∞ i=0γ irt+i is the discounted return . Centralized training with decentralized execution Centralized training with decentralized execution ( CTDE ) is a commonly used architecture in the MARL context . Each agent is conditioned only on its own action-observation history to make a decision using the learned policy . The centralized value function provides a centralized gradient to update the individual function based on its output . Therefore , a stronger individual value function can benefit the centralized training . 3.2 TRANSFORMER-BASED INDIVIDUAL VALUE FUNCTION . In this section , we present a mathematical formulation of our transformer-based model UPDeT . We describe the calculation of the global Q-function with self-attention mechanism . First , the observation O is embedded into a semantic embedding to handle the various observation space . For example , if an agent ai observes k other entities { oi,1 , ... , oi , k } at time step t , all observation entities are embedded via an embedding layer E as follows : eti = { E ( oti,1 ) , ... , E ( oti , k ) } . ( 1 ) Here , i is the index of the agent , i ∈ { 1 , ... , n } . Next , the value functions { Q1 , ... , Qn } for the n agents for each step are estimated as follows : qti = Qi ( h t−1 i , e t i , ut ) . ( 2 ) We introduce ht−1i , the temporal hidden state at the last time step t − 1 , since POMDP policy is highly dependent on the historical information . eti denotes the observation embedding , while u t i is the candidate action , uti ∈ U . θi is the parameter that defines Qi . Finally , the global Q-function Qπ is calculated by all individual value functions , as follows : Qπ ( st , ut ) = F ( q t 1 , .. , q t n ) ( 3 ) Fi is the credit assignment function for defined by φi for each agent ai , as utilized in Rashid et al . ( 2018 ) and Sunehag et al . ( 2017 ) . For example , in VDN , F is a sum function that can be expressed as F ( qt1 , .. , q t n ) = ∑n i=1 q t i . Implement Q-function with Self-attention Vaswani et al . ( 2017 ) adopts three matrices , K , Q , V representing a set of keys , queries and values respectively . The attention is computed as follows : Attention ( Q , K , V ) = softmax ( QKT√ dk ) V , ( 4 ) where dk is a scaling factor equal to the dimension of the key . In our method , we adopt the selfattention to learn the features and relationships from the observation entity embedding and the global temporal information . To learn the independent policy in decentralized multi-agent learning , we define Ki , Qi and Vi as the key , query and value metrics for each agent ai . We further consider the query , key and value for the same matrices Rli = Ki = Qi = Vi , where l ∈ { 1 , ... , L } is the number of layers of the transformer . Thus , we formulate our transformer as follows : R1i = { ht−1i , e t i } Qli , K l i , V l i = LFQ , K , V ( R l i ) Rl+1i = Attention ( Q l i , K l i , V l i ) . ( 5 ) where LF represents the linear functions used to compute K , Q , V. Finally we project the entity features of the last transformer layerRLi to the output space of the value functionQi . We implement the projection using a linear function P : Qi ( h t−1 i , e t i , ui ) = P ( R L i , ui ) . ( 6 )

Summary: 1. In this paper the authors proposed a transferrable framework for multi-agent RL, which enables the learned policies easily generalize to more challenging scenarios. This seems to be a good contribution to the community of multi-agent RL. It bears a potential to handle large-scale tasks with only limited training data, while also demonstrates more explanable policies.