Role Diversity Matters: A Study of Cooperative Training Strategies for Multi-Agent RL

1 INTRODUCTION . Recently , multi-agent reinforcement learning ( MARL ) has captured people ’ s attention due to its impressive achievements in the field of super human-level intelligence in video games [ 3 , 6 , 52 , 57 ] , card games [ 7 , 25 , 44 , 59 ] , and real-world applications [ 62–64 ] . These achievements have benefited substantially from the success of single-agent reinforcement learning ( RL ) [ 14 , 15 , 30 , 42 , 43 ] and rapid progress of MARL from both the competitive and the cooperative side . On the competitive side , existing works focus on game theory among adversary agents with guaranteed policy convergence via theoretical analysis [ 5 , 17 , 27 , 29 , 56 ] . Whereas the achievements on the cooperative MARL are more based on empirical results in cooperative multi-agent system ( MAS ) [ 4 , 21 , 28 , 39 , 46 , 51 , 61 ] . One key problem of cooperative MARL is that whether one algorithm is better than another depends on the MARL tasks as showed in Fig . 1a . Current researches focus on developing algorithms on the tasks they are good at but lack the study of why the performance declines on other tasks [ 16 , 35 , 55 , 55 , 58 ] . Even adopting the state-of-the-art algorithms does not guarantee a strong performance [ 13 , 35 , 48 , 55 , 58 ] . This may due to the varying characteristic ( e.g agent ’ s attributes and goals ) of MARL tasks and scenarios , one single algorithm is not able to cover them all , which means we have to change the training strategy according to the current scenario . From this perspective , we need to find a metric to describe different MARL tasks and use this description to help determine the best strategy combination as showed in Fig . 1b . Considering that Under review as a conference paper at ICLR 2022 Role Diversity Communication Sharing Credit Assignment Task 2 Task n … Role Diversity Algorithm for Task 2 Algorithm for Task n … Parameter Sharing Task 2 Task 1 MARL Algorithm 1 Performance the main component of MAS is the agents , we propose a new definition Role in MAS to quantify the agents ’ difference and use Role Diversity to describe MAS . We then analyze how the role diversity impacts the MARL both theoretical and experimental . For theoretical analysis , we use the decomposed estimation error of the joint action-value function Qtot in MARL to discuss how role diversity impacts the policy optimization process . The experiment further verifies the theoretical analysis that the role diversity is strongly related to the model performance and can serve as a good description of a MAS . As shown in Fig . 1 , with the role diversity description of each task , we can now avoid possible bottleneck of a MARL algorithm with the combination of different parameter sharing , communication , and credit assignment strategies . With the definition of role diversity and the analysis of its impact on MARL , we can also explain the question of why the model performance varies across different tasks . Role diversity are defined from three aspects : policy-based , trajectory-based , and contribution-based in Sec . 3 which are measured by action distribution , observation overlap , and Q/state value diversity . Through theoretical analysis , we find each type of role diversity has different impact to different terms of decomposed estimation error : algorithmic error , approximation error , and statistical error ( Sec . 4 ) . We conduct comprehensive experiments covering three main topics in MARL : parameter sharing , communication , and credit assignment in Sec . 5 and provide a set of guideline on choosing MARL training strategy in Sec . 6 . The main experiments are conducted on MPE [ 28 ] and SMAC [ 39 ] benchmarks , covering a variety of scenarios with different role diversity . The impact of role diversity is evaluated on representative MARL algorithms including IQL [ 49 ] , IA2C [ 31 ] , VDN [ 48 ] , QMIX [ 35 ] , MADDPG [ 28 ] , and MAPPO [ 58 ] , covering independent learning methods , centralised policy gradient methods , and value decomposition methods . The experiment results prove that the model performance of different algorithms and training strategies is largely dependent on the role diversity . Scenarios with large policy-based role diversity prefer no parameter sharing strategy . Communication is not needed in scenarios with large trajectory-based role diversity . Learnable credit assignment modules should be avoided when training on scenarios with large contributionbased role diversity . The key contributions of this study are as follows : First , the concepts of the role and role diversity are defined to describe MARL tasks . Second , a theoretical analysis of how role diversity impacts MARL policy optimization with estimation error decomposition is built . Third , role diversity is proven to be strongly related to performance variance when choosing different training strategies , including parameter sharing , communication , and credit assignment on the MARL benchmarks . Finally , a set of guidelines for selecting a training strategy based on role diversity is provided . 2 RELATED WORK . Researches on the development of cooperative MARL algorithms are mainly in three aspects : parameter sharing , communication and credit assignment . For parameter sharing , the common approach in cooperative MARL is to fully share the model parameters among the agents [ 16 , 35 , 48 , 55 ] . In this way , the policy optimization can benefit from the shared experience buffer with samples from all different agents , providing a higher data efficiency . However , it has also been noted in recent works that parameter sharing is not always a good choice [ 8 , 34 , 50 ] . In some scenarios , a selective parameter sharing strategy , or even no parameter sharing , can significantly benefit agent performance and surpass the full parameter sharing . However , the question of why different parameter sharing strategies have different impacts on different scenarios remains open . In this study , we find that the role diversity can serve as a strong signal for selecting the parameter sharing strategy . The communication mechanism is an intrinsic part of the multi-agent system ( MAS ) framework [ 18 , 19 , 23 , 28 , 47 ] . It provides the current agent with essential information of other agents to form a better joint policy , which substantially impacts the final performance . In some cases , communication restrictions exist , which hinder us from freely choosing communication methods [ 28 , 39 ] ; in most cases , however , the communication is available and it is optional on when to communicate and how to ingest the shared information [ 19 , 45 ] . We present a comprehensive study on the relationship between role diversity and information sharing via communication mechanisms and demonstrate that role diversity determines the necessity of communication . For the credit assignment , most cooperative MARL algorithms adopt Q-learning or policy gradient as the basic policy optimization method , which is combined with an extra value decomposition module [ 16 , 35 , 48 , 55 ] or shared critic function [ 13 , 28 , 58 ] to optimize the individual policy . Some other works find that leveraging the reward signal is unnecessary ; however , optimizing the individual policy independently ( independent learning , IL ) can still get a strong joint policy [ 34 , 49 ] . It then becomes slightly difficult to decide which credit assignment method ( including IL ) is better as there is no single method in cooperative MARL that is robust and always outperforms others ( compared to PPO [ 43 ] or SAC [ 14 ] in single-agent RL ) on different tasks . In this study , we contend that role diversity is the key factor that impacts the performance of different credit assignment strategies . In the next section , we present the role definition from three aspects including policy-based , trajectory-based , and contribution-based , and propose the measurement of different role diversity types to describe a MARL task . 3 ROLE DIVERSITY . Using role to describe the characteristic of the agents in the MARL context has been proven to be effective in many recent works [ 8 , 24 , 54 , 55 ] . However , the definition of the role concept remains largely unclear . In work [ 55 ] , the role is defined as the higher-level option in the hierarchical RL framework [ 20 ] . In work [ 8 ] , the role is defined as the environmental impact similarity of a random policy . These definitions are intuitive and can not accurately describe the role difference . In this study , we attempt to define the role in a more comprehensively way from three different aspects : policy-based , trajectory-based , and contribution-based . More specific scenario-based illustration can be found in Fig . 7 and Fig . 8 . With our refined role , a strong relationship between role diversity and the MARL optimization process can be built and the performance variance can be further explained . 3.1 POLICY-BASED ROLE . In MARL , different agents output different actions based on its current status . As common sense would indicate , actions taken at the same timestep can indicate different roles [ 55 ] . However , there are many exceptions . For instance , if we have two soccer players passing a ball to each other repeatedly [ 21 ] , although the action is different at each time step , the roles of these two soccer players can be very similar from the perspective of the whole soccer game . Therefore , it is not sufficient to distinguish the role difference based on a single timestep . Instead , we contend that this difference should be defined based on a period . As this role is purely based on policy distribution π , we refer to it as a policy-based role . Specifically , we define the policy-based role as the statistics of the actions ’ frequency over a period , which is n steps backward and forward from the current timestep . Here , n is the time interval that is half the length of the total time . More details can be found in Fig . 8a , where we provide a real scenario from SMAC . Policy-based role difference can be represented as follows : raT = 1 2n+ 1 T+n∑ t=T−n πat ( 1 ) where T represents the current timestep , n is the time interval , a is the agent index , π is the policy distribution . We adopt symmetrical KL divergence to measure the distance of different policy-based 1s1m1h1M_vs_5z4m_vs_3z roles . The total role distance of A agents can be computed as follows : dpT = 1∑A−1 0 A∑ a0=0 A∑ a1=a0 ( KL ( ra0T |r a1 T ) +KL ( r a1 T |r a0 T ) ) ( 2 ) where dpT represents the policy-based role distance at timestep T , A represents the total agent number , and KL represents Kullback–Leibler divergence . We also provide a case study of how policy-based role diversity varies in Fig . 2a . From a real battle scenario ( 4m vs 3z ) taken from SMAC , we find three stages including Find a good position , Focus on enemy and Find each one ’ s best strategy . In stage 1 , agents try to find their own best location ; the role diversity is large . In stage 2 , agents focus on the same enemy target ; the policies become similar and the role diversity is decreased . In stage 3 , the formation of the agents is broken up by the enemies . Policy-based role diversity again increases as each agent is required to find its own best strategy to deal with its current situation .

Summary: This work looks at defining role diversity as a means for analyzing multi agent dynamics. It proposes three different perspectives on analyzing roles in multiagent systems. One is from a policy perspective (based on KL divergences), an trajectory perspective (how much do the agents overlap in what they observe), and a team-contribution perspective (weighing the effects of the agent to the team reward).