PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration

1 INTRODUCTION . With the potential to solve complex real-world problems , Multi-Agent Reinforcement Learning ( MARL ) has attracted much attention in recent years ( Rashid et al. , 2018 ; Iqbal & Sha , 2019 ; Wang et al. , 2019b ) and been applied to many practical domains like Game AI ( Peng et al. , 2017 ) , robotics control ( Matignon et al. , 2012 ) , transportation ( Li et al. , 2019 ) . However , efficiently achieving collaboration and learning optimal policies still remains challenging in MARL ( Liu et al. , 2020 ; Wen et al. , 2019 ) . Prior efforts on improving collaboration of MARL agents mainly rely on efficient communication ( Eccles et al. , 2019 ; Das et al. , 2019 ; Kim et al. , 2019 ; 2021 ) , or resort to value function factorisation ( Sunehag et al. , 2017 ; Rashid et al. , 2018 ; Son et al. , 2019 ) . A popular paradigm of cooperative MARL is Centralized Training , Decentralized Execution ( CTDE ) ( Rashid et al. , 2018 ; Sunehag et al. , 2017 ) . During centralized training , agents are granted access to other agents ’ information and possibly the global state , while during decentralized execution , agents make decisions independently based on their individual policies . However , although global information is incorporated during centralized training of CTDE , guiding the decentralized policies of multiple agents only by reward signals is often inefficient , especially when the reward signal is stochastic or sparse . This requires additional mechanisms to facilitate multi-agent collaboration ( Roy et al. , 2019 ; Kim et al. , 2020 ) . Later , another branch of works which is complementary to the above two branches focuses on leveraging the correlation or influence among agents for better collaboration ( Jaques et al. , 2018 ; 2019 ; Xie et al. , 2020 ; Liu et al. , 2020 ; Merhej & Chetouani , 2021 ) .The intuition behind this is that if agents make decisions with the awareness of the influence or behaviors of other agents , the non-stationary problem could be mitigated , thus agents are more likely to achieve collaboration . Motivated by this , recently , several works ( Chen et al. , 2019 ; Mahajan et al. , 2019 ; Kim et al. , 2020 ) propose to maximize the correlation of agents ’ behaviors to promote collaboration . The correlation is commonly quantified by the mutual information ( MI ) of agents ’ behaviors . Signal Instructed Coordination ( SIC ) ( Chen et al. , 2019 ) incorporates a shared latent variable z ( sampled from a predefined distribution ) into agents ’ policies as a signal of collaboration . During training , the MI between latent variable z and agents ’ joint policy is maximized . By taking z as input , the correlation of each agent ’ s decision to the joint policy is enhanced . Similarly , Multi-agent Variational Exploration ( Maven ) ( Mahajan et al. , 2019 ) establishes an encoder network to encode the initial global state into the latent variable z instead of sampling from a predefined distribution . Then Maven maximizes the MI between z and the resulting trajectory of all agents . However , one common drawback of both methods is , the shared variable z required during decentralized execution violates the CTDE paradigm . This makes algorithms fail in some real-world deployment scenarios where global communication is not available . Apart from the above methods , VM3-AC ( Kim et al. , 2020 ) measures the MI between any two agents ’ policies to capture the correlation of agents ’ behaviors , where z is also introduced to guarantee the MI of any two policies is positive . Through maximizing the MI of any two agents ’ policies , VM3-AC reduces the uncertainty of agents ’ policies and thus facilitates multi-agent collaboration . However , VM3-AC also violates the CTDE paradigm due to the requirement of the shared variable z . Furthermore , VM3-AC calculates the MI between any two agents , which can be computationally infeasible with the increase of the number of agents . Another critical thing which is neglected by previous works is that , although agents with high degree of collaboration usually generate highly correlated behaviors , this does not necessarily results in high rewards . Thus , simply maximizing the MI of agents ’ behaviors can not guarantee desired collaboration . Worse still , maximizing MI exacerbates the problem : the agent can easily overfit its strategy to the behaviours of other agents ( Zhang et al. , 2019 ; Lanctot et al. , 2017 ) . An example is shown in Figure 1 where agents are easily falling into sub-optimal collaboration . In this paper , we propose a new collaboration criterion to evaluate the collaboration of multiple agents in terms of the joint policy , individual policy , and other agents ’ policies . Based on the criterion , we propose a new form of MI between the global state and joint policy to facilitate collaboration which is free of the reliance of explicit additional input as introduced in previous works and addresses the scalability issue . Moreover , to avoid falling into sub-optimal collaboration , we propose a novel framework , called Progressive Mutual Information Collaboration ( PMIC ) , which contains two main components . The first component is Dual Progressive Collaboration Buffer ( DPCB ) which includes a positive and a negative buffer to dynamically keep the superior and inferior collaboration separately . The second component is Dual Mutual Information Estimator ( DMIE ) which has two MI neural estimators to estimate the MI of global state and joint action based on DPCB . Specifically , one estimator is trained on the positive buffer to provide the lower bound of MI , the other is trained on the negative buffer to provide the upper bound of MI . By maximizing the lower bound and minimizing the upper bound of MI , the agents could progressively break the current suboptimal collaboration and learn towards better performance which promotes an efficient and stable learning process . PMIC is general and can be easily combined with existing MARL algorithms . Our experiments show that PMIC significantly accelerates existing MARL algorithms , outperforms other related algorithms on a wide range of MARL benchmarks . 2 PRELIMINARIES . We consider a fully cooperative multi-agent task where a team of agents are situated in a stochastic , partially observable environment , it can be modeled as a decentralised partially observable Markov decision process ( Dec-POMDP ) ( Oliehoek & Amato , 2016 ) , defined by a tuple ⟨I , S , U , O , T , R , γ⟩ . Here I = { 1 , ... , N } denotes the set of N agents . In Dec-POMDP , the full state of the environment st ∈ S can not be observed by agents at each time step t. Each agent i ∈ I can only observe its individual observation oit determined by observation function O ( st , i ) , each agent i uses a stochastic policy πi to choose actions uit ∼ πi ( ·|oit ) , yielding the joint action ut = { uit } Ni=1 ∈ U . After executing ut in state st , the environment transits to the next state st+1 according to transition function T ( st , ut ) and agents receive a common reward rt from R ( st , ut ) , with a discount factor γ . In cooperative MARL , the collaborative team aims to find a joint pol- icy π ( ut|st ) = ∏N a=1 πi ( u i t|oit ) that maximizes the total expected discounted cumulative reward Eπ [ ∑∞ t=0 γ trt ] where y is the discount factor . 3 PROGRESSIVE MUTUAL INFORMATION COLLABORATION 3.1 MOTIVATION In this section , we show the motivation of our work to explain why blindly maximizing correlation of agents ‘ behaviors may make agents stuck into suboptimal collaboration . Firstly , we give a motivating example shown in ( a ) of Figure 1 where agents need cooperate to catch the targets A , B and C with different sparse rewards and punishments . Capturing A is more rewarding than B and C. Agents may receive penalties for capturing different targets . We denote the behaviors generated by the joint policy of achieving a certain target concurrently as collaboration pattern . In this setting , promoting collaboration with simply maximizing the MI of agents ’ behaviors can not guarantee a better collaboration . As shown in Figure 2 , the patterns joint distribution is composed of patterns which capture targets A , B and C. This distribution is chaotic and contains various collaboration patterns . In this case , maximizing MI on any patterns of the distribution may enhance the inferior collaboration patterns , for example maximizing MI on the pattern B or C may make agents converge to a sub-optimal collaboration pattern . In fact , this is a very common situation . In complex environments , there is a wide variety of collaboration , maximizing MI could only stimulate agents towards a certain collaboration , which may be sub-optimal and in turn prevents from discovering better collaborations . This happens because , as learning preceeds , a seemly good collaboration will become sub-optimal once a better one is discovered . However , if we excessively maximize the MI using sub-optimal samples , the agents may be accelerated to fall into sub-optimal policies . Thus the motivation of our method is to solve the limitation with the maximization-minimization MI . As shown in the right of Figure 2 , we divide the distribution into the superior and inferior distributions . then we maximize the MI associated with superior collaboration patterns and minimize the MI associated with inferior collaboration patterns . In this manner , the current patterns strengthen towards superior patterns and are away from inferior patterns . Next we conduct experiments with four methods on the environment : MADDPG ( Lowe et al. , 2017 ) , SIC-MADDPG ( Chen et al. , 2019 ) , VM3-MADDPG ( Kim et al. , 2020 ) and our method where VM3 and SIC both promote collaboration by maximizing MI . Results of the count of algorithms converging to different collaboration patterns and overall performance are shown in ( b ) and ( c ) of Figure 1 . We can see that both SIC and VM3 have a greater probability of falling into the sub- optimal patterns than that into the optimal one . This demonstrates that maximizing MI only can not guarantee the optimal collaboration , it may also stuck into sub-optimal collaboration . In addition , our method outperforms other methods in both the number of effective patterns and the optimal patterns , which demonstrates that our method is more effective in facilitating collaboration while avoiding falling into sub-optimal collaboration patterns than other methods . In addition to the above problem , VM3-AC ( Kim et al. , 2020 ) has scalability issue which need to compute the MI of policies between any two agents . Besides , all previous methods ( Kim et al. , 2020 ; Chen et al. , 2019 ; Mahajan et al. , 2019 ) need additional inputs , e.g. , latent variable z in ( Chen et al. , 2019 ; Kim et al. , 2020 ) or global initial state in ( Mahajan et al. , 2019 ) , during the decentralized execution phase , which violates the rule of CTDE paradigm . To this end , we propose a new collaboration criterion as well as a novel framework to solve the above problems . Specifically , we propose a new collaboration criterion to evaluate the collaboration of multiple agents in terms of the joint policy , individual policy , and other agents ’ policies . Based on the criterion , a new way that calculates the MI of the global state and joint policy is proposed to promote collaboration which does not require an additional latent variable as input and addresses the scalability issue . To solve the problem of easily falling into sub-optimal collaboration by maximizing MI only , we then propose PMIC framework which leverages two distinct mechanisms to guide agents in a progressive manner . Finally , to solve the inefficiency of being guided only by rewards ( e.g. , stochastic or sparse ) , we use the estimated values of the new MI which are provided by DMIE as intrinsic rewards . To the best of our knowledge , we are the first to propose a combination of maximizing the lower bound of MI and minimizing the upper bound of MI to promote collaboration in MARL . In the following section , we first introduce the new collaboration criterion , then we show the details of PMIC framework , finally we give an combination of our method with MADDPG .

Summary: This paper aims to solve the collaboration among agents (that is usually called forced coordination in other literatures). The common method to enhance the correlations between agents by mutual information (MI) could lead to the sub-optimal collaboration. To address this issue, the author propose Progressive Mutual Information Collaboration (PMIC). This new MARL framework is composed of two components: (1) two replay buffers that respectively collect superior trajectories and inferior trajectories; (2) two mutual information estimators that respectively estimate the upper bound and the lower bound of MI that are trained with the data stored in the two replay buffers introduced above. These two estimators (MINE and CLUB) are from the prior works. The novelties of this work are applying these two estimators to collaboration problem for MARL and learning the mutual information that evaluates the correlation among agents constrained within these two bounds. In my view, the idea of appropriately modelling these MI bound estimators to the actual phenomenon in the MARL problem is novel. Seen from the algorithmic framework type, this algorithm add the learned MI term to the reward function that plays the role of auxiliary reward. From the perspective of optimization, this is a kind of implementation of Lagrangian method, to transform the constraints stated in Equation (2) and (3) to a unconstrained terms involved in the objective function. Similar to many prior works, the tuning of the multipliers could be a potential issue. The authors also conduct the ablation studies on these two multipliers and the results seem like it would not affect the performance in the case of Cooperation Navigation.