A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning

1 INTRODUCTION . With the success of RL in the single-agent domain ( Mnih et al . ( 2015 ) ; Lillicrap et al . ( 2015 ) ) , MARL is being actively studied and applied to real-world problems such as traffic control systems and connected self-driving cars , which can be modeled as multi-agent systems requiring coordinated control ( Li et al . ( 2019 ) ; Andriotis & Papakonstantinou ( 2019 ) ) . The simplest approach to MARL is independent learning , which trains each agent independently while treating other agents as a part of the environment . One such example is independent Q-learning ( IQL ) ( Tan ( 1993 ) ) , which is an extension of Q-learning to multi-agent setting . However , this approach suffers from the problem of non-stationarity of the environment . A common solution to this problem is to use fully-centralized critic in the framework of centralized training with decentralized execution ( CTDE ) ( OroojlooyJadid & Hajinezhad ( 2019 ) ; Rashid et al . ( 2018 ) ) . For example , MADDPG ( Lowe et al . ( 2017 ) ) uses a centralized critic to train a decentralized policy for each agent , and COMA ( Foerster et al . ( 2018 ) ) uses a common centralized critic to train all decentralized policies . However , these approaches assume that decentralized policies are independent and hence the joint policy is the product of each agent ’ s policy . Such non-correlated factorization of the joint policy limits the agents to learn coordinated behavior due to negligence of the influence of other agents ( Wen et al . ( 2019 ) ; de Witt et al . ( 2019 ) ) . Thus , learning coordinated behavior is one of the fundamental problems in MARL ( Wen et al . ( 2019 ) ; Liu et al . ( 2020 ) ) . In this paper , we introduce a new framework for MARL to learn coordinated behavior under CTDE . Our framework is based on regularizing the expected cumulative reward with mutual information among agents ’ actions induced by injecting a latent variable . The intuition behind the proposed framework is that agents can coordinate with other agents if they know what other agents will do with high probability , and the dependence between action policies can be captured by the mutual information . High mutual information among actions means low uncertainty of other agents ’ actions . Hence , by regularizing the objective of the expected cumulative reward with mutual information among agents ’ actions , we can coordinate the behaviors of agents implicitly without explicit dependence enforcement . However , the optimization problem with the proposed objective function has several difficulties since we consider decentralized policies without explicit dependence or communication in the execution phase . In addition , optimizing mutual information is difficult because of the intractable conditional distribution . We circumvent these difficulties by exploiting the property of the latent variable injected to induce mutual information , and applying variational lower bound on the mutual information . With the proposed framework , we apply policy iteration by redefining value functions to propose the VM3-AC algorithm for MARL with coordinated behavior under CTDE . 2 RELATED WORK . Learning coordinated behavior in multi-agent systems is studied extensively in the MARL community . To promote coordination , some previous works used communication among agents ( Zhang & Lesser ( 2013 ) ; Foerster et al . ( 2016 ) ; Pesce & Montana ( 2019 ) ) . For example , Foerster et al . ( 2016 ) proposed the DIAL algorithm to learn a communication protocol that enables the agents to coordinate their behaviors . Instead of relying on communication , Jaques et al . ( 2018 ) proposed social influence intrinsic reward which is related to the mutual information between actions to achieve coordination . The purpose of the social influence approach is similar to our approach and the social influence yields good performance in social dilemma environments . The difference between our algorithm and the social influence approach will be explained in detail and the effectiveness of our approach over the social influence approach will be shown in Section 6 . Wang et al . ( 2019 ) proposed an intrinsic reward capturing the influence based on the mutual information between an agent ’ s current actions/states and other agents ’ next states . In addition , they proposed an intrinsic reward based on a decision-theoretic measure . Although they used the mutual information to enhance exploration , our approach focuses on the mutual information between simultaneous actions capturing policy correlation not influence . Besides , they considered independent policies , whereas policies are correlated in our approach . Some previous works considered correlated policies instead of independent policies . For example , Liu et al . ( 2020 ) proposed explicit modeling of correlated policies for multi-agent imitation learning , and Wen et al . ( 2019 ) proposed a recursive reasoning framework for MARL to maximize the expected return by decomposing the joint policy into own policy and opponents ’ policies . Going beyond adopting correlated policies , our approach maximizes the mutual information between actions which is a measure of correlation . Our framework can be interpreted as enhancing correlated exploration by increasing the entropy of own policy while decreasing the uncertainty about other agents ’ actions . Some previous works proposed other techniques to enhance correlated exploration ( Zheng & Yue ( 2018 ) ; Mahajan et al . ( 2019 ) ) . For example , MAVEN addressed the poor exploration problem of QMIX by maximizing the mutual information between the latent variable and the observed trajectories ( Mahajan et al . ( 2019 ) ) . However , MAVEN does not consider the correlation among policies . 3 BACKGROUND . We consider a Markov Game ( Littman ( 1994 ) ) , which is an extention of Markov Decision Process ( MDP ) to multi-agent setting . An N -agent Markov game is defined by an environment state space S , action spaces forN agentsA1 , · · · , AN , a state transition probability T : S×A×S → [ 0 , 1 ] , where A = ∏N i=1Ai is the joint action space , and a reward functionR : S ×A→ R. At each time step t , Agent i executes action ait ∈ Ai based on state st ∈ S . The actions of all agents at = ( a1t , · · · , aNt ) yield next state st+1 according to T and yield shared common reward rt according toR under the assumption of fully-cooperative MARL . The discounted return is defined asRt = ∑∞ τ=t γ τrτ , where γ ∈ [ 0 , 1 ) is the discounting factor . We assume CTDE incorporating resource asymmetry between training and execution phases , widely considered in MARL ( Lowe et al . ( 2017 ) ; Iqbal & Sha ( 2018 ) ; Foerster et al . ( 2018 ) ) . Under CTDE , each agent can access all information including the environment state , observations and actions of other agents in the training phase , whereas the policy of each agent can be conditioned only on its own action-observation history τ it or observation o i t in the execution phase . For given joint policy π = ( π1 , · · · , πN ) , the goal of fully cooperative MARL is to find the optimal joint policy π∗ that maximizes the objective J ( π ) = Eπ [ R0 ] . Maximum Entropy RL The goal of maximum entropy RL is to find an optimal policy that maximizes the entropy-regularized objective function , given by J ( π ) = Eπ [ ∞∑ t=0 γt ( rt ( st , at ) + αH ( π ( ·|st ) ) ) ] ( 1 ) It is known that this objective encourages the policy to enhance exploration in the state and action spaces and helps the policy avoid converging to a local minimum . Soft actor-critic ( SAC ) , which is based on the maximum entropy RL principle , approximates soft policy iteration to the actor-critic method . SAC outperforms other deep RL algorithms in many continuous action tasks ( Haarnoja et al . ( 2018 ) ) . We can simply extend SAC to multi-agent setting in the manner of independent learning . Each agent trains its decentralized policy using decentralized critic to maximize the weighted sum of the cumulative return and the entropy of its policy . We refer to this method as Independent SAC ( I-SAC ) . Adopting the framework of CTDE , we can replace decentralized critic with centralized critic which incorporates observations and actions of all agents . We refer to this method as multi-agent soft actor-critic ( MA-SAC ) . Both I-SAC and MA-SAC are considered as baselines in the experiment section . 4 THE PROPOSED MAXIMUM MUTUAL INFORMATION FRAMEWORK . We assume that the environment is fully observable , i.e. , each agent can observe the environment state st for theoretical development in this section , and will consider partially observable environment for practical algorithm construction under CTDE in the next section . Under the proposed MMI framework , we aims to find the policy that maximizes the mutual information between actions in addition to cumulative return . Thus , the MMI-regularized objective function for joint policy π is given by J ( π ) = Eπ [ ∞∑ t=0 γt ( rt ( st , at ) + α ∑ ( i , j ) I ( πi ( ·|st ) ; πj ( ·|st ) ) ) ] ( 2 ) where ait ∼ πi ( ·|st ) and α is the temperature parameter that controls the relative importance of the mutual information against the reward . As aforementioned , we assume decentralized policies and want the decentralized policies to exhibit coordinated behavior . By regularization with mutual information in the proposed objective function ( 2 ) , the policy of each agent is implicitly encouraged to coordinate with other agents ’ policies without explicit dependency by reducing the uncertainty about other agents ’ policies . This can be seen as follows : Mutual information is expressed in terms of entropy and conditional entropy as I ( πi ( ·|st ) ; πj ( ·|st ) ) = H ( πj ( ·|st ) ) −H ( πj ( ·|st ) |πi ( ·|st ) ) . ( 3 ) If the knowledge of πi ( ·|st ) does not provide any information about πj ( ·|st ) , the conditional entropy reduces to the unconditional entropy , i.e. , H ( πj ( ·|st ) |πi ( ·|st ) ) = H ( πj ( ·|st ) ) , and the mutual information becomes zero . Maximizing mutual information is equivalent to minimizing the uncertainty about other agents ’ policies conditioned on the agent ’ s own policy , which can lead the agent to learn coordinated behavior based on the reduced uncertainty about other agents ’ policies . However , direct optimization of the objective function ( 2 ) is not easy . Fig . 1 ( a ) shows the causal diagram of the considered system model described in Section 3 in the case of two agents with decentralized policies . Since we consider the case of no explicit dependency , the two policy distributions can be expressed as π1 ( a1t |st ) and π2 ( a2t |st ) . Then , for given environment state st observed by both agents , π1 ( a1t |st ) and π2 ( a2t |st ) are conditionally independent and the mutual information I ( π1 ( ·|st ) ; π2 ( ·|st ) ) = 0 . Thus , the MMI objective ( 2 ) reduces to the standard MARL objective of only the accumulated return . In the following subsections , we present our approach to circumvent this difficulty and implement the MMI framework and its operation under CTDE .

Summary: The authors propose to include the mutual information between agents' simultaneous actions in the objective to encourage coordinated behaviour. To induce positive mutual information, the authors relax the assumption that the joint policy can be decomposed as the product of each agent's policy, independent of each other given the state, and they achieve so by introducing a latent variable that correlates agents behaviours. Since the mutual information is difficult to compute, the authors proposed to maximise a parametric lower bound. The algorithm is theoretically motivated as a policy iteration variation in its exact tabular form. But experiments are performed with neural network approximations on some environments. Numerical results show improvements over previous similar techniques.