Celebrating Diversity in Shared Multi-Agent Reinforcement Learning

1 Introduction . Cooperative multi-agent reinforcement learning ( MARL ) has drawn increasing interest in recent years , which provides a promise for solving many real-world challenging problems , such as sensor networks [ 1 ] , traffic management [ 2 ] , and coordination of robot swarms [ 3 ] . However , learning effective policies for such complex multi-agent systems remains challenging . One central problem is that the joint action-observation space grows exponentially with the number of agents , which imposes high demand on the scalability of learning algorithms . To address this scalability challenge , policy decentralization with shared parameters ( PDSP ) is widely used , where agents share their neural network weights . Parameter sharing significantly improves learning efficiency because it dramatically reduces the total number of policy parameters , while experiences and gradients of one agent can be used to train others . Enjoying these advantages , many advanced deep MARL approaches adopt the PDSP paradigm , including value-based methods [ 4–8 ] , policy gradients [ 9–13 ] and communication learning algorithms [ 14 , 15 ] . These approaches achieve state-of-the-art performance on tasks such as StarCraft II micromanagement [ 16 ] . While parameter sharing has been proven to accelerate training [ 17 ] , its drawbacks are also apparent in complex tasks . These tasks typically require substantial exploration and diversified strategies among agents . When parameters are shared , agents tend to acquire homogeneous behaviors because they typically adopt similar actions under similar observations , preventing efficient exploration and the emergence of sophisticated cooperative policies . This tendency becomes particularly problematic * Equal advising †Videos are available at https : //sites.google.com/view/celebrate-diversity-shared with codes . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for many challenging multi-agent coordination tasks , hindering deep MARL from broader applications . For example , the unsatisfactory performance of state-of-the-art MARL algorithms on Google Research Football ( Fig . 1 , and [ 18 ] ) highlights an urgent demand for diverse behaviors . Notably , sacrificing the merits of parameter sharing for diversity is also unfavorable . Like humans , sharing necessary experience or understanding of tasks can broadly accelerate cooperation learning . Without parameter sharing , agents search in a much larger parameter space , which may be wasteful because they do not need to behave differently all the time . Therefore , the question is how to adaptively trade-off diversity and sharing . In this paper , we solve this dilemma by proposing several structural and learning novelties . To encourage diversity , we propose a novel information-theoretical objective to maximize the mutual information between agents ’ identi- ties and trajectories . This objective enables each agent to distinguish themselves from others and thus involves the contribution of all agents . Accordingly , we derive an intrinsic reward for motivating diversity and optimize it with the global environmental reward by learning the total Q-function as a combination of individual Q-functions . Structurally , we further decompose individual Q-functions as the sum of shared and non-shared local Q-functions for sharing experiences while maintaining representation diversity . We hope agents can use and expand shared knowledge whenever possible . Thus we introduce L1 regularization on each non-shared Q-function , encouraging agents to share and be diverse when necessary on several critical actions . Combining these novelties achieves a dynamic balance between diversity and homogeneity , efficiently catalyzing adaptive and sophisticated cooperation . We benchmark our approach on Google Research Football ( GRF ) [ 18 ] , and StarCraft II micromanagement tasks ( SMAC ) [ 16 ] . The extraordinary performance of our approach on challenging benchmarking tasks shows that our approach achieve significantly higher coordination capacity than baselines while using diversity as a catalyst for more robust and talent policies . To our best knowledge , our approach achieves state-of-the-art performance on SMAC super hard maps and challenging GRF multi-agent tasks like academy_3_vs_1_with_keeper , academy_counterattack_hard , and a full-field scenario 3_vs_1_with_keeper ( full field ) . 2 Background . A fully cooperative multi-agent task can be formulated as a Dec-POMDP [ 19 ] , which is defined as a tuple G = 〈N , S , A , P , R , O , Ω , n , γ〉 , where N is a finite set of n agents , s ∈ S is the true state of the environment , A is the set of actions , and γ ∈ [ 0 , 1 ) is a discount factor . At each time step , each agent i ∈ N receives his own observation oi ∈ Ω according to the observation function O ( s , i ) , and selects an action ai ∈ A , which results in a joint action vector a . The environment then transitions to a new state s′ based on the transition function P ( s′|s , a ) , and inducing a global reward r = R ( s , a ) shared by all the agents . Each agent has its own action-observation history τi ∈ Ti . = ( Ωi×A ) ∗ . Due to partial observability , each agent conditions its policy πi ( ai|τi ) on τi . The joint policy π induces the joint action-value function Qπtot ( s , a ) = Es0 : ∞ , a0 : ∞ [ ∑∞ t=0 γ trt | s0 = s , a0 = a , π ] . 2.1 Centralized Training with Decentralized Execution . Our method adopts the framework of centralized training with decentralized execution ( CTDE ) [ 9 , 20 , 4 , 5 , 21 , 22 , 6 , 11 ] . This framework tackles the exponentially growing joint action space by decentralizing the control policies while adopting centralized training to learn cooperation . Agents learn in a centralized manner with access to global information but execute based on their local action-observation history . One promising approach to implement the CTDE framework is value function factorization . The IGM ( individual-global-max ) principle [ 21 ] guarantees the consistency between the local and global greedy actions . When IGM is satisfied , agents can obtain the optimal global action by simply choosing the local greedy action that maximizes each agent ’ s individual utility function Qi . Some algorithms have successfully used the IGM principle [ 5 , 6 , 23 ] to push forward the progress of MARL . 3 Method . In this section , we present a novel diversity-driven MARL framework ( Fig . 2 ) that balances each agent ’ s individuality with group coordination , which is a general approach that can be combined with existing CDTE value factorization methods . 3.1 Identity-Aware Diversity We first introduce how to encourage behavioral diversity by designing intrinsic motivations . Intuitively , to encourage the specialty of individual trajectories , agents need to behave differently to highlight themselves from others , taking different actions and visiting different local observations . To achieve this goal , we use an information-theoretic objective for maximizing the mutual information between individual trajectory and agents ’ identity : Iπ ( τT ; id ) = H ( τT ) −H ( τT |id ) = Eid , τT∼π [ log p ( τT |id ) p ( τT ) ] , ( 1 ) where τT and id is the random variable for agent ’ s local trajectory and identity , respectively . π is the joint policy . To optimize Eq . 1 , we expand p ( τT ) as p ( o0 ) ∏T−1 t=0 p ( at|τt ) p ( ot+1|τt , at ) , and p ( τT |id ) as p ( o0|id ) ∏T−1 t=0 p ( at|τt , id ) p ( ot+1|τt , at , id ) . Therefore , the mutual information can be written as : Iπ ( τT ; id ) = Eid , τ [ log p ( o0|id ) p ( o0 ) ︸ ︷︷ ︸ 1© + T−1∑ t=0 log p ( at|τt , id ) p ( at|τt ) ︸ ︷︷ ︸ 2© + T−1∑ t=0 log p ( ot+1|τt , at , id ) p ( ot+1|τt , at ) ] ︸ ︷︷ ︸ 3© . ( 2 ) Term 1© is determined by the environment , and we can ignore it when optimizing the mutual information . The second term quantifies the information gain about agent ’ s action selection when the identity is given , which measures action-aware diversity as I ( a ; id|τ ) . However , p ( at|τt , id ) is typically the distribution induced by -greedy , which only distinguishes the action with the highest possibility . Therefore , directly optimizing this term conceals most information about the local Qfunctions . To solve this problem , we use the Boltzmann softmax distribution of local Q values to replace p ( at|τt , id ) , which forms a lower bound of term 2© : Eid , τ [ log p ( at|τt , id ) p ( at|τt ) ] ≥ Eid , τ [ log SoftMax ( 1αQ ( at|τt , id ) ) p ( at|τt ) ] . ( 3 ) The inequity holds because the KL divergence DKL ( p ( ·|τt , id ) ‖SoftMax ( 1αQ ( ·|τt , id ) ) ) is nonnegative . We maximize this lower bound to optimize Term 2© . Inspired by variational inference approaches [ 24 ] , we derive and optimize a tractable lower bound for Term 3© at each timestep by introducing a variational posterior estimator qφ parameterized by φ : Eid , τ [ log p ( ot+1|τt , at , id ) p ( ot+1|τt , at ) ] ≥ Eid , τ [ log qφ ( ot+1|τt , at , id ) p ( ot+1|τt , at ) ] , ( 4 ) Similar to the second term , the inequality holds because for any qφ , the KL divergence DKL ( p ( ·|τt , at , id ) ‖qφ ( ·|τt , at , id ) ) is non-negative . Intuitively , optimizing Eq . 4 encourages agents to have diverse observations that are distinguishable by agents ’ identification and thus measures observation-aware diversity as I ( o′ ; id|τ , a ) . To tighten the this lower bound , we minimize the KL divergence with respect to the parameters φ . The gradient for updating φ is : ∇φL ( φ ) = ∇φEτ , a , id [ DKL ( p ( ·|τ , a , id ) ‖qφ ( ·|τ , a , id ) ) ] = ∇φEτ , a , id , o′ [ log p ( o′|τ , a , id ) qφ ( o′|τ , a , id ) ] = −Eτ , a , id , o′ [ ∇φ log qφ ( o′|τ , a , id ) ] . ( 5 ) Based on the lower bounds shown in Eq . 3 and Eq . 4 , we introduce intrinsic rewards to optimise the information-theoretic objective ( Eq . 1 ) for encouraging diverse behaviors : rI = Eid [ β2DKL ( SoftMax ( β1Q ( ·|τt , id ) ) ||p ( ·|τt ) ) +β1 log qφ ( ot+1|τt , at , id ) − log p ( ot+1|τt , at ) ] . ( 6 ) We introduce two scaling factors β1 , β2 ≥ 0 when calculating intrinsic rewards . When β1 is 0 , we only optimize the entropy term H ( τT ) in the mutual information objective ( Eq . 1 ) . β2 is used to adjust the importance of policy diversity compared with transition diversity . In Appendix A , we discuss and compare two different approaches for estimating p ( at|τt ) and p ( ot+1|τt , at ) .

Summary: This paper focuses on the centralized training with decentralized execution (CTDE) setting within cooperative multi agent reinforcement learning (MARL). The authors observe that in some cases training with a joint Q function can lead to myopic agents that do not effectively coordinate with one another. The authors introduce CDS, a new method that factorizes the Q function into shared and individual components, with an additional loss term maximizing the mutual information between the individual Q-function and the agent ID. They demonstrate the effectiveness of the approach on a toy problem and then outperform a set of strong baselines on two challenging benchmarks. Overall the work seems like a solid contribution to an active area of research.   I vote to weak accept the paper, with a possible increase if my questions below are answered in a satisfactory manner. The only reason why this paper is not an accept is a general impression that the authors prioritized SoTA over transparency. There is no discussion of how hyperparameters were tuned, very limited discussion of limitations and no discussion of how environments were chosen. Overall it leaves the impression that maybe the method doesn't generalize to other environments, and is brittle to hyperparameters.