Celebrating Diversity in Shared Multi-Agent Reinforcement Learning

1 Introduction . Cooperative multi-agent reinforcement learning ( MARL ) has drawn increasing interest in recent years , which provides a promise for solving many real-world challenging problems , such as sensor networks [ 1 ] , traffic management [ 2 ] , and coordination of robot swarms [ 3 ] . However , learning effective policies for such complex multi-agent systems remains challenging . One central problem is that the joint action-observation space grows exponentially with the number of agents , which imposes high demand on the scalability of learning algorithms . To address this scalability challenge , policy decentralization with shared parameters ( PDSP ) is widely used , where agents share their neural network weights . Parameter sharing significantly improves learning efficiency because it dramatically reduces the total number of policy parameters , while experiences and gradients of one agent can be used to train others . Enjoying these advantages , many advanced deep MARL approaches adopt the PDSP paradigm , including value-based methods [ 4–8 ] , policy gradients [ 9–13 ] and communication learning algorithms [ 14 , 15 ] . These approaches achieve state-of-the-art performance on tasks such as StarCraft II micromanagement [ 16 ] . While parameter sharing has been proven to accelerate training [ 17 ] , its drawbacks are also apparent in complex tasks . These tasks typically require substantial exploration and diversified strategies among agents . When parameters are shared , agents tend to acquire homogeneous behaviors because they typically adopt similar actions under similar observations , preventing efficient exploration and the emergence of sophisticated cooperative policies . This tendency becomes particularly problematic * Equal advising †Videos are available at https : //sites.google.com/view/celebrate-diversity-shared with codes . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for many challenging multi-agent coordination tasks , hindering deep MARL from broader applications . For example , the unsatisfactory performance of state-of-the-art MARL algorithms on Google Research Football ( Fig . 1 , and [ 18 ] ) highlights an urgent demand for diverse behaviors . Notably , sacrificing the merits of parameter sharing for diversity is also unfavorable . Like humans , sharing necessary experience or understanding of tasks can broadly accelerate cooperation learning . Without parameter sharing , agents search in a much larger parameter space , which may be wasteful because they do not need to behave differently all the time . Therefore , the question is how to adaptively trade-off diversity and sharing . In this paper , we solve this dilemma by proposing several structural and learning novelties . To encourage diversity , we propose a novel information-theoretical objective to maximize the mutual information between agents ’ identi- ties and trajectories . This objective enables each agent to distinguish themselves from others and thus involves the contribution of all agents . Accordingly , we derive an intrinsic reward for motivating diversity and optimize it with the global environmental reward by learning the total Q-function as a combination of individual Q-functions . Structurally , we further decompose individual Q-functions as the sum of shared and non-shared local Q-functions for sharing experiences while maintaining representation diversity . We hope agents can use and expand shared knowledge whenever possible . Thus we introduce L1 regularization on each non-shared Q-function , encouraging agents to share and be diverse when necessary on several critical actions . Combining these novelties achieves a dynamic balance between diversity and homogeneity , efficiently catalyzing adaptive and sophisticated cooperation . We benchmark our approach on Google Research Football ( GRF ) [ 18 ] , and StarCraft II micromanagement tasks ( SMAC ) [ 16 ] . The extraordinary performance of our approach on challenging benchmarking tasks shows that our approach achieve significantly higher coordination capacity than baselines while using diversity as a catalyst for more robust and talent policies . To our best knowledge , our approach achieves state-of-the-art performance on SMAC super hard maps and challenging GRF multi-agent tasks like academy_3_vs_1_with_keeper , academy_counterattack_hard , and a full-field scenario 3_vs_1_with_keeper ( full field ) . 2 Background . A fully cooperative multi-agent task can be formulated as a Dec-POMDP [ 19 ] , which is defined as a tuple G = 〈N , S , A , P , R , O , Ω , n , γ〉 , where N is a finite set of n agents , s ∈ S is the true state of the environment , A is the set of actions , and γ ∈ [ 0 , 1 ) is a discount factor . At each time step , each agent i ∈ N receives his own observation oi ∈ Ω according to the observation function O ( s , i ) , and selects an action ai ∈ A , which results in a joint action vector a . The environment then transitions to a new state s′ based on the transition function P ( s′|s , a ) , and inducing a global reward r = R ( s , a ) shared by all the agents . Each agent has its own action-observation history τi ∈ Ti . = ( Ωi×A ) ∗ . Due to partial observability , each agent conditions its policy πi ( ai|τi ) on τi . The joint policy π induces the joint action-value function Qπtot ( s , a ) = Es0 : ∞ , a0 : ∞ [ ∑∞ t=0 γ trt | s0 = s , a0 = a , π ] . 2.1 Centralized Training with Decentralized Execution . Our method adopts the framework of centralized training with decentralized execution ( CTDE ) [ 9 , 20 , 4 , 5 , 21 , 22 , 6 , 11 ] . This framework tackles the exponentially growing joint action space by decentralizing the control policies while adopting centralized training to learn cooperation . Agents learn in a centralized manner with access to global information but execute based on their local action-observation history . One promising approach to implement the CTDE framework is value function factorization . The IGM ( individual-global-max ) principle [ 21 ] guarantees the consistency between the local and global greedy actions . When IGM is satisfied , agents can obtain the optimal global action by simply choosing the local greedy action that maximizes each agent ’ s individual utility function Qi . Some algorithms have successfully used the IGM principle [ 5 , 6 , 23 ] to push forward the progress of MARL . 3 Method . In this section , we present a novel diversity-driven MARL framework ( Fig . 2 ) that balances each agent ’ s individuality with group coordination , which is a general approach that can be combined with existing CDTE value factorization methods . 3.1 Identity-Aware Diversity We first introduce how to encourage behavioral diversity by designing intrinsic motivations . Intuitively , to encourage the specialty of individual trajectories , agents need to behave differently to highlight themselves from others , taking different actions and visiting different local observations . To achieve this goal , we use an information-theoretic objective for maximizing the mutual information between individual trajectory and agents ’ identity : Iπ ( τT ; id ) = H ( τT ) −H ( τT |id ) = Eid , τT∼π [ log p ( τT |id ) p ( τT ) ] , ( 1 ) where τT and id is the random variable for agent ’ s local trajectory and identity , respectively . π is the joint policy . To optimize Eq . 1 , we expand p ( τT ) as p ( o0 ) ∏T−1 t=0 p ( at|τt ) p ( ot+1|τt , at ) , and p ( τT |id ) as p ( o0|id ) ∏T−1 t=0 p ( at|τt , id ) p ( ot+1|τt , at , id ) . Therefore , the mutual information can be written as : Iπ ( τT ; id ) = Eid , τ [ log p ( o0|id ) p ( o0 ) ︸ ︷︷ ︸ 1© + T−1∑ t=0 log p ( at|τt , id ) p ( at|τt ) ︸ ︷︷ ︸ 2© + T−1∑ t=0 log p ( ot+1|τt , at , id ) p ( ot+1|τt , at ) ] ︸ ︷︷ ︸ 3© . ( 2 ) Term 1© is determined by the environment , and we can ignore it when optimizing the mutual information . The second term quantifies the information gain about agent ’ s action selection when the identity is given , which measures action-aware diversity as I ( a ; id|τ ) . However , p ( at|τt , id ) is typically the distribution induced by -greedy , which only distinguishes the action with the highest possibility . Therefore , directly optimizing this term conceals most information about the local Qfunctions . To solve this problem , we use the Boltzmann softmax distribution of local Q values to replace p ( at|τt , id ) , which forms a lower bound of term 2© : Eid , τ [ log p ( at|τt , id ) p ( at|τt ) ] ≥ Eid , τ [ log SoftMax ( 1αQ ( at|τt , id ) ) p ( at|τt ) ] . ( 3 ) The inequity holds because the KL divergence DKL ( p ( ·|τt , id ) ‖SoftMax ( 1αQ ( ·|τt , id ) ) ) is nonnegative . We maximize this lower bound to optimize Term 2© . Inspired by variational inference approaches [ 24 ] , we derive and optimize a tractable lower bound for Term 3© at each timestep by introducing a variational posterior estimator qφ parameterized by φ : Eid , τ [ log p ( ot+1|τt , at , id ) p ( ot+1|τt , at ) ] ≥ Eid , τ [ log qφ ( ot+1|τt , at , id ) p ( ot+1|τt , at ) ] , ( 4 ) Similar to the second term , the inequality holds because for any qφ , the KL divergence DKL ( p ( ·|τt , at , id ) ‖qφ ( ·|τt , at , id ) ) is non-negative . Intuitively , optimizing Eq . 4 encourages agents to have diverse observations that are distinguishable by agents ’ identification and thus measures observation-aware diversity as I ( o′ ; id|τ , a ) . To tighten the this lower bound , we minimize the KL divergence with respect to the parameters φ . The gradient for updating φ is : ∇φL ( φ ) = ∇φEτ , a , id [ DKL ( p ( ·|τ , a , id ) ‖qφ ( ·|τ , a , id ) ) ] = ∇φEτ , a , id , o′ [ log p ( o′|τ , a , id ) qφ ( o′|τ , a , id ) ] = −Eτ , a , id , o′ [ ∇φ log qφ ( o′|τ , a , id ) ] . ( 5 ) Based on the lower bounds shown in Eq . 3 and Eq . 4 , we introduce intrinsic rewards to optimise the information-theoretic objective ( Eq . 1 ) for encouraging diverse behaviors : rI = Eid [ β2DKL ( SoftMax ( β1Q ( ·|τt , id ) ) ||p ( ·|τt ) ) +β1 log qφ ( ot+1|τt , at , id ) − log p ( ot+1|τt , at ) ] . ( 6 ) We introduce two scaling factors β1 , β2 ≥ 0 when calculating intrinsic rewards . When β1 is 0 , we only optimize the entropy term H ( τT ) in the mutual information objective ( Eq . 1 ) . β2 is used to adjust the importance of policy diversity compared with transition diversity . In Appendix A , we discuss and compare two different approaches for estimating p ( at|τt ) and p ( ot+1|τt , at ) .

Summary: The paper deals with the problem of imposing policy diversity along the different agents of a MARL problem. To do so, the paper proposes to introduce a novel objective to maximize the mutual information between the agents identities and trajectories. Additionally, the paper proposes a decoupled Q function architecture to represent shared and non-shared goals.