Context-Aware Sparse Deep Coordination Graphs

Learning sparse coordination graphs adaptive to the coordination dynamics among agents is a long-standing problem in cooperative multi-agent learning . This paper studies this problem and proposes a novel method using the variance of payoff functions to construct context-aware sparse coordination topologies . We theoretically consolidate our method by proving that the smaller the variance of payoff functions is , the less likely action selection will change after removing the corresponding edge . Moreover , we propose to learn action representations to effectively reduce the influence of payoff functions ’ estimation errors on graph construction . To empirically evaluate our method , we present the Multi-Agent COordination ( MACO ) benchmark by collecting classic coordination problems in the literature , increasing their difficulty , and classifying them into different types . We carry out a case study and experiments on the MACO and StarCraft II micromanagement benchmark to demonstrate the dynamics of sparse graph learning , the influence of graph sparseness , and the learning performance of our method . 1 INTRODUCTION . Many real-world problems involve the cooperation of multiple agents , such as unmanned aerial vehicles ( Pham et al. , 2018 ; Xu et al. , 2018 ) and sensor networks ( Stranders et al. , 2009 ) . Like in single-agent settings , learning control policies for multi-agent teams largely relies on the estimation of action-value functions , no matter in value-based ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; 2020 ) or policy-based approaches ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Wang et al. , 2021c ) . However , learning action-value functions for complex multi-agent tasks remains a major challenge . Learning individual action-value functions ( Tan , 1993 ) is scalable but suffers from learning non-stationarity because it treats other learning agents as part of its environment . Joint action-value learning ( Claus & Boutilier , 1998 ) is free from learning non-stationarity but requires access to global information that is often unavailable during execution due to partial observability and communication constraints . Factored Q-learning ( Guestrin et al. , 2002a ) combines the advantages of these two methods . Learning the global action-value function as a combination of local utilities , factored Q functions maintain learning scalability while avoiding non-stationarity . Enjoying these advantages , fully decomposed Q functions significantly contribute to the recent progress of multi-agent reinforcement learning ( Samvelyan et al. , 2019 ; Wang et al. , 2021b ) . However , when fully decomposed , local utility functions only depend on local observations and actions , which may lead to miscoordination problems in partially observable environments with stochastic transition functions ( Wang et al. , 2020 ; 2021a ) and a gametheoretical pathology called relative overgeneralization ( Panait et al. , 2006 ; Böhmer et al. , 2020 ) . Relative overgeneralization renders optimal decentralized policies unlearnable when the employed value function does not have enough representational capacity to distinguish other agents ’ effects on local utility functions . Coordination graphs ( Guestrin et al. , 2002b ) provide a promising approach to solving these problems . Using vertices to represent agents and ( hyper- ) edges to represent payoff functions defined over the joint action-observation space of the connected agents , a coordination graph expresses a higher-order value decomposition among agents . Finding actions with the maximum value in a coordination graph can be achieved by distributed constraint optimization ( DCOP ) algorithms ( Cheng , 2012 ) , which consists of multiple rounds of message passing along the edges . Recently , DCG ( Böhmer et al. , 2020 ) scales coordination graphs to large state-action spaces , shows its ability to solve the problem of relative overgeneralization , and obtains competitive results on StarCraft II micromanagement tasks . However , DCG focuses on predefined static and dense topologies , which largely lack flexibility for dynamic environments and induce intensive and inefficient message passing . The question is how to learn dynamic and sparse coordination graphs sufficient for coordinated action selection . This is a long-standing problem in multi-agent learning . Sparse cooperative Q-learning ( Kok & Vlassis , 2006 ) learns value functions for sparse coordination graphs , but the graph topology is static and predefined by prior knowledge . Zhang & Lesser ( 2013 ) propose to learn minimized dynamic coordination sets for each agent , but the computational complexity grows exponentially with the neighborhood size of an agent . Recently , Castellini et al . ( 2019 ) study the representational capability of several sparse graphs but focus on random topologies and stateless games . In this paper , we push these previous works further by proposing a novel deep method that learns context-aware sparse coordination graphs adaptive to the dynamic coordination requirements . For learning sparse coordination graphs , we propose to use the variance of pairwise payoff functions as an indicator to select edges . Sparse graphs are used when selecting greedy joint actions for execution and the update of Q-function . We provide a theoretical insight into our method by proving that the probability of greedy action selection changing after an edge is removed decreases with the variance of the corresponding payoff function . Despite the advantages of sparse topologies , they raise the concern of learning instability . To solve this problem , we further equip our method with network structures based on action representations for utility and payoff learning to reduce the influence of estimation errors on sparse topologies learning . We call the overall learning framework Context-Aware SparsE Coordination graphs ( CASEC ) . For evaluation , we present the Multi-Agent COordination ( MACO ) benchmark . This benchmark collects classic coordination problems raised in the literature of multi-agent learning , increases their difficulty , and classifies them into 6 classes . Each task in the benchmark represents a type of problem . We carry out a case study on the MACO benchmark to show that CASEC can discover the coordination dependence among agents under different situations and to analyze how the graph sparsity influences action coordination . We further show that CASEC can largely reduce the communication cost ( typically by 50 % ) and perform significantly better than dense , static graphs and several alternative methods for building sparse graphs . We then test CASEC on the StarCraft II micromanagement benchmark ( Samvelyan et al. , 2019 ) to demonstrate its scalability and effectiveness . 2 BACKGROUND . In this paper , we focus on fully cooperative multi-agent tasks that can be modelled as a DecPOMDP ( Oliehoek et al. , 2016 ) consisting of a tuple G=⟨I , S , A , P , R , Ω , O , n , γ⟩ , where I is the finite set of n agents , γ ∈ [ 0 , 1 ) is the discount factor , and s ∈ S is the true state of the environment . At each timestep , each agent i receives an observation oi ∈ Ω drawn according to the observation function O ( s , i ) and selects an action ai ∈ A . Individual actions form a joint action a ∈ An , which leads to a next state s′ according to the transition function P ( s′|s , a ) , a reward r = R ( s , a ) shared by all agents . Each agent has local action-observation history τi ∈ T ≡ ( Ω×A ) ∗ × Ω . Agents learn to collectively maximize the global return Qtot ( s , a ) = Es0 : ∞ , a0 : ∞ [ ∑∞ t=0 γ tR ( st , at ) |s0 = s , a0 = a ] . In a coordination graph ( Guestrin et al. , 2002b ) G = ⟨V , E⟩ , each vertex vi ∈ V represents an agent i , and ( hyper- ) edges in E represent coordination dependencies among agents . In this paper , we consider pairwise edges , and such a coordination graph induces a factorization of the global Q : Qtot ( τ , a ) = 1 |V| ∑ i qi ( τi , ai ) + 1 |E| ∑ { i , j } ∈E qij ( τij , aij ) , ( 1 ) where qi and qij is utility functions for individual agents and pairwise payoff functions , respectively . τij = ⟨τi , τj⟩ and aij = ⟨ai , aj⟩ is the joint action-observation history and action of agent i and j . Within a coordination graph , the greedy action selection required by Q-learning can not be completed by simply computing the maximum of individual utility and payoff functions . Instead , distributed constraint optimization ( DCOP ) ( Cheng , 2012 ) techniques can be used . Max-Sum ( Stranders et al. , 2009 ) is a popular implementation of DCOP , which finds optimal actions on a coordination graph G = ⟨V , E⟩ via multi-round message passing on a bipartite graph Gm = ⟨Va , Vq , Em⟩ . Each node i ∈ Va represents an agent , and each node g ∈ Vq represents a utility ( qi ) or payoff ( qij ) function . Edges in Em connect g with the corresponding agent node ( s ) . Message passing on this bipartite graph starts with sending messages from node i ∈ Va to node g ∈ Vq : mi→g ( ai ) = ∑ h∈Fi\g mh→i ( ai ) + cig , ( 2 ) where Fi is the set of nodes connected to node i in Vq , and cig is a normalizing factor preventing the value of messages from growing arbitrarily large . The message from node g to node i is : mg→i ( ai ) = max ag\ai q ( ag ) + ∑ h∈Vg\i mh→g ( ah ) , ( 3 ) where Vg is the set of nodes connected to node g in Va , ag= { ah|h ∈ Vg } , ag\ai= { ah|h ∈ Vg\ { i } } , and q represents utility or payoff functions conditioned on ag . After several iterations of message passing , each agent i can find its optimal action by calculating a∗i = argmaxai ∑ h∈Fi mh→i ( ai ) . A drawback of Max-Sum or other message passing methods ( e.g. , max-plus ( Pearl , 2014 ) ) is that running them for each action selection through the whole system results in intensive computation and communication among agents , which is impractical for most applications with limited computational resources and communication bandwidth . In the following sections , we discuss how to solve this problem by learning sparse coordination graphs . Previous works ( Naderializadeh et al. , 2020 ; Li et al. , 2021 ) study soft versions of fully-connected coordination graphs based on attention mechanisms . Specifically , Li et al . ( 2021 ) uses graphs whose edge weights are learned by self-attention so that agents attend to observations of other agents differently . The information is used in local actors or a centralized critic . Naderializadeh et al . ( 2020 ) learns soft full graphs in a similar way , but the graph is used to mix local utilities conditioned on local action-observation history . Different from our work , these methods do not learn pairwise payoff functions , and the learned graphs are still fully-connected . 3 LEARNING CONTEXT-AWARE SPARSE GRAPHS . In this section , we introduce our methods for learning context-aware sparse graphs . We first introduce how we construct a sparse graph for effective action selection in Sec . 3.1 . After that , we introduce our learning framework in Sec . 3.2 . Although sparse graphs can reduce communication overhead , they raise the concern of learning instability . We discuss this problem and how to alleviate it in Sec . 3.3 .

Summary: The paper introduces a technique called CASEC to train deep coordination graphs with sparse connectivity and a dynamic method to remove a fixed number of edges during inference. The latter removes edges in order of smallest variance of the corresponding payoff functions, which the authors motivate with a theoretical statement on the probability of changing the optimal action after removing an edge. CASEC is evaluated on a novel benchmark suit (MACO) for coordination dilemmas and StarCraft II micromanagement tasks. The proposed methods outperforms the baseline (DCG) in 2 out of 6 MACO tasks and shows in particular a clear advantage of the used action encoding method in StarCraft II.