Context-Aware Sparse Deep Coordination Graphs

Learning sparse coordination graphs adaptive to the coordination dynamics among agents is a long-standing problem in cooperative multi-agent learning . This paper studies this problem and proposes a novel method using the variance of payoff functions to construct context-aware sparse coordination topologies . We theoretically consolidate our method by proving that the smaller the variance of payoff functions is , the less likely action selection will change after removing the corresponding edge . Moreover , we propose to learn action representations to effectively reduce the influence of payoff functions ’ estimation errors on graph construction . To empirically evaluate our method , we present the Multi-Agent COordination ( MACO ) benchmark by collecting classic coordination problems in the literature , increasing their difficulty , and classifying them into different types . We carry out a case study and experiments on the MACO and StarCraft II micromanagement benchmark to demonstrate the dynamics of sparse graph learning , the influence of graph sparseness , and the learning performance of our method . 1 INTRODUCTION . Many real-world problems involve the cooperation of multiple agents , such as unmanned aerial vehicles ( Pham et al. , 2018 ; Xu et al. , 2018 ) and sensor networks ( Stranders et al. , 2009 ) . Like in single-agent settings , learning control policies for multi-agent teams largely relies on the estimation of action-value functions , no matter in value-based ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; 2020 ) or policy-based approaches ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Wang et al. , 2021c ) . However , learning action-value functions for complex multi-agent tasks remains a major challenge . Learning individual action-value functions ( Tan , 1993 ) is scalable but suffers from learning non-stationarity because it treats other learning agents as part of its environment . Joint action-value learning ( Claus & Boutilier , 1998 ) is free from learning non-stationarity but requires access to global information that is often unavailable during execution due to partial observability and communication constraints . Factored Q-learning ( Guestrin et al. , 2002a ) combines the advantages of these two methods . Learning the global action-value function as a combination of local utilities , factored Q functions maintain learning scalability while avoiding non-stationarity . Enjoying these advantages , fully decomposed Q functions significantly contribute to the recent progress of multi-agent reinforcement learning ( Samvelyan et al. , 2019 ; Wang et al. , 2021b ) . However , when fully decomposed , local utility functions only depend on local observations and actions , which may lead to miscoordination problems in partially observable environments with stochastic transition functions ( Wang et al. , 2020 ; 2021a ) and a gametheoretical pathology called relative overgeneralization ( Panait et al. , 2006 ; Böhmer et al. , 2020 ) . Relative overgeneralization renders optimal decentralized policies unlearnable when the employed value function does not have enough representational capacity to distinguish other agents ’ effects on local utility functions . Coordination graphs ( Guestrin et al. , 2002b ) provide a promising approach to solving these problems . Using vertices to represent agents and ( hyper- ) edges to represent payoff functions defined over the joint action-observation space of the connected agents , a coordination graph expresses a higher-order value decomposition among agents . Finding actions with the maximum value in a coordination graph can be achieved by distributed constraint optimization ( DCOP ) algorithms ( Cheng , 2012 ) , which consists of multiple rounds of message passing along the edges . Recently , DCG ( Böhmer et al. , 2020 ) scales coordination graphs to large state-action spaces , shows its ability to solve the problem of relative overgeneralization , and obtains competitive results on StarCraft II micromanagement tasks . However , DCG focuses on predefined static and dense topologies , which largely lack flexibility for dynamic environments and induce intensive and inefficient message passing . The question is how to learn dynamic and sparse coordination graphs sufficient for coordinated action selection . This is a long-standing problem in multi-agent learning . Sparse cooperative Q-learning ( Kok & Vlassis , 2006 ) learns value functions for sparse coordination graphs , but the graph topology is static and predefined by prior knowledge . Zhang & Lesser ( 2013 ) propose to learn minimized dynamic coordination sets for each agent , but the computational complexity grows exponentially with the neighborhood size of an agent . Recently , Castellini et al . ( 2019 ) study the representational capability of several sparse graphs but focus on random topologies and stateless games . In this paper , we push these previous works further by proposing a novel deep method that learns context-aware sparse coordination graphs adaptive to the dynamic coordination requirements . For learning sparse coordination graphs , we propose to use the variance of pairwise payoff functions as an indicator to select edges . Sparse graphs are used when selecting greedy joint actions for execution and the update of Q-function . We provide a theoretical insight into our method by proving that the probability of greedy action selection changing after an edge is removed decreases with the variance of the corresponding payoff function . Despite the advantages of sparse topologies , they raise the concern of learning instability . To solve this problem , we further equip our method with network structures based on action representations for utility and payoff learning to reduce the influence of estimation errors on sparse topologies learning . We call the overall learning framework Context-Aware SparsE Coordination graphs ( CASEC ) . For evaluation , we present the Multi-Agent COordination ( MACO ) benchmark . This benchmark collects classic coordination problems raised in the literature of multi-agent learning , increases their difficulty , and classifies them into 6 classes . Each task in the benchmark represents a type of problem . We carry out a case study on the MACO benchmark to show that CASEC can discover the coordination dependence among agents under different situations and to analyze how the graph sparsity influences action coordination . We further show that CASEC can largely reduce the communication cost ( typically by 50 % ) and perform significantly better than dense , static graphs and several alternative methods for building sparse graphs . We then test CASEC on the StarCraft II micromanagement benchmark ( Samvelyan et al. , 2019 ) to demonstrate its scalability and effectiveness . 2 BACKGROUND . In this paper , we focus on fully cooperative multi-agent tasks that can be modelled as a DecPOMDP ( Oliehoek et al. , 2016 ) consisting of a tuple G=⟨I , S , A , P , R , Ω , O , n , γ⟩ , where I is the finite set of n agents , γ ∈ [ 0 , 1 ) is the discount factor , and s ∈ S is the true state of the environment . At each timestep , each agent i receives an observation oi ∈ Ω drawn according to the observation function O ( s , i ) and selects an action ai ∈ A . Individual actions form a joint action a ∈ An , which leads to a next state s′ according to the transition function P ( s′|s , a ) , a reward r = R ( s , a ) shared by all agents . Each agent has local action-observation history τi ∈ T ≡ ( Ω×A ) ∗ × Ω . Agents learn to collectively maximize the global return Qtot ( s , a ) = Es0 : ∞ , a0 : ∞ [ ∑∞ t=0 γ tR ( st , at ) |s0 = s , a0 = a ] . In a coordination graph ( Guestrin et al. , 2002b ) G = ⟨V , E⟩ , each vertex vi ∈ V represents an agent i , and ( hyper- ) edges in E represent coordination dependencies among agents . In this paper , we consider pairwise edges , and such a coordination graph induces a factorization of the global Q : Qtot ( τ , a ) = 1 |V| ∑ i qi ( τi , ai ) + 1 |E| ∑ { i , j } ∈E qij ( τij , aij ) , ( 1 ) where qi and qij is utility functions for individual agents and pairwise payoff functions , respectively . τij = ⟨τi , τj⟩ and aij = ⟨ai , aj⟩ is the joint action-observation history and action of agent i and j . Within a coordination graph , the greedy action selection required by Q-learning can not be completed by simply computing the maximum of individual utility and payoff functions . Instead , distributed constraint optimization ( DCOP ) ( Cheng , 2012 ) techniques can be used . Max-Sum ( Stranders et al. , 2009 ) is a popular implementation of DCOP , which finds optimal actions on a coordination graph G = ⟨V , E⟩ via multi-round message passing on a bipartite graph Gm = ⟨Va , Vq , Em⟩ . Each node i ∈ Va represents an agent , and each node g ∈ Vq represents a utility ( qi ) or payoff ( qij ) function . Edges in Em connect g with the corresponding agent node ( s ) . Message passing on this bipartite graph starts with sending messages from node i ∈ Va to node g ∈ Vq : mi→g ( ai ) = ∑ h∈Fi\g mh→i ( ai ) + cig , ( 2 ) where Fi is the set of nodes connected to node i in Vq , and cig is a normalizing factor preventing the value of messages from growing arbitrarily large . The message from node g to node i is : mg→i ( ai ) = max ag\ai q ( ag ) + ∑ h∈Vg\i mh→g ( ah ) , ( 3 ) where Vg is the set of nodes connected to node g in Va , ag= { ah|h ∈ Vg } , ag\ai= { ah|h ∈ Vg\ { i } } , and q represents utility or payoff functions conditioned on ag . After several iterations of message passing , each agent i can find its optimal action by calculating a∗i = argmaxai ∑ h∈Fi mh→i ( ai ) . A drawback of Max-Sum or other message passing methods ( e.g. , max-plus ( Pearl , 2014 ) ) is that running them for each action selection through the whole system results in intensive computation and communication among agents , which is impractical for most applications with limited computational resources and communication bandwidth . In the following sections , we discuss how to solve this problem by learning sparse coordination graphs . Previous works ( Naderializadeh et al. , 2020 ; Li et al. , 2021 ) study soft versions of fully-connected coordination graphs based on attention mechanisms . Specifically , Li et al . ( 2021 ) uses graphs whose edge weights are learned by self-attention so that agents attend to observations of other agents differently . The information is used in local actors or a centralized critic . Naderializadeh et al . ( 2020 ) learns soft full graphs in a similar way , but the graph is used to mix local utilities conditioned on local action-observation history . Different from our work , these methods do not learn pairwise payoff functions , and the learned graphs are still fully-connected . 3 LEARNING CONTEXT-AWARE SPARSE GRAPHS . In this section , we introduce our methods for learning context-aware sparse graphs . We first introduce how we construct a sparse graph for effective action selection in Sec . 3.1 . After that , we introduce our learning framework in Sec . 3.2 . Although sparse graphs can reduce communication overhead , they raise the concern of learning instability . We discuss this problem and how to alleviate it in Sec . 3.3 .

Summary: The paper focuses on the problem of cooperative multi-agent Reinforcement learning and proposes a novel way to learn the dynamic (state dependent) coordination graph for the joint action selection from factorized joint value representations. The key idea here is to use the variance in payoff function estimates can be a good indicator for whether a coordination graph edge should be present or not. However, given the deep RL goal is learn these payoff functions there is a cyclic dependency between errors from payoff estimation and coordination graph estimation. The paper therefore also proposes an alternative action representation scheme that helps mitigate some of these problems. The paper evaluates the proposed approach on many classic coordination problems as well as subset of SMAC benchmark tasks.