Self-Organized Polynomial-time Coordination Graphs

1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) is a promising approach to a variety of real-world applications , such as sensor networks ( Zhang & Lesser , 2011 ; Ye et al. , 2015 ) , traffic light control ( Van der Pol & Oliehoek , 2016 ) , and multi-robot formation ( Alonso-Mora et al. , 2017 ) . One of the long-lasting challenges of cooperative MARL is how to organize coordination for a large multi-agent system . Coordination graph ( Guestrin et al. , 2001 ) is a classical approach to represent coordination relations in the reinforcement learning framework . It decomposes a multi-agent system into a suite of overlapping factors . Each factor is a hyper-edge covering a subset of agents that may involve coordinated behaviors . More specifically , the joint value function of the multi-agent system is factorized to the summation of local value functions based on factors or edges . Such a graph structure is a particular type of inductive bias that encodes task-specific coordination dependencies . It is known that conducting appropriate inductive bias can both reduce the model complexity and improve the generalizability ( Battaglia et al. , 2018 ) . One fundamental problem for coordination graphs is the trade-off between the representational capacity of value functions and the computational complexity of policy execution . To obtain value functions with high expressiveness , an advanced approach , deep coordination graphs ( DCG ) ( Böhmer et al. , 2020 ) , considers a static complete graph connecting all pairs of agents . This graph structure has high representational capacity in terms of function expressiveness but raises a challenge for computation in the execution phase . The greedy action selection over a coordination graph can be formalized to a decentralized constraint optimization problem ( DCOP ) , finding the maximum-value joint actions ( Guestrin et al. , 2001 ; Zhang & Lesser , 2013 ; Böhmer et al. , 2020 ) . Note that the DCOP and its any constant-ratio approximation are NP-hard problems , especially for the complete graph used in DCG ( Dagum & Luby , 1993 ; Park & Darwiche , 2004 ) . From the perspective of theoretical computer science , any scalable heuristic algorithm ( e.g. , max-sum algorithm ( Pearl , 1988 ) ) for DCOP problem may have an unpredictable gap with the optimal solution . To visualize this phenomenon , we conduct a motivating example for generating a suite of complete-graph DCOPs with random edge-values . More detailed experiment setting are deferred to to Appendix A . We investigate the accuracy and the relative joint Q error of the max-sum algorithm in computing optimal joint actions , which is the default implementation of MARL methods based on coordination graphs ( Stranders et al. , 2009 ; Zhang & Lesser , 2013 ; Böhmer et al. , 2020 ) . The results are shown in Figure 1 . As the number of agents increases , the accuracy of greedy action selection consistently decreases , and the relative joint Q error between the selected and optimal joint actions increases accordingly . How to incorporating polynomial-time complexity guarantee into coordination graphs with sufficient function expressiveness remains an open problem in MARL . In this paper , we aim to overcome a dilemma in coordination graphs , i.e. , expressive graph structures lead to computationally intractable DCOPs but concise graph structures are in lack of function expressiveness . To tackle this problem , we propose a novel graph-based MARL method for multiagent coordination , named Self-Organized Polynomial-time Coordination Graphs ( SOP-CG ) , that ( 1 ) utilizes structured graph topologies with polynomial-time guarantees for DCOPs and ( 2 ) extends their representation expressiveness through a dynamic graph organization mechanism . These two characteristics open up a new family of MARL algorithms . More specifically , we first construct graph classes guaranteeing polynomial-time DCOPs and then conduct a state-dependent graph selection mechanism . Such a dynamic factorization structure can be self-organized . We introduce an imaginary coordinator agent , whose action is to select a graph topology in a given class at each time step . SOP-CG incorporates this imaginary agent into the Bellman optimality equation supporting end-to-end learning . Figure 2 illustrates the basic idea of SOP-CG . We integrate the search of graph structures into the MARL paradigm . The utilities of different graph topologies are learned along with the RL objective . As the graph utilities are updated through agent-environment interactions , the coordination graph will evolve to the topology characterizing the coordination dependencies . We evaluate the performance of SOP-CG in both grid-world and simulated physical environments . Empirical results on challenging tasks demonstrate that SOP-CG outperforms state-of-the-art baselines . By extensive ablation studies , we verify that harnessing specific structures with optimality guarantee of the induced DCOPs improves the sample quality and the accuracy of the one-step TD target . Furthermore , we show that SOP-CG can learn interpretable and context-dependent coordination graphs , which induces effective and dynamic coordination among agents . 2 RELATED WORK . Multi-agent reinforcement learning ( OroojlooyJadid & Hajinezhad , 2019 ) is challenged by the size of joint action space , which grows exponentially with the number of agents . Independent Q-learning ( Tan , 1993 ; Foerster et al. , 2017 ) models agents as independent learners , which makes the environment non-stationary in the perspective of each agent . An alternative paradigm called centralized training and decentralized execution ( CTDE ; Kraemer & Banerjee , 2016 ) is widely used in both policy-based and value-based methods . Policy-based multi-agent reinforcement learning methods use a centralized critic to compute gradient for the local actors ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Wen et al. , 2019 ; Wang et al. , 2020d ) . Value-based methods usually decompose the joint value function into individual value functions under the IGM ( individual-global-max ) principle , which guarantees the consistency between local action selection and joint action optimization ( Sunehag et al. , 2018 ; Rashid et al. , 2020b ; Son et al. , 2019 ; Wang et al. , 2021a ; Rashid et al. , 2020a ) . Other work also studies this problem from the perspective of agent roles and individuality ( Wang et al. , 2020a ; b ; Jiang & Lu , 2021 ) or communication learning ( Singh et al. , 2018 ; Das et al. , 2019 ; Wang et al. , 2020c ) . Compared to these methods , our work is built upon graph-based value decomposition , which explicitly models the interaction among agents . Coordination graphs are classical technique for planning in multi-agent systems ( Guestrin et al. , 2001 ; 2002b ) . They are combined with multi-agent deep reinforcement learning by recent work ( Castellini et al. , 2019 ; Böhmer et al. , 2020 ; Li et al. , 2020 ; Wang et al. , 2021b ) . Joint action selection on coordination graphs can be modeled as a decentralized constraint optimization problem ( DCOP ) , and previous methods compute approximate solutions by message passing among agents ( Pearl , 1988 ) . As the work which is most closed to our method , Zhang & Lesser ( 2013 ) presents an algorithm based on coordination graph searching . They define a measurement to quantify the potential loss of the lack of coordination between agents and search for a coordination structure to minimize the communication cost within restricted loss of utilities . However , their induced DCOPs still remain NP-hard . In contrast , minimizing communication is not our core motivation , and we aim to use a structured graph class to maintain sufficient function expressiveness when bypassing the computational hardness of large-scale DCOPs ( Dagum & Luby , 1993 ; Park & Darwiche , 2004 ) . 3 BACKGROUND . In this paper , we consider about fully cooperative multi-agent tasks that can be modelled as a DecPOMDP ( Oliehoek et al. , 2016 ) defined asM = ⟨D , S , { Ai } ni=1 , T , { Oi } ni=1 , { σi } ni=1 , R , h , b0 , γ⟩ , where D = { 1 , . . . , n } is the set of n agents , S is a set of states , h is the horizon of the environment , γ ∈ [ 0 , 1 ) is the discount factor , and b0 ∈ ∆ ( S ) denotes the initial state distribution . At each stage t , each agent i takes an action ai ∈ Ai and forms the joint action a = ( a1 . . . , an ) , which leads to a next state s′ according to the transition function T ( s′|s , a ) and an immediate reward R ( s , a ) shared by all agents . Each agent i observes the state only partially by drawing observations oi ∈ Oi , according to σi . The joint history of agent i ’ s observations oi , t and actions ai , t is denoted as τi , t = ( oi,0 , ai,0 , . . . , oi , t−1 , ai , t−1 , oi , t ) ∈ ( Oi ×Ai ) t ×Oi . Deep Q-Learning . Q-learning algorithms is a well-known algorithm to find the optimal joint action-value function Q∗ ( s , a ) = r ( s , a ) +γEs′ [ maxa′ Q∗ ( s′ , a′ ) ] . Deep Q-learning approximates the action-value function with a deep neural network with parameters θ . In Multi-agent Q-learning algorithms ( Sunehag et al. , 2018 ; Rashid et al. , 2020b ; Son et al. , 2019 ; Wang et al. , 2021a ) , a replay memory D is used to store the transition tuple ( τ , a , r , τ ′ ) , where r is the immediate reward when taking action a at joint action-observation history τ with a transition to τ ′ . Q ( τ , a ; θ ) is used in place of Q ( s , a ; θ ) , because of partial observability . Hence , parameters θ are learnt by minimizing the following expect TD error : L ( θ ) = E ( τ , a , r , τ ′ ) ∈D [ ( r + γV ( τ ′ ; θ− ) −Q ( τ , a ; θ ) ) 2 ] ( 1 ) where V ( τ ′ ; θ− ) = maxa′ Q ( τ ′ , a′ ; θ− ) is the one-step expected future return of the TD target and θ− are the parameters of the target network , which will be periodically updated with θ . Coordination graphs . An undirect coordination graph ( CG , Guestrin et al . ( 2001 ) ) G = ⟨V , E⟩ contains vertex vi ∈ V for each agent 1 ≤ i ≤ n and a set of ( hyper- ) edges in E ⊆ 2V which represents coordination dependencies among agents . Prior work considers higher order coordination where the edges depend on actions of several agents ( Guestrin et al. , 2002a ; Kok & Vlassis , 2006 ; Guestrin et al. , 2002b ) . Such a coordination graph induces a factorization of global Q function : Qtot ( s , a ) = ∑ vi∈V f i ( ai|s ) + ∑ e= { e1 , ... , e|e| } ∈E fe ( ae1 , . . . , ae|e| |s ) ( 2 ) , where f i represents the individual utility of agent i and fe specifies the payoff contribution for the actions of the agents connected by the ( hyper- ) edge e , so that the global optimal solution can be found through maximizing this joint value . The special case that E is an empty set yields VDN , but each additional edge enables the value representation of the joint actions of a pair of agents and can thus help to avoid relative-overgeneralization ( Böhmer et al. , 2020 ) . In many coordination graph learning works ( Zhang & Lesser , 2013 ; Böhmer et al. , 2020 ; Wang et al. , 2021b ) , the hyper-edges are simplified into pairwise edges . The graph is usually considered to be specified before training . Guestrin et al . ( 2002b ) and Zhang & Lesser ( 2013 ) suggest that the graph could also depend on states , which means each state can have its own unique CG .

Summary: This paper proposes an extension of deep coordination graph, called Self-Organized Polynomial-time Coordination Graphs (SOP-CG). Instead of pre-specified graph topology used in DCG, their method allows graph topology to be state-dependent, which is achieved by a coordinator agent, and the optimization of this agent is incorporated in a modified temporal difference learning paradigm. Two pre-specified undirected acyclic graph classes are used to ensure polynomial-time graph selection and accurate greedy action selection. The result on sensor network, grid world and MPE shows that such a trade-off between the representational capacity of graph topology and the computational accuracy can improve the performance of MARL and learn meaningful graph topology.