Deep Coordination Graphs

1 INTRODUCTION . One of the central challenges in cooperative multi-agent reinforcement learning ( MARL , Oliehoek & Amato , 2016 ) is coping with the size of the joint action space , which grows exponentially in the number of agents . For example , this paper evaluates tasks where eight agents each have six actions to choose from , yielding a joint action space with more than a million actions . Efficient MARL methods must thus be able to generalize over large joint action spaces , in the same way that convolutional neural networks allow deep RL to generalize over large visual state spaces . Even though few benchmark tasks actually require agent policies to be independently executable , one common approach to coping with large action spaces is to decentralize the decision policy and/or value function . For example , Figure 1a shows how the joint value function can be factorized into utility functions that each depend only on the actions of one agent ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ) . Consequently , the joint value function can be efficiently maximized if each agent simply selects the action that maximizes its corresponding utility function . This factorization can represent any deterministic policy and thus can represent at least one optimal policy . However , that policy may not be learnable due to a game-theoretic pathology called relative overgeneralization1 ( Panait et al. , 2006 ) : during exploration other agents act randomly and punishment caused by uncooperative agents may outweigh rewards that would be achievable with coordinated actions . If the employed value function does not have the representational capacity to distinguish the values of coordinated and uncoordinated actions , an optimal policy can not be learned . However , Castellini et al . ( 2019 ) show that higher-order factorization of the value function works surprisingly well in one-shot games that are vulnerable to relative overgeneralization , even if each factor depends on the actions of only a small subset of agents . Such a higher-order factorization can be expressed as an undirected coordination graph ( CG , Guestrin et al. , 2002a ) , where each vertex represents one agent and each ( hyper- ) edge one payoff function over the joint action space of the connected agents . Figure 1b shows a CG with pairwise edges and the corresponding value factorization . Depending on the CG topology , the value can thus depend nontrivially on the actions of all agents , yielding a richer representation . Although the value can no longer be maximized by each agent individually , the greedy action can be found using message passing along the edges ( also known as belief propagation , Pearl , 1988 ) . Sparse cooperative Q-learning ( Kok & Vlassis , 2006 ) applies CGs to MARL but does not scale to modern benchmarks , as each payoff function ( f12 and f23 in Figure 1b ) is represented as a table over the state and joint action space of the connected agents . Castellini et al . ( 2019 ) use neural networks to approximate payoff functions , but only in one-shot games , and still require a unique function for each edge in the CG . Consequently , each agent group , represented by an edge , must still experience all corresponding action combinations , which can require executing a significant subset of the joint action space . 1 Not to be confused with generalization in the context of function approximation . To address these issues , this paper proposes the deep coordination graph ( DCG ) , a deep RL algorithm that scales to modern benchmark tasks . DCG represents the value function as a CG with pairwise payoffs2 ( Figure 1b ) and individual utilities ( Figure 1a ) . This improves the representational capacity beyond state-of-the-art value factorization approaches like VDN ( Sunehag et al. , 2018 ) and QMIX ( Rashid et al. , 2018 ) . To achieve scalability , DCG employs parameter sharing between payoffs and utilities . Parameter sharing has long been a staple of factorized MARL . Methods like VDN and QMIX condition an agent ’ s utility on its history , that is , its past observations and actions , and share the parameters of all utility functions . Experiences of one agent are thus used to train all . This can dramatically improve the sample efficiency compared to unfactored methods ( Foerster et al. , 2016 ; 2018 ; Lowe et al. , 2017 ; Schröder de Witt et al. , 2019 ; Son et al. , 2019 ) , which correspond to a CG with one hyper-edge connecting all agents ( Figure 1c ) . DCG takes parameter sharing one step further by approximating all payoff functions with the same neural network . To allow unique outputs for each payoff , the network is conditioned on a learned embedding of the participating agents ’ histories . This requires only one linear layer more than VDN and has thus less parameters as QMIX . DCG is trained end-to-end with deepQ-learning ( DQN , Mnih et al. , 2015 ) , but uses message passing to coordinate greedy action selection between all agents in the graph . For k message passes over n agents with m actions each , the time complexity of maximization is onlyO ( km ( n+m ) |E| ) , where |E| ≤ n 2−n 2 is the number of ( pairwise ) edges , compared to O ( m n ) for DQN without factorization . We compare DCG ’ s performance with that of other MARL Q-learning algorithms in a challenging family of predator-prey tasks that require coordinated actions . Here DCG is the only algorithm that solves the harder tasks . We also investigate the influence of graph topologies on the performance . 2 RELATED WORK . A general overview over cooperative deep MARL can be found in OroojlooyJadid & Hajinezhad ( 2019 ) . Independent Q-learning ( IQL Tan , 1993 ) decentralizes the agents ’ policy by modeling each agent as an independentQ-learner . However , the task from the perspective of a single agent becomes nonstationary as other agents change their policies . To address this , Foerster et al . ( 2017 ) show how to stabilize IQL when using experience replay buffers . Another approach to decentralized agents is centralized training and decentralized execution ( Foerster et al. , 2016 ) with a factorized value function . Value decomposition networks ( VDN , Sunehag et al. , 2018 ) performs central Q-learning with a value function that is the sum of independent utility functions for each agent ( Figure 1a ) . The greedy policy can be executed by maximizing each utility independently . QMIX ( Rashid et al. , 2018 ) improves upon this approach by combining the agents ’ utilities with a mixing network , which is monotonic in the utilities and depends on the global state . This allows different mixtures in different states and the central value can be maximized independently due to monotonicity . All of these approaches are derived in Appendix A.1 and can use parameter sharing between the value/utility functions . However , they represent the joint value with independent values/utilities and are therefore susceptible to the relative overgeneralization pathology . We demonstrate this by comparing DCG with all the above algorithms . 2 The method can be generalized to CG with hyper-edges , that is , to payoff functions for more than 2 agents . Another straightforward way to decentralize in MARL is to define the joint policy as a product of independent agent policies . This lends itself to the actor-critic framework , where the critic is discarded during execution and can therefore condition on the global state and all agents ’ actions during training . Examples are MADDPG ( Lowe et al. , 2017 ) for continuous actions and COMA ( Foerster et al. , 2018 ) for discrete actions . Wei et al . ( 2018 ) specifically investigate the relative overgeneralization pathology in continuous multi-agent tasks and show improvement over MADDPG by introducing policy entropy regularization . MACKRL ( Schröder de Witt et al. , 2019 ) follows the approach in Foerster et al . ( 2018 ) , but uses common knowledge to coordinate agents during centralized training . Son et al . ( 2019 ) define QTRAN , which also has a centralized critic but uses a greedy actor w.r.t . a VDN factorized function . The corresponding utility functions are distilled from the critic under constraints that ensure proper decentralization . Böhmer et al . ( 2019 ) present another approach to decentralize a centralized value function , which is locally maximized by coordinate ascent and decentralized by training IQL agents from the same replay buffer . Centralized joint Q-value functions do not allow to share parameters to the same extent as value factorization , and we compare DCG to QTRAN to demonstrate the advantage in sample efficiency . That being said , DCG value factorization can in principle be applied to any of the above centralized critics to equally improve sample efficiency at the same cost of representational capacity . We leave this to future work . Other work deals with gigantic numbers of agents , which requires additional assumptions to reduce the sample complexity . For example , Yang et al . ( 2018 ) introduce mean-field multi-agent learning ( MF-MARL ) , which factorizes a tabular value function for hundreds of agents into pairwise payoff functions between neighbors in a uniform grid of agents . These payoffs share parameters similar to DCG . Chen et al . ( 2018 ) introduce a value factorization for a similar setup based on a low-rank approximation of the joint value . This approach is restricted by uniformity assumptions between agents , but uses otherwise parameter sharing similar to DCG . The value function can not be maximized globally and must be locally maximized with coordinate ascent . These techniques are designed for much larger sets of agents and do not perform well in the usual MARL settings considered in this paper . While they use similar parameter sharing techniques as DCG , we do therefore not compare against them . Coordination graphs ( CG ) have been extensively studied in multi-agent robotics with given payoffs ( e.g . Rogers et al. , 2011 ; Yedidsion et al. , 2018 ) . Sparse cooperative Q-learning ( SCQL , Kok & Vlassis , 2006 ) uses CG in discrete state and action spaces by representing all utility and payoff functions as tables . However , the tabular approach restricts practical application of SCQL to tasks with few agents and small state and action spaces . Castellini et al . ( 2019 ) use neural networks to approximate payoff functions , but only in one-shot games , and still require a unique function for each edge in the CG . DCG expands greatly upon these works by introducing parameter sharing between all payoffs ( as in VDN/QMIX ) , conditioning on local information ( as in MF-MARL ) and evaluating in more complex tasks that are vulnerable to relative overgeneralization . 3 BACKGROUND . In this paper we assume a Dec-POMDP for n agents 〈S , { Ai } ni=1 , P , r , { Oi } ni=1 , { σi } ni=1 , n , γ〉 ( Oliehoek & Amato , 2016 ) . S denotes a finite or continuous set of environmental states and Ai the discrete set of actions available to agent i . At discrete time t , the next state st+1 ∈ S is drawn from transition kernel st+1 ∼ P ( ·|st , at ) , conditioned on the current state st ∈ S and joint action at ∈ A : = A1 × . . . × An of all agents . A transition yields collaborative reward rt : = r ( st , at ) , and γ ∈ [ 0 , 1 ) denotes the discount factor . Each agent i observes the state only partially by drawing observations oit ∈ Oi from its observation kernel oit ∼ σi ( ·|st ) . The history of agent i ’ s observations oit ∈ Oi and actions ait ∈ Ai is in the following denoted as τ it : = ( oi0 , ai0 , oi1 , . . . , oit−1 , ait−1 , oit ) ∈ ( Oi ×Ai ) t ×Oi . Without loss of generality , this paper restricts itself to episodic tasks , which yield episodes ( s0 , { oi0 } ni=1 , a0 , r0 , . . . , sT , { oiT } ni=1 ) of varying ( but finite ) length T . 3.1 DEEP Q-LEARNING The goal of collaborative multi-agent reinforcement learning ( MARL ) is to find an optimal policy π∗ : S × A → [ 0 , 1 ] , that chooses joint actions at ∈ A such that the expected discounted sum of future reward is maximized . This can be achieved by estimating the optimal Q-value function3 : 3 We overload the notation f ( y|x ) to also indicate the inputs x and outputs y of multivariate functions f . q∗ ( a|s ) : = IEπ∗ [ T−1∑ t=0 γtrt ∣∣ s0=s a0=a ] = r ( s , a ) + γ ∫ P ( s′|s , a ) max q∗ ( ·|s′ ) ds′ . ( 1 ) The optimal policy π∗ ( ·|st ) chooses greedily the action a ∈ A that maximizes the corresponding optimal Q-value q∗ ( a|st ) . In fully observable discrete state and action spaces , q∗ can be learned in the limit from interactions with the environment ( Watkins & Dayan , 1992 ) . For large or continuous state spaces , q∗ can only be approximated , e.g. , with a deep neural network qθ ( DQN , Mnih et al. , 2015 ) , parameterized by θ , by minimizing the mean-squared Bellman loss with gradient descent : LDQN : = IE [ 1 T T−1∑ t=0 ( rt + γmax qθ̄ ( ·|st+1 ) − qθ ( at|st ) ) 2 ∣∣∣ { st , at , rt , st+1 } Tt=0 ∼ D ] . ( 2 ) The expectation is estimated with samples from an experience replay buffer D holding previously observed episodes ( Lin , 1992 ) , and θ̄ denotes the parameter of a separate target network , which is periodically replaced with a copy of θ to improve stability . Double Q-learning further stabilizes training by choosing the next action greedily w.r.t . the current network qθ , i.e. , qθ̄ ( arg max qθ ( ·|st+1 ) |st+1 ) instead of the target network max qθ̄ ( ·|st+1 ) ( van Hasselt et al. , 2016 ) . In partially observable environments , the learned policy can not condition on the state st . Instead , Hausknecht & Stone ( 2015 ) approximate a Q-function that conditions on the agent ’ s history τt : = { τ it } ni=1 , i.e. , qθ ( a|τt ) , by conditioning a recurrent neural network ( e.g. , a GRU , Chung et al. , 2014 ) on the agents ’ observations ot : = ( o1t , . . . , o n t ) and last actions at−1 , that is , qθ ( a|ht ) conditions on the recurrent network ’ s hidden state hψ ( ht|ht−1 , ot , at−1 ) , where h0 is initialized with zeros . Applying DQN to multi-agent tasks quickly becomes infeasible , due to the combinatorial explosion of state and action spaces . Moreover , DQN value functions can not be maximized without evaluating all possible actions . To allow MARL Q-learning with efficient maximization , various algorithms based on value factorization have been developed . We derive IQL ( Tan , 1993 ) , VDN ( Sunehag et al. , 2018 ) , QMIX ( Rashid et al. , 2018 ) and QTRAN ( Son et al. , 2019 ) in Appendix A.1 .

Summary: Teh authors propose to learn value functions that are a sum of utility (single-agent) and payoff (2-agent) components. The weights between all function are shared (common RNN). This stands in contrast to VDNs (only uses utility functions or centralized value functions. The authors evaluate on predator-prey, where they argue that e.g. VDN fails to learn.