Learning to Share in Multi-Agent Reinforcement Learning

1 INTRODUCTION . In multi-agent reinforcement learning ( MARL ) , there are multiple agents interacting with the environment via their joint action to cooperatively optimize an objective . Many methods of centralized training and decentralized execution ( CTDE ) have been proposed for cooperative MARL , such as VDN ( Sunehag et al. , 2018 ) , QMIX ( Rashid et al. , 2018 ) , and QTRAN ( Son et al. , 2019 ) . However , these methods suffer from the overgeneralization issue ( Palmer et al. , 2018 ; Castellini et al. , 2019 ) . Moreover , they may not easily scale up with the number of agents due to centralized learning ( Qu et al. , 2020a ) . In many MARL applications , there are a large number of agents that are deployed as a partially connected network and collaboratively make decisions to optimize the globally averaged return , such as smart grids ( Dall ’ Anese et al. , 2013 ) , network routing ( Jiang et al. , 2020 ) , traffic signal control ( Chu et al. , 2020 ) , and IoT ( Xu et al. , 2019 ) . To deal with such scenarios , networked MARL is formulated to decompose the dependency among all agents into dependencies between only neighbors in such scenarios . To avoid decision-making with insufficient information , agents are permitted to exchange messages with neighbors over the network . In such settings , it is feasible for agents to learn to make decisions in a decentralized way ( Zhang et al. , 2018 ; Qu et al. , 2020b ) . However , there are still difficulties of dependency if anyone attempts to make decision independently , e.g. , prisoner ’ s dilemma and tragedy of the commons ( Pérolat et al. , 2017 ) . Existing methods tackle these problems by consensus update of value function ( Zhang et al. , 2018 ) , credit assignment ( Wang et al. , 2020 ) , or reward shaping ( Chu et al. , 2020 ) . However , these methods rely on either access to global state and joint action ( Zhang et al. , 2018 ) or handcrafted reward functions ( Wang et al. , 2020 ; Chu et al. , 2020 ) . Inspired by the fact that sharing plays a key role in human ’ s learning of cooperation , in this paper , we propose Learning To Share ( LToS ) , a hierarchically decentralized learning method for networked MARL . LToS enables agents to learn to dynamically share reward with neighbors so as to collaboratively optimize the global objective . The high-level policies decompose the global objective into local ones by determining how to share their rewards , while the low-level policies optimize local objectives induced by the high-level policies . LToS learns in a decentralized manner , and we prove that the high-level policies are a mean-field approximation of the joint high-level policy . Moreover , the high-level and low-level policies form a bi-level optimization and alternately learn to optimize the global objective . LToS is easy to implement and currently realized by DDPG ( Lillicrap et al. , 2016 ) as the high-level policy and DGN ( Jiang et al. , 2020 ) as the low-level policy . We empirically demonstrate that LToS outperforms existing methods for networked MARL in both social dilemma and two real-world scenarios . To the best of our knowledge , LToS is the first to learn to share reward for global optimization in networked MARL . 2 RELATED WORK . There are many recent studies for collaborative MARL . Most of them adopt centralized training and decentralized execution , such as COMA ( Foerster et al. , 2018 ) , VDN ( Sunehag et al. , 2018 ) , QMIX ( Rashid et al. , 2018 ) , and QTRAN ( Son et al. , 2019 ) . Many are constructed on the basis of factorizing the joint Q-function by assuming additivity ( Sunehag et al. , 2018 ) , monotonicity ( Rashid et al. , 2018 ) , or factorizable tasks ( Son et al. , 2019 ) . However , they are learned in a centralized way and hence may not easily scale up with the number of agents in networked MARL ( Qu et al. , 2020a ) . Moreover , these factorized methods suffer from the overgeneralization issue ( Palmer et al. , 2018 ; Castellini et al. , 2019 ) . Other studies focus on decentralized training specifically in networked MARL , to which our work is more closely related . Zhang et al . ( 2018 ) proposed consensus update of value function , but it requires global state at each agent , which is usually unavailable in decentralized training . Chu et al . ( 2020 ) introduced a spatial discount factor to capture the influence between agents , but the spatial discount factor remains hand-tuned . Sodomka et al . ( 2013 ) and Peysakhovich & Lerer ( 2018b ) involved the concept of transferable utility to encourage cooperation , and Peysakhovich & Lerer ( 2018a ) resorted to game theory and gave more complex reward designs . However , these methods can not be extended beyond two-player games . Hughes et al . ( 2018 ) proposed the inequity aversion model to balance agents ’ selfish desire and social fairness . Wang et al . ( 2020 ) considered to learn the Shapley value as the credit assignment . However , these methods still rely on hand-crafted reward designs . Mguni et al . ( 2019 ) added an extra part to the original reward as non-potential based reward shaping and used Bayesian optimization to induce the convergence to a desirable equilibrium between agents . However , the extra part remains fixed during an episode , which makes it less capable of dealing with dynamic environments . Moreover , the reward shaping alters the original optimization problem . 3 BACKGROUND . 3.1 NETWORKED MULTI-AGENT REINFORCEMENT LEARNING . Assume N agents interact with an environment . Let V = { 1 , 2 , · · · , N } be the set of agents . The multi-agent system is modeled as an undirected graph G ( V , E ) , where each agent i serves as vertex i and E ⊆ V × V is the set of all edges . Two agents i , j ∈ V can communicate with each other if and only if eij = ( i , j ) ∈ E . We denote agent i and its all neighbors in the graph together as a set Ni . The state of the environment s ∈ S transitions upon joint action a ∈ A according to transition probability Pa : S ×A× S → [ 0 , 1 ] , where joint action set A = ×i∈VAi . Each agent i has a policy πi ∈ Πi : S ×Ai → [ 0 , 1 ] , and we denote the joint policy of all agents as π ∈ Π = ×i∈VΠi . For networked MARL , a common and realistic assumption is that the reward of each agent i just depends on its action and the actions of its neighbors ( Qu et al. , 2020a ) , i.e. , ri ( s , a ) = ri ( s , aNi ) . Moreover , each agent i may only obtain partial observation oi ∈ Oi , but can approximate the state by the observations ofNi ( Jiang et al. , 2020 ) or the observation history ( Chu et al. , 2020 ) , which are all denoted by oi for simplicity . The global objective is to maximize the sum of cumulative rewards of all agents , i.e. , ∑∞ t=0 ∑N i=1 γ trti . 3.2 MARKOV GAME . In such a setting , each agent can individually maximizes its own expected return , which is known as Markov game . This may lead to stable outcome or Nash equilibrium , which however is usually sub-optimal . Given π , the value function of agent i is given by vπi ( s ) = ∑ a π ( a|s ) ∑ s′ pa ( s ′|s , a ) [ ri + γvπi ( s′ ) ] , ( 1 ) where pa ∈ Pa describes the state transitions . A Nash equilibrium is defined as ( Mguni et al. , 2019 ) v ( πi , π−i ) i ( s ) ≥ v ( π′i , π−i ) i ( s ) , ∀π ′ i ∈ Πi , ∀s ∈ S , ∀i ∈ V , ( 2 ) where π−i = ×j∈V\ { i } πj . 4 METHOD . The basic idea of LToS is to enable agents to learn how to share reward with neighbors such that agents are encouraged to collaboratively optimize the global objective in networked MARL . LToS is a decentralized hierarchy . At each agent , the high-level policy determines the weights of reward sharing based on low-level policies while the low-level policy directly interacts with the environment to optimize local objective induced by high-level policies . Therefore , they form a bi-level optimization and alternately learn towards the global objective . 4.1 REWARD SHARING . The intuition of reward sharing is that if agents share their rewards with others , each agent has to consider the consequence of its actions on others , and thus it promotes cooperation . In networked MARL , as the reward of an agent is assumed to depend on the actions of neighbors , we allow reward sharing between neighboring agents . For the graph of V , we additionally define a set of directed edges , D , constructed from E . Specifically , we add a loop dii ∈ D for each agent i and split each undirected edge eij ∈ E into two directed edges : dij = ( i , j ) and dji = ( j , i ) ∈ D. Each agent i determines a weight wij ∈ [ 0 , 1 ] for each directed edge dij , ∀j ∈ Ni , subject to the constraint ∑ j∈Ni wij = 1 , so that wij proportion of agent i ’ s environment reward ri will be shared to agent j. Letw ∈ W = ×dij∈Dwij be the weights of the graph . Therefore , the shaped reward after sharing for each agent i is defined as rwi = ∑ j∈Ni wjirj . ( 3 ) 4.2 HIERARCHY . Assume there is a joint high-level policy φ ∈ Φ : S ×W → [ 0 , 1 ] to determinew . Given φ andw , we can define the value function of π at each agent i based on ( 1 ) as vπi ( s ; φ ) = ∑ w φ ( w|s ) ∑ a π ( a|s , w ) ∑ s′ pa ( s ′|s , a ) [ rwi + γvπi ( s′ ; φ ) ] , ( 4 ) vπi ( s ; w , φ ) = ∑ a π ( a|s , w ) ∑ s′ pa ( s ′|s , a ) [ rwi + γvπi ( s′ ; φ ) ] . ( 5 ) We express w as a discrete action for simplicity . It also holds for continuous action as long as we change all the summations to integrals . Let V φV ( s ; π ) . = ∑ i∈V v π i ( s ; φ ) and Q φ V ( s , w ; π ) . = ∑ i∈V v π i ( s ; w , φ ) . Proposition 4.1 . Given π , V φV ( s ; π ) and Q φ V ( s , w ; π ) are respectively the value function and action-value function of φ . Proof . Let rφi . = ∑ a π ( a|s , w ) rwi and pw ( s′|s , w ) . = ∑ a π ( a|s , w ) pa ( s′|s , a ) . As commonly assumed the reward is deterministic given s and a , from ( 4 ) , we have , vπi ( s ; φ ) = ∑ w φ ( w|s ) ∑ a π ( a|s , w ) [ rwi + ∑ s′ pa ( s ′|s , a ) γvπi ( s′ ; φ ) ] ( 6 ) = ∑ w φ ( w|s ) ∑ s′ pw ( s ′|s , w ) [ rφi + γv π i ( s ′ ; φ ) ] , ( 7 ) where pw ∈ Pw : S ×W × S → [ 0 , 1 ] describes the state transitions given π . Let rφV . = ∑ i∈V r φ i , and from ( 7 ) we have V φV ( s ; π ) = ∑ i∈V ∑ w φ ( w|s ) ∑ s′ pw ( s ′|s , w ) [ rφi + γv π i ( s ′ ; φ ) ] ( 8 ) = ∑ w φ ( w|s ) ∑ s′ pw ( s ′|s , w ) [ ∑ i∈V rφi + γ ∑ i∈V vπi ( s ′ ; φ ) ] ( 9 ) = ∑ w φ ( w|s ) ∑ s′ pw ( s ′|s , w ) [ rφV + γV φ V ( s ′ ; π ) ] , ( 10 ) and similarly , QφV ( s , w ; π ) = ∑ i∈V ∑ s′ pw ( s ′|s , w ) [ rφi + γ ∑ w′ φ ( w′|s′ ) vπi ( s′ ; w′ , φ ) ] ( 11 ) = ∑ s′ pw ( s ′|s , w ) [ ∑ i∈V rφi + γ ∑ w′ φ ( w′|s′ ) ∑ i∈V vπi ( s ′ ; w′ , φ ) ] ( 12 ) = ∑ s′ pw ( s ′|s , w ) [ rφV + γ ∑ w′ φ ( w′|s′ ) QφV ( s ′ , w′ ; π ) ] . ( 13 ) Moreover , from the definitions of rwi and r φ i we have rφV = ∑ a π ( a|s , w ) ∑ i∈V rwi = ∑ a π ( a|s , w ) ∑ i∈V ∑ j∈Ni wjirj ( 14 ) = ∑ a π ( a|s , w ) ∑ ( i , j ) ∈D wijri = ∑ a π ( a|s , w ) ∑ i∈V ri , ( 15 ) Thus , given π , V φV ( s ) and Q φ V ( s , w ) are respectively the value function and action-value function of φ in terms of the sum of expected cumulative rewards of all agents , i.e. , the global objective . Proposition 4.1 implies that φ directly optimizes the global objective by generating w given π . Unlike existing hierarchical RL methods , we can directly construct the value function and actionvalue function of φ based on the value function of π at each agent . As φ optimizes the global objective given π while πi optimizes the shaped reward individually at each agent given φ ( assuming π convergent to Nash equilibrium or stable outcome , denoted as lim ) , they form a bi-level optimization . Let Jφ ( π ) and Jπ ( φ ) denote the objectives of φ and π respectively . The bi-level optimization can be formulated as follows , max φ Jφ ( π ∗ ( φ ) ) s.t . π∗ ( φ ) = arg lim π Jπ ( φ ) . ( 16 )

Summary: The paper present a new method, called LToS which enables agents to share rewards in MARL. Two levels of policies, high-level and low-level, determines rewards and optimize global objectives. Three diverse scenarios were used to test the performance of LToS compared to other baseline methods. LToS consistently outperforms other methods. In the second scenario, authors also show the need for high-level policy by introduction fixed LToS.