Inducing Cooperation via Learning to reshape rewards in semi-cooperative multi-agent reinforcement learning

We propose a deep reinforcement learning algorithm for semi-cooperative multiagent tasks , where agents are equipped with their separate reward functions , yet with willingness to cooperate . Under these semi-cooperative scenarios , popular methods of centralized training with decentralized execution for inducing cooperation and removing the non-stationarity problem do not work well due to lack of a common shared reward as well as inscalability in centralized training . Our algorithm , called Peer Evaluation based Dual DQN ( PED-DQN ) , proposes to give peer evaluation signals to observed agents , which quantifies how they feel about a certain transition . This exchange of peer evaluation over time turns out to render agents to gradually reshape their reward functions so that their action choices from the myopic best-response tend to result in the good joint action with high cooperation . This evaluation-based method also allows flexible and scalable training by not assuming knowledge of the number of other agents and their observation and action spaces . We provide the performance evaluation of PED-DQN for the scenarios ranging from a simple two-person prisoner ’ s dilemma to more complex semi-cooperative multi-agent tasks . In special cases where agents share a common reward function as in the centralized training methods , we show that inter-agent evaluation leads to better performance . 1 INTRODUCTION . Recent advances in reinforcement learning have been focused on the utilization of deep learning techniques . These deep learning methods have been applied to several challenging tasks such as playing games ( Mnih et al. , 2015 ; Silver et al. , 2016 ) , and robotic control ( Gu et al. , 2017 ) . However , most of these recent techniques were developed under the assumption of controlling a single learning agent . Applications such as robot swarms ( Yogeswaran et al. , 2013 ; Huttenrauch et al. , 2017 ) and network routing ( Ye et al. , 2015 ) are better modeled as multi-agent systems , where most RL methods designed for single-agent purposes do not directly translate to their multi-agent counterparts . A prominent reason is that other learning agents are usually regarded as part of the environment , where it is difficult to learn cooperation because a small change in one agent ’ s policy may cause another agent ’ s policy to perform sub-optimally ( i.e. , non-stationarity ) . Many approaches have been proposed in order to induce cooperation among agents using multiagent reinforcement learning . A popular approach , referred to as centralized training and distributed execution ( CTDE ) , assumes that agents are fully cooperative , possibly controlled by a single administrative domain . It uses a centralized training module with a shared reward function that can be detached from the individual policies during execution . Using a centralized module , they address the non-stationarity problem and instill in agents the context of cooperation , which naturally appears in distributed execution after training . Recent related work includes COMA ( Foerster et al. , 2018 ) , VDN ( Sunehag et al. , 2018 ) , QMIX ( Rashid et al. , 2018 ) , CommNet ( Sukhbaatar et al. , 2016 ) , BiCNet ( Peng et al. , 2017 ) , and DIAL ( Foerster et al. , 2016 ) , just to name a few . However , CTDE-based multi-agent reinforcement learning ( MARL ) has a couple of limitations . First , many other multi-agent tasks exist with semi-cooperative agents . A semi-cooperative agent naturally has its own separate reward function ( the shared reward scenario is a special case ) , but is willing to cooperate if an incentive for cooperation1 is appropriately provided . Semi-cooperation is often more practical due to the complexity of defining a global reward function , or the agents are controlled under different administrative domains . Second , CTDE-based methods involve the concerns of scalability and sampling inefficiency . The input for the centralized neural network involves the concatenation of the observations and actions , and optionally , the full state . Since the order of the agents in the input vector is likely to be pre-determined , the number of samples needed in order to obtain all the possible action combinations for the same state increases drastically . The concatenation of each agent ’ s information is also proportional to the number of agents , which is often large depending on the task . This problem can be disregarded if a central state whose dimensionality is not defined by the number of agents can be used as a sole input to the central module such as Central-V ( Foerster et al. , 2018 ) . MADDPG ( Lowe et al. , 2017 ) addresses individual reward functions , yet through ‘ pseudo-centralization ’ by exchanging each agent ’ s observation and action during training , still suffering from inscalability . In this paper , we propose a deep MARL that addresses aforementioned two concerns . The key idea is to look at the problem from the mechanism design ’ s perspective in game theory . In semicooperative sequential tasks , agents will naturally choose the action that is the best response to its current observation and local reward experience . In our approach , we gradually change the game that agents play so that their local-view actions from the best responses become closer to a socially-good joint action . To this end , each agent receives simple peer evaluations in the form of small feedback messages from nearby agents . These peer evaluations quantify how an agent feels about a certain transition , rendering each agent reshape its reward function , so as for agents ’ local best responses to result in the global cooperation . This approach is empowered to be scalable and flexible , because ( i ) the knowledge of the number of existing agents beyond an agent ’ s observation is not required , and ( ii ) this method does not require agents to share states and actions . We instantiate our idea to a value-based algorithm and propose a Peer-Evaluation based Dual DQN ( PED-DQN ) . We use two DQNs trained on a separate timescale , defined by the length of their experience replays . One DQN , called Action DQN ( A-DQN ) , is trained using the evaluated rewards from the shorter experience replay , while another DQN , called Mission-DQN ( M-SDQN ) is trained using the agent ’ s base reward from the long-term experience replay . We use the A-DQN to decide the behavior of the agents , while the M-DQN is used for incentivization . We demonstrate the performance of PED-DQN by comparing it with two variants of QMIX and independent DQN on various levels of cooperation of the predator-prey environment . Related Work Multi-agent reinforcement learning ( MARL ) with decentralized execution has been studied extensively ( Busoniu et al. , 2008 ) . One of the earlier approaches in fully decentralized deep MARL is Tampuu et al . ( 2017 ) that used independent Q-learning in Deep Q-Networks . Other fully decentralized approaches were mainly focused on solving the non-stationarity problem in cooperative agents . This problem is especially prevalent in fully decentralized MARL due to the agents ’ lack of ability to communicate . Omidshafiei et al . ( 2017 ) and Palmer et al . ( 2017 ) proposed a dynamic learning rate that depends on the TD-error . Multiple studies about learning communication protocols have been published ( Das et al. , 2017 ; Lazaridou et al. , 2016 ; Mordatch & Abbeel , 2017 ) . Closely related to our line of research is by Foerster et al . ( 2016 ) which proposed to push gradients from one agent to another through the communication channel . This can be regarded a form of providing a feedback from one agent to another . A significant downside to their approach is inscalability when all agents are allowed to communicate at the same time . A more scalable communication approach was proposed by CommNet ( Sukhbaatar et al. , 2016 ) where agents are allowed to “ negotiate ” before deciding their actions . This is done by having a message output for each neural network layer and then averaging the messages for each agent which will be fed to the next neural network layer . Peng et al . ( 2017 ) suggest using Bi-directional RNNs before selecting the actions . Their approach induces a predetermined flow of information since they used the Bi-directional RNN as the communication channel between agents . More sophisticated approaches use centralized training with decentralized execution . MADDPG ( Lowe et al. , 2017 ) proposed an algorithm , which includes the state-action pairs of the other agents as an input to the critic network . This can be done because the critic network is not used during execution . However , the complexity of training the critic network increases as the number of agent 1In cooperative game theory , Nash bargaining solution ( Nash , 1950 ) and Shapley value ( Shapley , 1953 ) are the axiomatic rules for fair distribution of cooperation incentive . The way of allocating cooperation incentive to agents is beyond the scope of this paper . increases . A similar approach is COMA ( Foerster et al. , 2018 ) , which tries to solve the credit assignment problem . However , COMA assumes a fully-cooperative scenario in which there is a common , shared reward among all the agents . For value-based approaches , QMIX ( Rashid et al. , 2018 ) and VDN ( Sunehag et al. , 2018 ) uses some centralized extension to DQN for value function factorization . Similar to Foerster et al . ( 2016 ) and Mordatch & Abbeel ( 2017 ) , we allow communication among agents . However , instead of using messages as supplementary observations , we use these messages to communicate how an agent evaluates a certain transition . Leibo et al . ( 2017 ) studied the environment where each agent has a separate reward function . In particular , agents face a social dilemma where mutual cooperation may give them higher rewards , but they have no guarantee of other agent ’ s cooperation . Hughes et al . ( 2018 ) attempts to solve these social dilemmas by explicitly reshaping the rewards using inequity aversion terms . Their method induces cooperation from the agent itself , whereas our method reshapes the payoff function of each agent based on peer evaluations . 2 BACKGROUND . 2.1 STOCHASTIC GAME MODEL . We consider a semi-cooperative multi-agent task by modeling it with a stochastic game . We consider n agents , indexed by a ∈ A = { 1 , 2 , ... , n } . We denote a−1 as the set of other agents A \ a . Since we assume the possibility of heterogeneous agents , each agent a has a set of actions Ua . At each time step , each agent has a partial observation oa , which it uses for its policy πa ( ua|oa ) to decide its action ua ∈ Ua . The set of observations and states from all agents at a time is denoted by O and S , respectively . A joint action of all the agents is denoted by u ∈ U , where U : = U1 ⊗ · · · ⊗ Un is the set of all joint actions . We represent the true state as s ∈ S. Once each agent takes its action , forming a joint action , say u , a transition to the next state will occur , which follows a transition function P ( s′|s , u ) . Each agent will then receive a reward ra ( s , u ) that may differ across agents . The agents ’ goal is to maximize its own total expected return Ra = E [ ∑T t=0 γ trta ] where γ is the discount factor and T is the length of an episode . Throughout this paper , we use time step ‘ t ’ in the superscript and agent index ‘ a ’ in the subscript , in all notations having their dependence .

Summary: The paper introduces a DQN based, hierarchical, peer-evaluation scheme for reward design that induces cooperation in semi-cooperative multi-agent RL systems. The key feature of this approach is its scalability since only local “communication” is required -- the number of agents is impertinent; no states and actions are shared between the agents. Moreover this “communication” is bound to be low dimensional since only scalar values are shared and has interesting connections to sociology. Interesting metaphor of “feel” about a transition.