Probabilistic View of Multi-agent Reinforcement Learning: A Unified Approach

1 INTRODUCTION . The traditional reinforcement learning ( RL ) paradigm , that formalizes learning based on trial and error , has primarily been developed for scenarios where a single trainable agent is learning in an environment . In this setup , although the agent changes its behavior as learning progresses , the environment dynamics themselves do not change . Thus , the environment appears to be stationary from the point of view of the learning agent . However , in a setting where several agents are learning in the same environment simultaneously ( multi-agent setting ) , this is not true as a change in one agent ’ s behavior manifests itself as a change in environment dynamics from the point of view of other agents ( Busoniu et al. , 2008 ) . It has been established that stability issues can arise if each agent is independently trained using standard single-agent RL methods ( Tan , 1993 ) . While , in theory , it is possible to treat a collection of multiple agents as a single centralized metaagent to be trained , in practice , this approach becomes infeasible as the action space of the centralized meta-agent grows exponentially with the number of agents . Moreover , executing the resultant centralized policy is not always possible due to various reasons like geographic separation between agents , communication overhead and so on ( Foerster et al. , 2018b ) . Even if these issues are taken care of , when the agents are competitive , designing a reward function for the centralized meta-agent is very challenging and thus , in general , such a setup can not be used with competitive agents . There are numerous practical scenarios that require several intelligent agents to function together ( either cooperatively or competitively ) . Consider , for instance , a soccer game between two teams : agents within a team must cooperate while being competitive with the opponents . Considering that traditional single-agent RL methods can not satisfactorily handle problems from the multi-agent domain , completely new RL algorithms that explicitly acknowledge and exploit the presence of other intelligent agents in the environment are required . In this paper , we pose the multi-agent reinforcement learning problem as the problem of performing probabilistic inference in a particular graphical model . While such a formulation is well known in the single-agent RL setting ( Levine , 2018 ) , its extension to the multi-agent setup is non-trivial especially in the general case where the agents may be cooperative and/or competitive . We model the environment as seen by each of the agents using separate but related Markov Decision Processes ( MDPs ) . Each agent then tries to maximize the expected return it gets from the environment under its own MDP ( Section 4 ) . Using our framework , we derive an off-policy maximum entropy actor-critic algorithm that generalizes the Soft Actor-Critic ( SAC ) algorithm ( Haarnoja et al. , 2018a ) to a multi-agent setup . We refer to this algorithm as Multi-agent Soft Actor-Critic ( MA-SAC ) . Like SAC , it is a maximum entropy algorithm , i.e. , the learned policies try to maximize the rewards while at the same time maximizing entropy of the stochastic actor . Such algorithms are known to be more stable and easier to train ( Haarnoja et al. , 2018a ) . MA-SAC follows the centralized training , decentralized execution paradigm . As we demonstrate in Section 4 , each agent learns its own policy while being actively aware of the presence of other agents . The learned policy of any given agent only utilizes its local observation at test time . Thus , MA-SAC avoids the pitfalls of both independent training of agents ( being unaware of other agents leads to non-stationarity and hence instability ) and training a centralized agent ( centralized policies are hard to execute ) as described above . By setting a tunable temperature parameter ( Section 4.3 ) to zero , MA-SAC yields an algorithm that is very similar to the Multi-agent Deep Deterministic Policy Gradients algorithm ( MADDPG ) ( Lowe et al. , 2017 ) apart from a minor change in updating the actor . The utility of this modification is clearly reflected in our derivation of the inference procedure . When the temperature parameter is non-zero , agents trained using MA-SAC outperform agents trained using MADDPG on multiple cooperative and competitive tasks as we demonstrate in Section 5.3 . Our main contributions are : ( i ) we present a probabilistic view of the multi-agent reinforcement learning problem where each agent models the environment using a separate but related MDP ; ( ii ) we derive an off-policy maximum entropy actor-critic algorithm ( MA-SAC ) by performing structured variational inference in the proposed model ; ( iii ) we empirically demonstrate that MA-SAC performs well in practice and highlight different ways in which our framework can utilize ideas from other existing approaches in multi-agent RL ; and ( iv ) although we only present an actor-critic algorithm in this paper , our framework allows derivation of maximum-entropy variants of other reinforcement learning algorithms in the multi-agent setting . 2 RELATED WORK . In recent years , RL has made significant progress in complicated domains such as game playing ( Mnih et al. , 2015 ; Silver et al. , 2016 ) and robotics ( Levine et al. , 2016 ) to name a few . Inspired by these successes , researchers have tried independently training all agents in a multi-agent setup using single-agent RL algorithms as in the case of independent Q-learning ( Tan , 1993 ) and independent deep Q-learning ( Tampuu et al. , 2017 ) . However , as discussed in Section 1 , such an approach does not perform well in practice . Recently , Foerster et al . ( 2018a ) have proposed a modification to independent Q-learning in an attempt to stabilize the training process . To deal with non-stationarity , agents need additional information to explain the changes in their environment . Existing approaches exploit either ( i ) multi-agent communication or ( ii ) centralized training of decentralized policies to provide the required additional information to agents . In communication based approaches ( Sukhbaatar et al. , 2016 ; Foerster et al. , 2016 ; Lazaridou et al. , 2017 ; Cao et al. , 2018 ) , agents use an emergent language to jointly execute a task . Such approaches require communication between agents even after the training is over . This needs additional computational resources and communication infrastructure which limits the applicability of these methods . The centralized training and decentralized execution paradigm can be thought of as learning with the help of a coach that coordinates activities while training . Once the training process is over , agents can independently take decisions based on their local observations ( Gupta et al. , 2017 ) . These approaches learn centralized critic ( s ) for all agents along with decentalized policies . While some approaches only support cooperative agents ( Sunehag et al. , 2018 ; Foerster et al. , 2018b ; Rashid et al. , 2018 ; de Witt et al. , 2018 ) others support competitive agents as well ( He et al. , 2016 ; Lowe et al. , 2017 ; Iqbal & Sha , 2019 ) . MA-SAC follows the centralized training and decentralized execution paradigm and it supports both cooperative and competitive agents . MA-SAC is a maximum-entropy based algorithm . In the single-agent RL literature , maximumentropy variant of many standard RL algorithms have been proposed like soft-Q-learning ( Haarnoja et al. , 2017 ) and soft-actor-critic ( Haarnoja et al. , 2018a ) . These algorithms are derived by casting the problem of learning optimal behavior as an inference problem in an appropriate graphical model ( Levine , 2018 ) . This not only provides a different perspective on the control problem , it leads to algorithms that are more stable and easier to train ( Haarnoja et al. , 2018a ) . A multi-agent variant of soft-Q-learning exists ( Wei et al. , 2018 ) , but it is only applicable in cooperative environments . Note that MA-SAC is just one of the algorithms that can be derived using our proposed framework . As in the case of single-agent reinforcement learning , in the multi-agent setup as well , one can derive maximum entropy variants of other algorithms based on our proposed probabilistic model . Most relevant to our work is the multi-agent variant of DDPG ( Lillicrap et al. , 2016 ) algorithm ( called MADDPG ) that has been proposed in Lowe et al . ( 2017 ) . MADDPG is based on an actorcritic framework where each agent has its own centralized critic . In Section 4.3 , we show that we recover an algorithm that is very similar to MADDPG by setting a tunable temperature parameter used in MA-SAC to zero . In Section 5 , we show that MA-SAC outperforms MADDPG on several tasks . If one ignores the attention mechanism proposed in Iqbal & Sha ( 2019 ) , the resultant algorithm is equivalent to MA-SAC . While the main contribution in Iqbal & Sha ( 2019 ) is an improvement in scalability of MADDPG using an attention based mechanism in the critic , it is mostly heuristic based and a principled derivation of the algorithm is missing . On the other hand , we provide a unified probabilistic framework for multi-agent RL and use it to derive MA-SAC . 3 NOTATION AND PRELIMINARIES . A Markov Decision Process ( MDP ) , specified by the tuple ( S , A , p , r , γ ) , models an environment with a single trainable agent ( Sutton & Barto , 2018 ) . Here , S is the set of states , A is the set of actions , p : S × A → ∆ ( S ) specifies environment dynamics , r : S × A → R is the reward function and γ ∈ [ 0 , 1 ] is the discount factor . We use ∆ ( S ) to denote the set of all probability distributions over set S. A Markov Game ( MG ) generalizes MDPs to support multiple agents in the environment and is specified by the tuple ( S , { Ai } ni=1 , p , { ri } ni=1 , γ ) ( Littman , 1994 ) . Here , n is the number of agents in the environment . Symbols S and γ have the same meaning as before . Ai and ri : S × A1 × · · · × An → R respectively denote the action space and reward function of agent i . The transition probability function , p : S×A1×· · ·×An → ∆ ( S ) , now considers the actions taken by all agents . The same holds for the reward functions r1 , r2 , . . . , rn . In practical cases , it is also common to assume that each agent can only observe a part of the environment . Let Oi denote the observation space of agent i , then , a function fi : S→ Oi computes the observation of agent i from the environment state S. Note that both MDP and MG simply specify the evolution of environment in response to the behavior of agent ( s ) , and as such , they do not encode any notion of optimal behavior . An agent formulates its own notion of optimality which it strives to achieve by modulating its policy π : S → ∆ ( A ) where , as before , ∆ ( A ) is the set of all probability distributions over the set of actions A . If there are multiple agents , we use πi : S → ∆ ( Ai ) to denote the policy followed by agent i . When the environment is partially observable , the policy of an agent can only take its local observation as input and hence πi : Oi → ∆ ( Ai ) . In single-agent RL , the agent often solves the following optimization problem ( either exactly or approximately ) to learn optimal behavior : J ( π ) = max π Eπ [ T∑ t=0 r ( s ( t ) , a ( t ) ) ] . ( 1 ) The agent acts in the environment only for a finite number of time steps T ( a finite horizon problem ) . The state and action at time t are given by s ( t ) ∈ S and a ( t ) ∈ A respectively . The agent uses its stationary policy to choose an action a ( t ) ∼ π ( ·|s ( t ) ) at each time step . In MARL setting , different agents may have different reward functions . Moreover , because agents act simultaneously in the same environment , the optimal behavior of an agent now depends on the behavior of other agents . The notion of a best response policy of an agent becomes important in this setup . Each agent wants to maximize its own expected sum of rewards but has to do so in the context of the policies being followed by other agents .

Summary: This paper proposes a new algorithm named Multi-Agent Soft Actor-Critic (MA-SAC) based on the off-policy maximum-entropy actor critic algorithm Soft Actor-Critic (SAC). Based on variational inference framework, the authors derive the objectives for multi-agent reinforcement learning. In experiments section, the authors compare the proposed algorithm with the previous algorithm called Multi-Agent Deep Deterministic Policy Gradient (MADDPG) on several multi-agent domain.