Actor-Attention-Critic for Multi-Agent Reinforcement Learning

1 INTRODUCTION . Reinforcement learning has recently made exciting progress in many domains , including Atari games ( Mnih et al. , 2015 ) , the ancient Chinese board game , Go ( Silver et al. , 2016 ) , and complex continuous control tasks involving locomotion ( Lillicrap et al. , 2016 ; Schulman et al. , 2015 ; 2017 ; Heess et al. , 2017 ) . While most reinforcement learning paradigms focus on single agents acting in a static environment ( or against themselves in the case of Go ) , real-world agents often compete or cooperate with other agents in a dynamically shifting environment . In order to learn effectively in multi-agent environments , agents must not only learn the dynamics of their environment , but also those of the other learning agents present . To this end , several approaches for multi-agent reinforcement learning have been developed . The simplest approach is to train each agent independently to maximize their individual reward , while treating other agents as part of the environment . However , this approach violates the basic assumption underlying reinforcement learning , that the environment should be stationary and Markovian . Any single agent ’ s environment is dynamic and nonstationary due to other agents ’ changing policies . As such , standard algorithms developed for stationary Markov decision processes fail . At the other end of the spectrum , all agents can be collectively modeled as a single-agent whose action space is the joint action space of all agents ( Buşoniu et al. , 2010 ) . While allowing coordinated behaviors across agents , this approach is not scalable due to the action space size increasing exponentially with the number of agents . It also demands a high degree of communication during execution , as the central policy must collect observations from and distribute actions to the individual agents . In real-world settings , this demand can be problematic . Recent work ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) attempts to combine the strengths of these two approaches . In particular , a critic ( or a number of critics ) is centrally learned with information from all agents . The actors , however , receive information only from their corresponding agents . Thus , during testing , executing the policies does not require the knowledge of other agents ’ actions . This paradigm circumvents the challenge of non-Markovian and non-stationary environments during learning . Despite those progresses , however , algorithms for multi-agent reinforcement learning are still far from being scalable ( to a larger number of agents ) and being generically applicable to environments and tasks that are co-operative ( sharing a global reward ) , competitive , or mixed . Our approach extends these prior works in several directions . The main idea is to centrally learn a critic with an attention mechanism . The intuition behind our idea is that in many real-world environ- ments , it is beneficial for agents to know what other agents it should pay attention to . For example , a soccer defender needs to pay attention to attackers in their vicinity as well as the player with the ball , while she/he rarely needs to pay attention to the opposing team ’ s goalie . The specific attackers that the defender is paying attention to can change at different parts of the game , depending on the formation and strategy of the opponent . A typical centralized approach to multi-agent reinforcement learning does not take these dynamics into account , instead simply considering all agents at all timepoints . Our attention mechanism is able to dynamically select which agents to attend to at each time point , improving performance in multi-agent domains with complex interactions . The proposed approach has an input space linearly increasing with respect to the number of agents , as opposed to the quadratic increase in a previous approach Lowe et al . ( 2017 ) . It also works well in co-operative , competitive , and mixed environments , exceeding the capability of some prior work that focuses only on co-operative environments Foerster et al . ( 2018 ) . We have validated our approach on two simulated environments and tasks . We plan to release the code for both the model and the environments after the reviewing period ends . The rest of the paper is organized as follows . In section 2 , we discuss related work , followed by a detailed description of our approach in section 3 . We report experimental studies in section 4 and conclude in section 5 . 2 RELATED WORK . Multi-Agent Reinforcement Learning ( MARL ) is a long studied problem ( Buşoniu et al. , 2010 ) . Topics within MARL are diverse , ranging from learning communication between cooperative agents ( Tan , 1993 ; Fischer et al. , 2004 ) to algorithms for optimal play in competitive settings ( Littman , 1994 ) , though , until recently , they have been focused on simple gridworld environments with tabular learning methods . As deep learning based approaches to reinforcement learning have grown more popular , they have , naturally , been applied to the MARL setting ( Tampuu et al. , 2017 ; Gupta et al. , 2017 ) , allowing multi-agent learning in high-dimensional/continuous state spaces ; however , naive applications of Deep RL methods to MARL naturally encounter some limitations , such as nonstationarity of the environment from the perspective of individual agents ( Foerster et al. , 2017 ; Lowe et al. , 2017 ; Foerster et al. , 2018 ) , lack of coordination/communication in cooperative settings ( Sukhbaatar et al. , 2016 ; Mordatch & Abbeel , 2018 ; Lowe et al. , 2017 ; Foerster et al. , 2016 ) , credit assignment in cooperative settings with global rewards ( Rashid et al. , 2018 ; Sunehag et al. , 2018 ; Foerster et al. , 2018 ) , and the failure to take opponent strategies into account when learning agent policies ( He et al. , 2016 ) . Most relevant to this work are recent , non-attention approaches that propose an actor-critic framework consisting of centralized training with decentralized execution ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) , as well as some approaches that utilize attention in a fully centralized multi-agent setting ( Choi et al. , 2017 ; Jiang & Lu , 2018 ) . Lowe et al . ( 2017 ) investigate the challenges of multiagent learning in mixed reward environments ( Buşoniu et al. , 2010 ) . They propose an actor-critic method that uses separate centralized critics for each agent which take in all other agents ’ actions and observations as input , while training policies that are conditioned only on local information . This practice reduces the non-stationarity of multi-agent environments , as considering the actions of other agents to be part of the environment makes the state transition dynamics stable from the perspective of one agent . In practice , these ideas greatly stabilize learning , due to reduced variance in the value function estimates . Similarly Foerster et al . ( 2018 ) introduce a centralized critic for cooperative settings with shared rewards . Their method incorporates a ” counterfactual baseline ” for calculating the advantage function which is able to marginalize a single agent ’ s actions while keeping others fixed . This method allows for complex multi-agent credit assignment , as the advantage function only encourages actions that directly influence an agent ’ s rewards . Attention models have recently emerged as a successful approach to intelligently selecting contextual information , with applications in computer vision ( Ba et al. , 2015 ; Mnih et al. , 2014 ) , natural language processing ( Vaswani et al. , 2017 ; Bahdanau et al. , 2015 ; Lin et al. , 2017 ) , and reinforcement learning ( Oh et al. , 2016 ) . In a similar vein , Jiang & Lu ( 2018 ) proposed an attention-based actor-critic algorithm for MARL . This work follows the alternative paradigm of centralizing policies while keeping the critics decentralized . Their focus is on learning an attention model for sharing information between the policies . As such , this approach is complementary to ours , and a combination of both approaches could yield further performance benefits in cases where centralized policies are desirable . Our proposed approach is more flexible than the aformentioned approaches for MARL . Our algorithm is able to train policies in environments with any reward setup , different action spaces for each agent , a variance-reducing baseline that only marginalizes the relevant agent ’ s actions , and with a set of centralized critics that dynamically attend to the relevant information for each agent at each time point . As such , our approach is more scalable to the number of agents , and is more broadly applicable to different types of environments . 3 OUR APPROACH . We start by introducing the necessary notation and basic building blocks for our approach . We then describe our ideas in detail . 3.1 NOTATION AND BACKGROUND . We consider the framework of Markov Games ( Littman , 1994 ) , which is a multi-agent extension of Markov Decision Processes . They are defined by a set of states , S , action sets for each of N agents , A1 , ... , AN , a state transition function , T : S × A1 × ... × AN → P ( S ) , which defines the probability distribution over possible next states , given the current state and actions for each agent , and a reward function for each agent that also depends on the global state and actions of all agents , Ri : S × A1 × ... × AN → R. We will specifically be considering a partially observable variant in which an agent , i receives an observation , oi , which contains partial information from the global state , s ∈ S. Each agent learns a policy , πi : Oi → P ( Ai ) which maps each agent ’ s observation to a distribution over it ’ s set of actions . The agents aim to learn a policy that maximizes their expected discounted returns , Ji ( πi ) = Ea1∼π1 , ... , aN∼πN , s∼T [ ∑∞ t=0 γ trit ( st , a1t , .. , aNt ) ] , where γ ∈ [ 0 , 1 ] is the discount factor that determines how much the policy favors immediate reward over long-term gain . Policy Gradients Policy gradient techniques ( Sutton et al. , 2000 ; Williams , 1992 ) aim to estimate the gradient of an agent ’ s expected returns with respect to the parameters of its policy . This gradient estimate takes the following form : ∇θJ ( πθ ) = Ea∼πθ [ ∇θ log ( πθ ( at|st ) ) ∞∑ t′=t γt ′−trt′ ( st′ , at′ ) ] ( 1 ) Actor-Critic and Soft Actor-Critic The term ∑∞ t′=t γ t′−trt′ ( st′ , at′ ) in the policy gradient estimator leads to high variance , as these returns can vary drastically between episodes . Actor-critic methods ( Konda & Tsitsiklis , 2000 ) aim to ameliorate this issue by using a function approximation of the expected returns , and replacing the original return term in the policy gradient estimator with this function . One specific instance of actor-critic methods learns a function to estimate expected discounted returns , given a state and action , Qψ ( st , at ) = E [ ∑∞ t′=t γ t′−trt′ ( st′ , at′ ) ] , learned through temporal-difference learning by minimizing the regression loss : LQ ( ψ ) = Es , a , r , s′ [ ( Qψ ( s , a ) − y ) 2 ] , where y = r ( s , a ) + γEa′∼π ( s′ ) [ Qψ̄ ( s ′ , a′ ) ] ( 2 ) where Qψ̄ is the target Q-value function . To encourage exploration and avoid converging to non-optimal deterministic policies , recent approaches of maximum entropy reinforcement learning learn a soft value function by modifying the policy gradient to incorporate an entropy term ( Haarnoja et al. , 2018 ) : ∇θJ ( πθ ) = Ea∼πθ [ ∇θ log ( πθ ( a|s ) ) ( α log ( πθ ( a|s ) ) −Qψ ( s , a ) + b ( s ) ) ] ( 3 ) where b ( s ) is a state-dependent baseline ( for the Q-value function ) . The loss function for temporaldifference learning of the value function is also revised accordingly with a new target : y = r ( s , a ) + γEa′∼π ( s′ ) [ Qψ̄ ( s ′ , a′ ) − α log ( πθ ( a′|s′ ) ) ] ( 4 ) While an estimate of the value function Vφ ( s ) can be used a baseline , we provide an alternative that further reduces variance and addresses credit assignment in the multi-agent setting in section 3.2 .

Summary: This paper introduces a new method for multi-agent reinforcement learning. The proposed algorithm -- which uses shared critics at training time but individual policies at test time -- makes use of a specialised attention mechanism. The benefits include better scalability (as the dependency of the inputs is linear in the number of agents, rather than quadratic), and also being more amenable to diverse reward and action structures than the previous work.