Multi-agent Deep Reinforcement Learning with Extremely Noisy Observations

1 INTRODUCTION . Reinforcement Learning ( RL ) is concerned with enabling agents to learn how to accomplish a task by taking sequential actions in a given stochastic environment so as to maximise some notion of cumulative reward , and relies on Markov Decision Processes ( MDP ) ( Sutton et al. , 1998 ) . The decision maker , or agent , follows a policy defining which actions should be chosen under each environmental state . In recent years , Deep Reinforcement Learning ( DRL ) , which leverages RL approaches using Deep Neural Network based function approximators , has been proved to achieve human-level performance in a number of applications , mostly gaming environments , requiring an individual agent ( Mnih et al. , 2015 ; Silver et al. , 2016 ) . Many real-world applications , on the other hand , can be modelled as multi-agent systems , e.g . autonomous driving ( Dresner & Stone , 2008 ) , energy management ( Jun , 2011 ; Col , 2011 ) , fleet control ( Stranjak et al. , 2008 ) , trajectory planning ( Bento et al. , 2013 ) , and network packet routing ( Di Caro et al. , 1998 ) . In such cases , the agents interact with each other to successfully accomplish the underlying task . Straightforward applications of single-agent DRL methodologies for learning a multi-agent policy are not well suited for a number of reasons . Firstly , from the point of view of an individual agent , the environment behaves in a highly non-stationary manner as it now depends not only on the agent ’ s own actions , but also on the joint action of all other agents ( Lowe et al. , 2017 ) . The severe non-stationarity violates the Markov assumption and prevents the naive use of experience replay ( Lin , 1992 ) , which is important to stabilise training and improve sample efficiency . Furthermore , only rarely each individual agent has a complete and accurate representation of the entire environment . Typically , an agent receives its own private observations providing only a partial view of the true state of the world . Determining which agent should be credited for the observed reward is also non-trivial ( Foerster et al. , 2018 ) . Training each agent independently , thus effectively ignoring the non-stationarity , is the simplest possible approach . Independent Q-Learning ( IQL ) ( Tan , 1993 ) leverages traditional Q-learning in this fashion , and some empirical success has been reported ( Matignon et al. , 2012 ) despite convergence issues . Tampuu et al . ( 2017 ) has extended IQL for Deep Q-learning to play Pong in a competitive multi-agent setting , and Tesauro ( 2003 ) has addressed the non-stationarity problem by allowing each agent ’ s policy to depend upon the estimates of the other agents ’ policies . The Centralised Training Decentralised Execution ( CTDE ) paradigm has been widely adopted to overcome the non-stationarity problem when training multi-agent systems ( Foerster et al. , 2016 ) . CTDE enables to leverage the observations of each agent , as well as their actions , to better estimate the action-value functions during training . As the policy of each agent only depends on its own private observations during training , the agents are able to decide which actions to take in a decentralised manner during execution . Recently , Lowe et al . ( 2017 ) have combined this paradigm with a Deep Deterministic Policy Gradient ( DDPG ) algorithm ( Lillicrap et al. , 2015 ) to solve Multi-agent Particle Environments , and Foerster et al . ( 2018 ) have used a similar approach integrated with a counter-factual baseline to address the credit-assignment problem . In this article we consider a particularly challenging multi-agent learning problem whereby all agents operate under partial information , and the observations they receive are weakly correlated to the true state of the environment . Similar settings arise in real-world applications , e.g . in cooperative and autonomous driving , when an agent ’ s view of the world at any given time , obtained through a number of sensors , may carry a high degree of uncertainty and can occasionally be wrong . Learning to collaboratively solve the underlying task under these conditions becomes unfeasible unless an appropriate information filtering mechanism is in place allowing only the accurate observations to be shared across agents and inform their policies . The rationale is that , when an observation is accurate , sharing it with others will progressively contribute to form a better view of the world and make more educated decisions , whereas indiscriminately sharing of all the information can be detrimental due to the high level of noise . To keep the setting realistic , we do not assume that the agents are explicitly told whether their private observations are accurate or noisy , rather they need to discover this through experience . We propose a multi-agent deep deterministic policy gradient algorithm enhanced by a communication medium ( MADDPG-M ) to address these requirements . During training , each agent ’ s policy depends on its own observations as well as those explicitly shared by other agents ; every agent simultaneously learns whether its current private observations contribute to maximising future expected rewards , and therefore are worth sharing with others at an given time , whilst also collaboratively learning the underlying task . Extensive experimental results demonstrate that MADDPG-M performs well in highly non-stationary environments , even when the agents acquiring relevant observations continuously change within an episode . As the execution operates in a decentralised manner , the algorithm complexity per time-step is linear in the number of agents . In order to assess the performance of MADDPG-M , we have designed and tested a number of environments as extensions of the original Cooperative Navigation problem ( Lowe et al. , 2017 ) . Each agent is a 2D object aiming to reach a different landmark while avoiding collisions with other agents . We consider two types of settings . In the first one , a single `` gifted '' agent can see the true location of all the landmarks , whilst the other agents receive their wrong whereabouts . In the second one , the agents may have information that is only valuable to other agents , and has to be redirected to the right recipient . We compare MADDPG-M against existing baselines on all the environments and discuss its relative merits and potential future improvements . 2 RELATED WORK . Multi-agent Deep Deterministic Policy Gradient ( MADDPG ) ( Lowe et al. , 2017 ) adopts the CTDE paradigm and builds a multi-agent approach upon DDPG ( Lillicrap et al. , 2015 ) to solve mixed cooperative-competitive environments . No explicit communication is allowed . Instead , centralised training helps agents learn a coordinated behaviour . The CTDE paradigm has been shown to work well on Multi-agent Particle Environments ( Lowe et al. , 2017 ) and StarCraft unit micromanagement ( Foerster et al. , 2018 ) . Extensive efforts have also been spent towards enabling communication amongst agents . The large majority of existing methods enable communication by introducing a differential channel through which the gradients can be sent across agents . For instance , in Differentiable Inter-agent Learning ( DIAL ) ( Foerster et al. , 2016 ) , the agents communicate in a discrete manner through their actions . DIAL uses Q-learning based Deep Recurrent Q-Networks ( DRQN ) ( Hausknecht & Stone , 2015 ) with weight sharing . The algorithm is end-to-end trainable across agents due to its ability to pass gradients from agent to agent over the messages , which requires a communication medium scaling quadratically with the number of agents . This approach has been used to solve problems such as switch riddle where communicating over 1-bit messages is sufficient . Analogously to DIAL , Communication Neural Net ( CommNet ) ( Sukhbaatar et al. , 2016 ) uses a parameter-sharing strategy across agents and defines a differentiable communication channel between them . Every agent has access to this shared channel carrying the average of the messages of all agents . CommNet uses a large single network for all the agents , which may not be easily scalable . Multi-agent Bidirectionally-Coordinated Network ( BiCNet ) ( Peng et al. , 2017 ) adopts a recurrent neural network-based memory to form a communication channel among agents and uses a centralised control policy conditioning on the true state . Other authors have also studied the emergence of language in multi-agent systems ( Lazaridou et al. , 2016 ; Mordatch & Abbeel , 2018 ; Lazaridou et al. , 2018 ) . In these works , the environment typically provides explicit feedback about the communication actions . Unlike existing studies , we do not assume the existence of explicit rewards guiding the communication actions . Our problem therefore requires the hierarchical arrangement of communication policies and local agent policies that act on the environment , which must be learned concurrently . Furthermore , we consider settings where the observations are either wrong or randomly allocated across agents so that , without an appropriate communication strategy , no optimal policies can be learned . 3 BACKGROUND . 3.1 PARTIALLY OBSERVABLE MARKOV GAMES . Partially observable Markov Games ( POMGs ) ( Littman , 1994 ) are multi-agent extensions of MDPs consisting of N agents with partial observations and characterised by a set of true states S , a collection of action sets A = { A1 , . . . , AN } , a state transition function T , a reward function R , a collection of private observation functions Q = { Q1 , . . . , QN } , a collection of private observations O = { O1 , . . . , ON } and a discount factor γ ∈ [ 0 , 1 ) . A POMG is then defined by a tuple , G = 〈S , A , T , R , Q , O , γ , N〉 . The agents do not have full access to the true state of the environment s ∈ S , instead each receives a private partial observation correlated with the true state , i.e . oi = Qi ( s ) : S → Oi . Each agent chooses an action according to a ( either deterministic or stochastic ) policy parameterised by θi and conditioned on its own private observation , i.e . ai = µθi ( oi ) : Oi → Ai or πθi ( ai|oi ) : Oi ×Ai → [ 0 , 1 ] , and obtains a reward as a result of this action , i.e . ri = R ( s , ai ) : S ×Ai → R. The environment then moves into the next state s′ ∈ S according to the state transition function conditioned on actions of all agents , i.e . s′ = T ( s , a1 , . . . , aN ) : S ×A1 × . . .×AN → S . Each agent aims to maximise its own total expected return , E [ Ri ] = E [ ∑T t=0 γ trti ] where r t i is the collected reward by the ith agent at time t and T is the time horizon . 3.2 DETERMINISTIC POLICY GRADIENT ALGORITHMS . Policy Gradient ( PG ) algorithms are based on the idea that updating the policy ’ s parameter vector θ in the direction of ∇θJ ( θ ) maximises the objective function , J ( θ ) = E [ R ] . The current policy πθ is specified by a stochastic function using a set of probability measures on the action space , i.e . πθ : S → P ( A ) . Deterministic Policy Gradient ( DPG ) ( Silver et al. , 2014 ) extends the policy gradient framework by adopting a policy function that deterministically maps states to actions , i.e . µθ : S → A . The gradient used to optimise the objective J ( θ ) is ∇θJ ( θ ) = Es∼D [ ∇θµθ ( a|s ) ∇aQµ ( s , a ) |a=µθ ( s ) ] ( 1 ) where D is an experience replay buffer and Qµ ( s , a ) is the corresponding action-value function . In DPG , the policy gradient takes the expectation only over the state space , which introduces data efficiency advantages . On the other hand , as the policy gradient also relies on ∇aQµ ( s , a ) , DPG requires a continuous policy µ . Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) builds upon DPG by adopting Deep Neural Networks to approximate µ and Qµ . Analogously to DQN ( Mnih et al. , 2015 ) , the experience replay buffer D and target networks µ′ , Qµ′ help stabilise the learning .

Summary: This paper addresses the challenge of learning in extremely noisy environments. The fundamental idea is to combine deep reinforcement learning of individuals, in which individuals can choose whether they share information in order to maximise the overall reward, which is a substantial difference from existing solutions in the area. To achieve this, the authors propose a hierarchical approach in which agents learn from experience, before deciding whether to share information.