Correcting experience replay for multi-agent communication

1 INTRODUCTION . Since the introduction of deep Q-learning ( Mnih et al. , 2013 ) , it has become very common to use previous online experience , for instance stored in a replay buffer , to train agents in an offline manner . An obvious difficulty with doing this is that the information concerned may be out of date , leading the agent woefully astray in cases where the environment of an agent changes over time . One obvious strategy is to discard old experiences . However , this is wasteful – it requires many more samples from the environment before adequate policies can be learned , and may prevent agents from leveraging past experience sufficiently to act in complex environments . Here , we consider an alternative , Orwellian possibility , of using present information to correct the past , showing that it can greatly improve an agent ’ s ability to learn . We explore a paradigm case involving multiple agents that must learn to communicate to optimise their own or task-related objectives . As with deep Q-learning , modern model-free approaches often seek to learn this communication off-policy , using experience stored in a replay buffer ( Foerster et al. , 2016 ; 2017 ; Lowe et al. , 2017 ; Peng et al. , 2017 ) . However , multi-agent reinforcement learning ( MARL ) can be particularly challenging as the underlying game-theoretic structure is well known to lead to non-stationarity , with past experience becoming obsolete as agents come progressively to use different communication codes . It is this that our correction addresses . Altering previously communicated messages is particularly convenient for our purposes as it has no direct effect on the actual state of the environment ( Lowe et al. , 2019 ) , but a quantifiable effect on the observed message , which constitutes the receiver ’ s ‘ social environment ’ . We can therefore determine what the received message would be under the communicator ’ s current policy , rather than what it was when the experience was first generated . Once this is determined , we can simply relabel the past experience to better reflect the agent ’ s current social environment , a form of off-environment correction ( Ciosek & Whiteson , 2017 ) . We apply our ‘ communication correction ’ using the framework of centralised training with decentralised control ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) , in which extra information – in this case the policies and observations of other agents – is used during training to learn decentralised multi- agent policies . We show how it can be combined with existing off-policy algorithms , with little computational cost , to achieve strong performance in both the cooperative and competitive cases . 2 BACKGROUND . Markov Games A partially observable Markov game ( POMG ) ( Littman , 1994 ; Hu et al. , 1998 ) for N agents is defined by a set of states S , sets of actions A1 , ... , AN and observations O1 , ... , ON for each agent . In general , the stochastic policy of agent i may depend on the set of action-observation histories Hi ≡ ( Oi × Ai ) ∗ such that πi : Hi × Ai → [ 0 , 1 ] . In this work we restrict ourselves to history-independent stochastic policies πi : Oi×Ai → [ 0 , 1 ] . The next state is generated according to the state transition functionP : S×A1×. . .×An×S → [ 0 , 1 ] . Each agent i obtains deterministic rewards defined as ri : S × A1 × ... × An → R and receives a deterministic private observation oi : S → Oi . There is an initial state distribution ρ0 : S → [ 0 , 1 ] and each agent i aims to maximise its own discounted sum of future rewards Es∼ρπ , a∼π [ ∑∞ t=0 γ tri ( s , a ) ] where π = { π1 , . . . , πn } is the set of policies for all agents , a = ( a1 , . . . , aN ) is the joint action and ρπ is the discounted state distribution induced by these policies starting from ρ0 . Experience Replay As an agent continually interacts with its environment it receives experiences ( ot , at , rt+1 , ot+1 ) at each time step . However , rather than using those experiences immediately for learning , it is possible to store such experience in a replay buffer , D , and sample them at a later point in time for learning ( Mnih et al. , 2013 ) . This breaks the correlation between samples , reducing the variance of updates and the potential to overfit to recent experience . In the single-agent case , prioritising samples from the replay buffer according to the temporal-difference error has been shown to be effective ( Schaul et al. , 2015 ) . In the multi-agent case , Foerster et al . ( 2017 ) showed that issues of non-stationarity could be partially alleviated for independent Q-learners by importance sampling and use of a low-dimensional ‘ fingerprint ’ such as the training iteration number . MADDPG Our method can be combined with a variety of algorithms , but we commonly employ it with multi-agent deep deterministic policy gradients ( MADDPG ) ( Lowe et al. , 2017 ) , which we describe here . MADDPG is an algorithm for centralised training and decentralised control of multi-agent systems ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) , in which extra information is used to train each agent ’ s critic in simulation , whilst keeping policies decentralised such that they can be deployed outside of simulation . It uses deterministic policies , as in DDPG ( Lillicrap et al. , 2015 ) , which condition only on each agent ’ s local observations and actions . MADDPG handles the nonstationarity associated with the simultaneous adaptation of all the agents by introducing a separate centralised critic Qµi ( o , a ) for each agent where µ corresponds to the set of deterministic policies µi : O → A of all agents . Here we have denoted the vector of joint observations for all agents as o . The multi-agent policy gradient for policy parameters θ of agent i is : ∇θiJ ( θi ) = Eo , a∼D [ ∇θiµi ( oi ) ∇aiQ µ i ( o , a ) |ai=µi ( oi ) ] . ( 1 ) where D is the experience replay buffer which contains the tuples ( o , a , r , o′ ) . Like DDPG , each Qµi is approximated by a critic Q w i which is updated to minimise the error with the target . L ( wi ) = Eo , a , r , o′∼D [ ( Qwi ( o , a ) − y ) 2 ] ( 2 ) where y = ri + γQwi ( o ′ , a′ ) is evaluated for the next state and action , as stored in the replay buffer . We use this algorithm with some additional changes ( see Appendix A.3 for details ) . Communication One way to classify communication is whether it is explicit or implicit . Implicit communication involves transmitting information by changing the shared environment ( e.g . scattering breadcrumbs ) . By contrast , explicit communication can be modelled as being separate from the environment , only affecting the observations of other agents . In this work , we focus on explicit communication with the expectation that dedicated communication channels will be frequently integrated into artificial multi-agent systems such as driverless cars . Although explicit communication does not formally alter the environmental state , it does change the observations of the receiving agents , a change to what we call its ‘ social environment ’ 1 . For agents which act in the environment and communicate simultaneously , the set of actions for each agent Ai = Aei × Ami is the Cartesian product of the sets of regular environment actions Aei and explicit communication actions Ami . Similarly , the set of observations for each receiving agent Oi = Oei × Omi is the Cartesian product of the sets of regular environmental observations Oei and explicit communication observations Omi . Communication may be targeted to specific agents or broadcast to all agents and may be costly or free . The zero cost formulation is commonly used and is known as ‘ cheap talk ’ in the game theory community ( Farrell & Rabin , 1996 ) . In many multi-agent simulators the explicit communication action is related to the observed communication in a simple way , for example being transmitted to the targeted agent with or without noise on the next time step . Similarly , real world systems may transmit communication in a well understood way , such that the observed message can be accurately predicted given the sent message ( particularly if error-correction is used ) . By contrast , the effect of environment actions is generally difficult to predict , as the shared environment state will typically exhibit more complex dependencies . 3 METHODS . Our general starting point is to consider how explicit communication actions and observed messages might be relabelled using an explicit communication model . This model often takes a simple form , such as depending only on what was communicated on the previous timestep . The observed messages omt+1 given communication actions a m t are therefore sampled ( denoted by ∼ ) from : omt+1 ∼ p ( omt+1 | amt ) ( 3 ) Examples of such a communication model could be an agent i receiving a noiseless message from a single agent j such that omi , t+1 = a m j , t , or receiving the message corrupted by Gaussian noise omi , t+1 ∼ N ( amj , t , σ ) where σ is a variance parameter . We consider the noise-free case in the multiagent simulator for all bar one of our experiments , although the general idea can be applied to more complex , noisy communication models . A communication model such as this allows us to correct past actions and observations in a consistent way . To understand how this is possible , we consider a sample from a multi-agent replay buffer which is used for off-policy learning . In general , the multi-agent system at current time t′ receives observations ot′ , collectively takes actions at′ using the decentralised policies π , receives rewards rt′+1 ( split into environmental rewards ret′+1 and messaging costs r m t′+1 ) and the next observations ot′+1 . These experiences are stored as a tuple in the replay buffer for later use to update the multiagent critic ( s ) and policies . For communicating agents , we can describe a sample from the replay buffer D as the tuple : ( oet , o m t , a e t , a m t , r e t+1 , r m t+1 , o e t+1 , o m t+1 ) ∼ D ( 4 ) where we separately denote environmental ( e ) and communication ( m ) terms , and t indexes a time in the past ( rather than the current time t′ ) . For convenience we can ignore the environmental tuple of observations , actions and reward as we do not alter these , and consider only the communication tuple ( omt , a m t , r m t+1 , o m t+1 ) . Using the communication model at time t ′ , we can relate a change in amt to a change in omt+1 . If we also keep track of a m t−1 we can similarly change o m t . In our experiments we assume for simplicity that communication is costless ( the ‘ cheap talk ’ setting ) , which means that rmt+1 = 0 , however in general we could also relabel rewards using a model of communication cost p ( rmt+1 | amt ) . Equipped with an ability to rewrite history , we next consider how to use it , to improve multi-agent learning .

Summary: This paper considers communication games when agents use experience replay. The agents' communication protocol may change over time, leaving outdated symbols in the replay buffer which are then trained on. This paper proposes replacing the old communication actions with up-to-date actions as the transitions are sampled, and shows that this leads to greatly improved convergence speed and higher performance plateaus.