Agent Modelling under Partial Observability for Deep Reinforcement Learning

1 Introduction . An important aspect of autonomous decision-making agents is the ability to reason about the unknown intentions and behaviours of other agents . Much research has been devoted to this agent modelling problem [ Albrecht and Stone , 2018 ] , with recent works focused on learning informative representations about another agent ’ s policy using deep learning architectures for agent modelling and reinforcement learning ( RL ) [ He et al. , 2016 , Raileanu et al. , 2018 , Grover et al. , 2018 , Rabinowitz et al. , 2018 , Zintgraf et al. , 2021 ] . A common assumption in existing methods is that the modelling agent has access to the local trajectory of the modelled agents during execution [ Albrecht and Stone , 2018 ] , which may include their local observations of the environment state and their past actions . While it is certainly desirable to be able to observe the agents ’ local contexts in order to reason about their past and future decisions , in practice such an assumption may be too restrictive . Agents may only have a limited view of their surroundings , communication with other agents may be infeasible or unreliable [ Stone et al. , 2010 ] , and knowledge of the perception system of other agents may be unavailable [ Gmytrasiewicz and Doshi , 2005 ] . In such cases , an agent must reason with only locally available information . We consider the following question : Can effective agent modelling be achieved using only the locally available information of the controlled agent during execution ? A strength of deep learning techniques is their ability to identify informative features in data . Here , we use deep learning techniques to extract informative features about the trajectory of the modelled agent from a stream of local observations for the purpose of agent modelling . Specifically , we consider a multi-agent setting in which we control one agent which must learn to interact with a set of other agents . We assume a set of possible policies for the non-learning agents and that these policies are fixed . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We propose Local Information Agent Modelling ( LIAM ) 1 , which can be seen as the general idea of learning the relationship between the trajectory of the controlled agent and the trajectory of the modelled agent . In this work we propose one instantiation of this idea : an encoder-decoder agent modelling method that can extract a compact yet informative representation of the modelled agents given only the local information of the controlled agent ( its local state observations , and past actions ) . The model is trained to replicate the observations and actions of the modelled agents from the local information only . During training , the modelled agent ’ s observations and actions are utilised as reconstruction targets for the decoder ; after training , only the encoding component is retained which generates representations using local observations of the controlled agent . The learned representation conditions the policy of the controlled agent in addition to its local observation , and the policy and model are optimised concurrently during the RL learning process . We evaluate LIAM in three different multi-agent environments : double speaker-listener [ Mordatch and Abbeel , 2017 , Lowe et al. , 2017 ] , level-based foraging ( LBF ) [ Albrecht and Stone , 2017 , Papoudakis et al. , 2021 ] , and a modified version of predator-prey proposed by [ Böhmer et al. , 2020 ] . Our results support the idea that effective agent modelling can be achieved using only local information during execution : the same RL algorithm generally achieved higher average returns when combined with representations generated by LIAM than without , and in some cases the average returns are comparable to those achieved by an ideal baseline which has access to the modelled agent ’ s trajectory during execution . We also provide detailed evaluations of the learned encoder and decoder of LIAM as well as comparison with different instantiations of LIAM . 2 Related Work . Learning Agent Models : We are interested in agent modelling methods that use neural networks to learn representations of the other agents . He et al . [ 2016 ] proposed a method which learns a modelling network to reconstruct the modelled agent ’ s actions given its observations . Raileanu et al . [ 2018 ] developed an algorithm for learning to infer an agent ’ s intentions using the policy of the controlled agent . Grover et al . [ 2018 ] proposed an encoder-decoder method for modelling the agent ’ s policy . The encoder learns a point-based representation of different agent trajectories , and the decoder learns to reconstruct the modelled agent ’ s policy . Rabinowitz et al . [ 2018 ] proposed the Theory of mind Network ( TomNet ) , which learns embedding-based representations of modelled agents for meta-learning . Tacchetti et al . [ 2019 ] proposed relational forward models to model agents using graph neural networks . [ Zintgraf et al. , 2021 ] uses a VAE for agent modelling for fully-observable tasks . All these aforementioned works either assume that the controlled agent has direct access to the trajectories of the modelled agent during execution or that the evaluation environments are fully-observable . Xie et al . [ 2020 ] learned latent representations from local information to influence the modelled agents , however , in contrast to our work , they did not utilise the modelled agent ’ s trajectories . Finally , agent modelling from local information has been researched under the I-POMDP model [ Gmytrasiewicz and Doshi , 2005 ] and in the Poker domain research . In contrast to our work , I-POMDPs utilise recursive reasoning [ Albrecht and Stone , 2018 ] which assumes knowledge of the observation models of the modelled agents ( which is unavailable in our setting ) . In the Poker domain , Johanson et al . [ 2008 ] , Bard et al . [ 2013 ] created a mixture-of-expert counter-strategies during training against several type of opponent policies , while during execution they cast the selection of the best counter-strategy against a specific opponent type as a bandit problem . The main difference between LIAM and these works is that the latter adapt ( to select the best strategy ) over a number of hands ( we consider that a hand is equivalent to an episode ) against each opponent . In contrast , in our work we use a single episode for adaptation . Representation Learning in Reinforcement Learning : Another related topic which has received significant attention is representation learning in RL . Using unsupervised learning techniques to learn low-dimensional representations of the environment state has led to significant improvements both in the returns as well as the sample efficiency of the RL . Recent works on representation learning focus on learning world models and use them to train the RL algorithm [ Ha and Schmidhuber , 2018 , Hafner et al. , 2020 ] or learning state representations for improving the sample efficiency of RL algorithms using reconstruction-based [ Corneil et al. , 2018 , Igl et al. , 2018 , Kurutach et al. , 2018 , Gregor et al. , 2019 , Gelada et al. , 2019 ] or non-reconstruction-based methods [ Laskin et al. , 2020 , Zhang et al. , 2020 ] , using contrastive learning or bi-simulation metrics . Works on meta-RL and transfer learning 1We provide an implementation of LIAM in https : //github.com/uoe-agents/LIAM focus on learning representations over tasks and use them to solve new and previously unseen tasks [ Doshi-Velez and Konidaris , 2016 , Hausman et al. , 2018 , Zhang et al. , 2018 , Rakelly et al. , 2019 , Zintgraf et al. , 2019 ] . In contrast to these work , LIAM focuses on learning representations about the relationship between the trajectories of the controlled agent and of the modelled agent . Multi-agent Reinforcement Learning ( MARL ) : MARL algorithms are designed to train multiple agents to solve tasks in a shared environment [ Hernandez-Leal et al. , 2017 , Papoudakis et al. , 2019 ] . At the beginning of the training , a number of agents are untrained and typically initialised with random policies . A common paradigm in MARL is Centralised Training with Decentralised Execution ( CTDE ) , in which the trajectories of all agents are utilised during training , while during execution each agent conditions its individual policy only on its local trajectory . The information of all agents can be utilised during training for various reasons , such as computing a joint value function [ Lowe et al. , 2017 , Foerster et al. , 2018 , Sunehag et al. , 2018 , Rashid et al. , 2018 ] , or generating intrinsic rewards [ Jaques et al. , 2019 ] . During execution , only the local information of each agent is used for selecting actions in the environment . Our work relates to CTDE in that we train LIAM in a centralised fashion using the trajectories of all the existing agents in the environment . However , during execution , the learned model uses only the local trajectory of the controlled agent . 3 Approach . 3.1 Problem Formulation . We control a single agent which must learn to interact with other agents that use one of a fixed number of policies . We model the problem as a Partially-Observable Stochastic Game ( POSG ) [ Shapley , 1953 , Hansen et al. , 2004 ] which consists of N agents I = { 1 , 2 , ... , N } , a state space S , the joint action space A = A1 × ... × AN , a transition function P : S × A × S → [ 0 , 1 ] specifying the transition probabilities between states given a joint action , and for each agent i ∈ I a reward function ri : S × A × S → R. We consider that each agent i has access to its observation oi ∈ Oi , where Oi is the observation set of agent i . The observation function Ωi : S × A ×Oi → [ 0 , 1 ] defines a probability distribution over the possible next observations of agent i given the previous state and the joint action of all agents . We denote the agent under our control by 1 , and the modelled agents by −1 where for notational convenience we will treat the modelled agents as a single “ combined ” agent with joint observations o−1 and actions a−1 . We assume a set of fixed policies , Π = { π−1 , k|k = 1 , ... , K } , which may be defined manually ( heuristic ) or pretrained using RL . Each fixed policy determines the modelled agent ’ s actions as a mapping π−1 , k ( o−1 ) from the modelled agent ’ s local observation o−1 to a distribution over actions a−1 . Our goal is to find a policy πθ parameterised by θ for agent 1 which maximises the average return against the fixed policies from the training set Π , assuming that each fixed policy is initially equally probable and fixed during an episode : arg max θ Eπθ , π−1 , k∼U ( Π ) [ H−1∑ t=0 γtr1t+1 ] ( 1 ) where r1t+1 is the reward received by agent 1 at time t+ 1 after performing the action a 1 t , H is the episode length ( horizon ) , and γ ∈ ( 0 , 1 ) is a discount factor . It is also important to note that neither during training nor during execution the controlled agent has access to the identity k of the policy that is used by the modelled agent .

Summary: The authors introduce a method for learning in a multi-agent setting that centers around predicting what other agents will do in the given context. Crucially, global information is not assumed (at execution), and so the techniques introduced have a focus on agent perspective in the information used to drive behaviour.  The methods are evaluated on three domains requiring multi-agent consideration (coordination or combativeness), and the results demonstrate the merit in taking the local perspective for these partial-information domains.