Agent Modelling under Partial Observability for Deep Reinforcement Learning

1 Introduction . An important aspect of autonomous decision-making agents is the ability to reason about the unknown intentions and behaviours of other agents . Much research has been devoted to this agent modelling problem [ Albrecht and Stone , 2018 ] , with recent works focused on learning informative representations about another agent ’ s policy using deep learning architectures for agent modelling and reinforcement learning ( RL ) [ He et al. , 2016 , Raileanu et al. , 2018 , Grover et al. , 2018 , Rabinowitz et al. , 2018 , Zintgraf et al. , 2021 ] . A common assumption in existing methods is that the modelling agent has access to the local trajectory of the modelled agents during execution [ Albrecht and Stone , 2018 ] , which may include their local observations of the environment state and their past actions . While it is certainly desirable to be able to observe the agents ’ local contexts in order to reason about their past and future decisions , in practice such an assumption may be too restrictive . Agents may only have a limited view of their surroundings , communication with other agents may be infeasible or unreliable [ Stone et al. , 2010 ] , and knowledge of the perception system of other agents may be unavailable [ Gmytrasiewicz and Doshi , 2005 ] . In such cases , an agent must reason with only locally available information . We consider the following question : Can effective agent modelling be achieved using only the locally available information of the controlled agent during execution ? A strength of deep learning techniques is their ability to identify informative features in data . Here , we use deep learning techniques to extract informative features about the trajectory of the modelled agent from a stream of local observations for the purpose of agent modelling . Specifically , we consider a multi-agent setting in which we control one agent which must learn to interact with a set of other agents . We assume a set of possible policies for the non-learning agents and that these policies are fixed . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We propose Local Information Agent Modelling ( LIAM ) 1 , which can be seen as the general idea of learning the relationship between the trajectory of the controlled agent and the trajectory of the modelled agent . In this work we propose one instantiation of this idea : an encoder-decoder agent modelling method that can extract a compact yet informative representation of the modelled agents given only the local information of the controlled agent ( its local state observations , and past actions ) . The model is trained to replicate the observations and actions of the modelled agents from the local information only . During training , the modelled agent ’ s observations and actions are utilised as reconstruction targets for the decoder ; after training , only the encoding component is retained which generates representations using local observations of the controlled agent . The learned representation conditions the policy of the controlled agent in addition to its local observation , and the policy and model are optimised concurrently during the RL learning process . We evaluate LIAM in three different multi-agent environments : double speaker-listener [ Mordatch and Abbeel , 2017 , Lowe et al. , 2017 ] , level-based foraging ( LBF ) [ Albrecht and Stone , 2017 , Papoudakis et al. , 2021 ] , and a modified version of predator-prey proposed by [ Böhmer et al. , 2020 ] . Our results support the idea that effective agent modelling can be achieved using only local information during execution : the same RL algorithm generally achieved higher average returns when combined with representations generated by LIAM than without , and in some cases the average returns are comparable to those achieved by an ideal baseline which has access to the modelled agent ’ s trajectory during execution . We also provide detailed evaluations of the learned encoder and decoder of LIAM as well as comparison with different instantiations of LIAM . 2 Related Work . Learning Agent Models : We are interested in agent modelling methods that use neural networks to learn representations of the other agents . He et al . [ 2016 ] proposed a method which learns a modelling network to reconstruct the modelled agent ’ s actions given its observations . Raileanu et al . [ 2018 ] developed an algorithm for learning to infer an agent ’ s intentions using the policy of the controlled agent . Grover et al . [ 2018 ] proposed an encoder-decoder method for modelling the agent ’ s policy . The encoder learns a point-based representation of different agent trajectories , and the decoder learns to reconstruct the modelled agent ’ s policy . Rabinowitz et al . [ 2018 ] proposed the Theory of mind Network ( TomNet ) , which learns embedding-based representations of modelled agents for meta-learning . Tacchetti et al . [ 2019 ] proposed relational forward models to model agents using graph neural networks . [ Zintgraf et al. , 2021 ] uses a VAE for agent modelling for fully-observable tasks . All these aforementioned works either assume that the controlled agent has direct access to the trajectories of the modelled agent during execution or that the evaluation environments are fully-observable . Xie et al . [ 2020 ] learned latent representations from local information to influence the modelled agents , however , in contrast to our work , they did not utilise the modelled agent ’ s trajectories . Finally , agent modelling from local information has been researched under the I-POMDP model [ Gmytrasiewicz and Doshi , 2005 ] and in the Poker domain research . In contrast to our work , I-POMDPs utilise recursive reasoning [ Albrecht and Stone , 2018 ] which assumes knowledge of the observation models of the modelled agents ( which is unavailable in our setting ) . In the Poker domain , Johanson et al . [ 2008 ] , Bard et al . [ 2013 ] created a mixture-of-expert counter-strategies during training against several type of opponent policies , while during execution they cast the selection of the best counter-strategy against a specific opponent type as a bandit problem . The main difference between LIAM and these works is that the latter adapt ( to select the best strategy ) over a number of hands ( we consider that a hand is equivalent to an episode ) against each opponent . In contrast , in our work we use a single episode for adaptation . Representation Learning in Reinforcement Learning : Another related topic which has received significant attention is representation learning in RL . Using unsupervised learning techniques to learn low-dimensional representations of the environment state has led to significant improvements both in the returns as well as the sample efficiency of the RL . Recent works on representation learning focus on learning world models and use them to train the RL algorithm [ Ha and Schmidhuber , 2018 , Hafner et al. , 2020 ] or learning state representations for improving the sample efficiency of RL algorithms using reconstruction-based [ Corneil et al. , 2018 , Igl et al. , 2018 , Kurutach et al. , 2018 , Gregor et al. , 2019 , Gelada et al. , 2019 ] or non-reconstruction-based methods [ Laskin et al. , 2020 , Zhang et al. , 2020 ] , using contrastive learning or bi-simulation metrics . Works on meta-RL and transfer learning 1We provide an implementation of LIAM in https : //github.com/uoe-agents/LIAM focus on learning representations over tasks and use them to solve new and previously unseen tasks [ Doshi-Velez and Konidaris , 2016 , Hausman et al. , 2018 , Zhang et al. , 2018 , Rakelly et al. , 2019 , Zintgraf et al. , 2019 ] . In contrast to these work , LIAM focuses on learning representations about the relationship between the trajectories of the controlled agent and of the modelled agent . Multi-agent Reinforcement Learning ( MARL ) : MARL algorithms are designed to train multiple agents to solve tasks in a shared environment [ Hernandez-Leal et al. , 2017 , Papoudakis et al. , 2019 ] . At the beginning of the training , a number of agents are untrained and typically initialised with random policies . A common paradigm in MARL is Centralised Training with Decentralised Execution ( CTDE ) , in which the trajectories of all agents are utilised during training , while during execution each agent conditions its individual policy only on its local trajectory . The information of all agents can be utilised during training for various reasons , such as computing a joint value function [ Lowe et al. , 2017 , Foerster et al. , 2018 , Sunehag et al. , 2018 , Rashid et al. , 2018 ] , or generating intrinsic rewards [ Jaques et al. , 2019 ] . During execution , only the local information of each agent is used for selecting actions in the environment . Our work relates to CTDE in that we train LIAM in a centralised fashion using the trajectories of all the existing agents in the environment . However , during execution , the learned model uses only the local trajectory of the controlled agent . 3 Approach . 3.1 Problem Formulation . We control a single agent which must learn to interact with other agents that use one of a fixed number of policies . We model the problem as a Partially-Observable Stochastic Game ( POSG ) [ Shapley , 1953 , Hansen et al. , 2004 ] which consists of N agents I = { 1 , 2 , ... , N } , a state space S , the joint action space A = A1 × ... × AN , a transition function P : S × A × S → [ 0 , 1 ] specifying the transition probabilities between states given a joint action , and for each agent i ∈ I a reward function ri : S × A × S → R. We consider that each agent i has access to its observation oi ∈ Oi , where Oi is the observation set of agent i . The observation function Ωi : S × A ×Oi → [ 0 , 1 ] defines a probability distribution over the possible next observations of agent i given the previous state and the joint action of all agents . We denote the agent under our control by 1 , and the modelled agents by −1 where for notational convenience we will treat the modelled agents as a single “ combined ” agent with joint observations o−1 and actions a−1 . We assume a set of fixed policies , Π = { π−1 , k|k = 1 , ... , K } , which may be defined manually ( heuristic ) or pretrained using RL . Each fixed policy determines the modelled agent ’ s actions as a mapping π−1 , k ( o−1 ) from the modelled agent ’ s local observation o−1 to a distribution over actions a−1 . Our goal is to find a policy πθ parameterised by θ for agent 1 which maximises the average return against the fixed policies from the training set Π , assuming that each fixed policy is initially equally probable and fixed during an episode : arg max θ Eπθ , π−1 , k∼U ( Π ) [ H−1∑ t=0 γtr1t+1 ] ( 1 ) where r1t+1 is the reward received by agent 1 at time t+ 1 after performing the action a 1 t , H is the episode length ( horizon ) , and γ ∈ ( 0 , 1 ) is a discount factor . It is also important to note that neither during training nor during execution the controlled agent has access to the identity k of the policy that is used by the modelled agent .

Summary: This paper proposes an encoder-decoder type neural network for agent modelling in partially observable domains. The method seeks to learn a useful encoding of a controlled agent's action-observation histories to a latent vector. By useful, it is meant that the learned representation used as an input to a downstream RL method allows finding "best response" policies to a set of fixed opponent/other agent policies. At training time, decoders are trained to predict the observations and actions of the other agents. At test time, the decoder is discarded, and the encoder part is used to output latent vectors. The proposed approach is demonstrated in three MARL domains, showing improvement over baselines such as simply classifying other agents' policies, or discarding the agent information completely. Ablation studies confirm the effectiveness of the design choices for the proposed method.