Agent Modelling under Partial Observability for Deep Reinforcement Learning
Authors: Georgios Papoudakis
Filippos Christianos
Stefano V Albrecht
Venue: NeurIPS 2021 Poster (2021)
Keywords: agent modelling
deep reinforcement learning
partial observability
opponent modelling

Abstract: Modelling the behaviours of other agents is essential for understanding how agents interact and making effective decisions. Existing methods for agent modelling commonly assume knowledge of the local observations and chosen actions of the modelled agents during execution. To eliminate this assumption, we extract representations from the local information of the controlled agent using encoder-decoder architectures. Using the observations and actions of the modelled agents during training, our models learn to extract representations about the modelled agents conditioned only on the local observations of the controlled agent. The representations are used to augment the controlled agent's decision policy which is trained via deep reinforcement learning; thus, during execution, the policy does not require access to other agents' information. We provide a comprehensive evaluation and ablations studies in cooperative, competitive and mixed multi-agent environments, showing that our method achieves significantly higher returns than baseline methods which do not use the learned representations.


Agent Modelling under Partial Observability for Deep Reinforcement Learning
Georgios Papoudakis
Filippos Christianos
Stefano V. Albrecht
School Informatics
University of Edinburgh
(g. papoudakis, f.
s s.albrecht eedd ac.uk
Abstract
Modelling the behaviours of other agents is essential for understanding how agents interact and making effective decisions. Existing methods for agent modelling commonly assume knowledge of the local observations and chosen actions of the modelled agents during execution. To eliminate this assumption, we extract representations from the local information of the controlled agent using encoder- decoder architectures. Using the observations and actions of the modelled agents during training, our models learn extract representations about the modelled agents conditioned only on the local observations the controlled agent. The representations used augment the controlled agent's decision policy which is trained via deep reinforcement learning; thus, during execution, the policy does require access other agents' information. We provide comprehensive evaluation and ablations studies in cooperative, competitive and mixed multi-agent environments, showing that method achieves higher returns than baseline methods which do not use the learned representations.
Introduction
An important aspect of autonomous decision-making agents is the ability reason about the known intentions and behaviours of other agents. Much research has been devoted to this agent modelling problem [Albrecht and Stone 2018), with recent works focused learning informative representations about another agent's policy using deep learning architectures for agent modelling and reinforcement learning (RL) [He al. 2016 Raileanu et al. 2018 Grover et al., 2018 Rabinowitz
et al. 2018 Zintgraf et al. 2021].
A common assumption in existing methods is that the modelling agent has access to the local trajectory of the modelled agents during execution [Albrecht and Stone 2018). which may include their local observations of the environment state and their past actions. While is certainly desirable to be able to observe the agents' local contexts in order to reason about their past and future decisions, in practice such an assumption may be too restrictive. Agents may only have limited view their surroundings, communication with other agents may be infeasible or unreliable [Stone et al. 2010). and knowledge of the perception system of other agents may be unavailable [Gmytrasiewicz and Doshi 2005). In such cases, an agent must reason with only locally available information.
We consider the following question: Can effective agent modelling be achieved using only the locally available information of the controlled agent during execution? A strength of deep learning techniques is their ability to identify informative features in data. Here, use deep learning techniques to extract informative features about the trajectory of the modelled agent from a stream of local observations for the purpose of agent modelling. Specifically, in which we control one agent which must learn to interact with set of other agents. We assume of possible policies for the non-learning agents and that these policies fixed.
consider multi-agent setting
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We propose Local Information Agent Modelling (LIAM) which can be seen as the general idea of learning the relationship between the trajectory of the controlled agent and the trajectory of the modelled agent. In this work we propose one instantiation of this idea: an encoder-decoder agent modelling method that can extract a compact yet informative representation of the modelled agents given only the local information of the controlled agent (its local state observations, and past actions). The model is trained to replicate the observations and actions of the modelled agents from the local information only. During training, the modelled agent's observations and actions are utilised as reconstruction targets for the decoder; after training, only the encoding component is retained which generates representations using local observations of the controlled agent. The learned representation conditions the policy of the controlled agent in addition to its local observation, and the policy and model are optimised concurrently during the RL learning process.
We evaluate LJAMM in three different multi-agent environments: double speaker-listener MMrdatcch and Abbeel 2017 Lowe et al. 2017]] level-based foraging (LBF) [Albrecht and Stone [017]] |017 Papoudakis et al. 2021]. and modified version of predator-prey proposed by [Böhmer et al., 2020J. Our
results support the idea that effective agent modelling can be achieved using only local information during execution: the same RL algorithm generally achieved higher average returns when combined with representations generated by LIAM than without, and in some cases the average returns are comparable to those achieved by an ideal baseline which has access to the modelled agent's trajectory during execution. We also provide detailed evaluations of the learned encoder and decoder of LIAM as well as comparison with different instantiations of LIAM.
2 Related Work
Learning Agent Models: We are interested in agent modelling methods that use neural networks to learn representations of the other agents. He et al. (2016) proposed a method which learns a modelling network to reconstruct the modelled agent's actions given its observations, [Raileanu et al.) (2018) developed an algorithm for learning to infer an agent's intentions using the policy of the controlled agent. Grover al. (2018) proposed an encoder-decoder method for modelling the agent's policy. The encoder learns a basnt-based representation of different agent trajectories, and the decoder learns to reconstruct the modelled agent's 's policy. [Rabinowitz et al. [2018) proposed the Theory of mind Network (TomNet), which learns umbedding--asse representations of modelled agents for meta-learning Tacchetti et al. [2019] proposed relational forward models to model agents using graph neural networks. [Zintgraf et al. 202T) uses VAE for agent modelling for fully-observable tasks. All these aforementioned works either assume that the controlled agent has direct access to the trajectories of the modelled agent during execution that the evaluation environments are fully-observable. Xie et al. 2020] learned latent representations from local information to influence the modelled agents, however, in contrast to our work, they did not utilise the modelled agent's 's trajectories. Finally, agent modelling from local information has been researched under the I-POMDP model (Gmytrasiewicz and Doshi 2005) and in the Poker domain research. In contrast to work, I-POMDPs utilise recursive reasoning [Albrecht and Stone 2018) which assumes knowledge of the observation models of the modelled agents (which is unavailable in our setting). In the Poker domain, Johanson et al.) 22008] Bard et al. (2013) created a mixture-of-expprr counter-strategies during training against several type of opponent policies, while during execution they cast the selection of the best counter-strategy against specific opponent type as a bandit problem. The main difference between LIAM and these works is that the latter adapt (to select the best strategy) over number of hands (we consider that a hand is equivalent to an episode) against each opponent. In contrast, in our work we use a single episode for adaptation.
Representation Learning in Reinforcemenn Learning: Another related topic which has received significant attention is representation learning in RL. Using unsupervised learning techniques to learn low-dimensional representations of the environment state has led to significant improvements both in the returns as well as the sample efficiency of the RL. Recent works on representation learning focus on learning world models and use them to train the RI. algorithm (Ha and Schmidhuber) 2018 Hafher ct al. 2020) or learning state representations for improving the sample efficiency of RL algorithms
using reconstruction based [Corneil et al. 2018 Igl ct al. 2018 Kurutach et al. 2018 Gregor et al.
2019
Gelada et al. 2 or non-reconstruction
methods [Laskin et al. 2020 Zhang et al.
2020
using contrastive learning or bi-simulation metrics. Works on meta-RL and transler learning
We provide an implementation of LIAM in htttp:://gtttieii wtwp:////mmmmmmege////t...
2

---

focus on learning representations over tasks and use them to solve new and previously unseen tasks [Doshi-Velee Velez and Konidaris, 2016, Hausman al. 2018 Zhang et al., 2018 Rakelly et al. 2019 Zintgraf et al., 2019 In contrast to these work, LIAM focuses on learning representations about the relationship between the trajectories of the controlled agent and of the modelled agent.
Multi-agent Reinforcement Learning (MARL); MARL algorithms are designed to train multiple agents to solve tasks in shared environment [Hernandez-Lecee
al. 2017 Papoudakis al. 2019]. At the beginning of the training, a number of agents are untrained and typically initialised with random policies. A common paradigm in MARL is Centralised Training with Decentralised Execution (CTDE), in which the trajectories of all agents are utilised during training, while during execution each agent conditions its individual policy only on its local trajectory. The information of all agents can be utilised during training for various reasons, such as computing joint value function
[Lowe et al., 2017 Foerster et al.] 2018 Sunehag et al. 2018 Rashid et al. 20088, or generating intrinsic rewards [laques et al. 2019]. During execution, only the local information of each agent is used for selecting actions in the environment. Our work relates to CTDE in that we train LIAM in centralised fashion using the trajectories of all the existing agents in the environment. However, during execution, the learned model uses only the local trajectory of the controlled agent.
3 Approach
3.1 Problem Formulation
We control single agent which must learn to interact with other agents that use one of fixed number of policies. We model the problem as a Obsteally-Oseevvable Stochastic Game (POSG) (Shapley 1953 Hansen al., 2004) which consists of N agents II = 2,, ...; N), a state space S, the joint action space A = A x x AA a transition function P : S x A x S + [0, 1] specifying the transition probabilities between states given a joint action, and for each agent E I a reward function r': : S x A x S + E. We consider that each agent i has access to its observation of E O', where O' is the observation set of agent i. The observation function O S A x O² + [0, 1] defines probability distribution over the possible next observations of agent i given the previous state and the joint action of all agents.
We denote the agent under our control by 1, and the modelled agents by - where for notational convenience we will treat the modelled agents as a single "combined"" agent with joint observations o-1 and actions a-1 We assume a set of fixed policies, II = ( (m- = 1....,K K}, which may be defined manually (heuristic) or pretrained using RI. Each fixed policy determines the modelled agent's s actions as a mapping π =II from the modelled agent's ' local observation o-1 to a distribution over actions a 1 Our goal is find a policy To parameterised by 0 for agent 1 which maximises the average return against the fixed policies from the training set 11, assuming that each fixed policy initially equally probable and fixed during an episode:
H Σ t=
arg max E 0 To. (I(
(1)
where r141 is the reward received by agent I at time / 4+1 1 after performing the action at. H is the episode length (horizon), and 7 E (0, 1) is a discount factor. It is also important to note that neither during training nor during execution the controlled agent has access the identity is of the policy that is used by the modelled agent.
3.2 Local Information Agent Modelling
We aim to learn the relationship between the trajectory of the controlled agent and the trajectory of the modelled agent. We denote by -- = (o a,," === the trajectory of the modelled agent where or and a, are the modelled agent's 's observation and action time step t in the trajectory, up to horizon H. These trajectories are generated from the fixed policies in II. We assume the existence of some latent variables (or embeddings) in the space 2, and at each time step the latent variables 2, contain information both about the fixed policy that is used by the modelled agent as well as the dynamics of the environment as perceived by the modelled agent. To learn the relationship between the modelled agent's 's trajectory and the latent variables we can use parametric decoder. The decoder
3

---

is denoted as fu Z + -= and is the model that decodes the latent variables to the trajectory of the modelled agent.
The last step to learn the relationship between the local and the modelled agent's trajectories is to recurrent encoding model, that denote as fw 7 - 2, with parameters w, to learn the relationship between the local trajectory of the controlled agent and the latent variables. Specifically, we learn the function that relates the modelled agent's trajectory the latent variables with an encoder that only depends on local information of the controlled agent. Since during execution only the encoder required to generate the latent variables of the modelled agent, this approach removes the assumption that access to the modelled agent's 's observations and actions is available during execution.
At each time step the recurrent encoder net- work generates an embedding 2;, which is condi- tioned on the information of the agent under control (oi.> (1.-1) until time step l. At each time step the parametric decoder learns reconstruct the modelled agent's s observation and action (o and o, 1 condi- tioned on the embedding 2t. Therefore, the decoder consists of a fully-connected feed-forwart network with two output heads; the observation reconstruction head fo and the policy reconstruction head fü (see Figure I). In each time step i, the decoder receives as input embedding 2, and the observation reconstruc- tion head reconstructs the modelled agent's observa- tion or while the action reconstruction head outputs categorical distribution over the modelled agent's s action a We observe that the output dimensions of the two decoder's s reconstruction heads grow linearly with respect the number of the agents in the environment. We refer this method LIAM (Local Information Agent Modelling). LIAM uses the information of both the controlled agent and the modelled agent during training, but during execution only the information of the controlled agent used. The encoder-decoder loss is defined as:
Training
Downatream
(RL)
Execution
Figure I: Diagram of LIAM architecture. The two solid-line rectangles show the compo- nents of LIAM that are used during training and during execution respectively.
CED E o -, log fu (a, [2)]] where 2; = fw
a
(2)
3.3 Reinforcement Learning Training
The latent variables z augmented with the controlled agent's observation can be used to condition the RL optimised policy. Consider the augmented space O Ouug 0 x Z, where O is the original observation space of the controlled agent in the POSG, and is the representation space about the agent's models. The advantage of learning the policy O Oang compared to O is that the policy can specialise for different E Z. In our experiments we optimised the policy of the controlled agent using A2C, however, other RL algorithms could be used in its place. The input to the actor and critic are the local observation and the generated representation. We do not back-propagate the gradient from the actor-critic loss the parameters of the encoder. We different learning rates for optimising the parameters of the networks of RI and the encoder-decoder, We empirically observed that LIAM exhibits high stability during learning, allowing us to use larger learning rate compared to RL. Additionally, we subtract the policy entropy from the policy gradient loss to encourage exploration MMnih et al. 2016). Given batch B of trajectories, the objective of A2C is:
Lacc (10.00)) (r441 VVs(o #++1) Volot =x)
(3)
A log (a( jo), =1) (H lo, =))]
The pseudocode of LIAM is given in Appendix A and the implementation details in Appendix D
Intuitively, at the beginning of each episode, LIAM starts with uninformative embeddings "average" over the possible agent trajectories. At each time step, the controlled agent interacts with the environment and the modelled agent, and updates the embeddings based on the local trajectory that perceives.

---

Figure 2: The three evaluation environments. Double speaker-listener (left), level-based foraging (middle), predator prey (right).
4 Experiments
4.1 Multi-Agent Environments
We evaluate the proposed method in three multi-agent environments (one cooperative, one mixed, one competitive): double speaker-listener [Mordatch and Abbeel 2017 level-based foraging [Albrecht and Stone 2017 Papoudakis et al. [021], and version of predator-prey proposed by [Böhmer et al. 2020]. For each environment, we create ten different policies which are used for training (set II). More details about the process of generating the fixed policies are presented in Appendix
Double speaker-listener (DSL): The environment consists of two agents and three designated landmarks. At the start of each episode, the agents and landmarks are generated in random positions, and are randomly assigned one of three possible colours red, green, blue. The task of each agent is to navigate to the landmark that has the same colour. However, each agent cannot perceive its colour. The colour of each agent can only be observed by the other agents. Each agent must learn both to communicate a -iv-dimensional message to the other agent as well as navigate to the correct landmark. The controlled agent's 's observation includes the relative positions of all landmarks and the other agents as well as the communication message from the previous time step and the colour of the other agent. The cooperative reward each time step the negative average Euclidean distance between two agents and their corresponding correct landmark.
Level-based foraging (LBF): The environment is 20 20 grid-world, consisting of two agents and four food locations. The agents and the foods are assigned random levels and positions at the beginning of an episode. The goal is for the agents to collect all foods. Agents can either move in one of the four directions or attempt to pick pick food if the agents are positioned in the adjacent cells to the food and if the sum of the agents' levels is at least as high as the food's level. The controlled agent has to learn to cooperate to load foods with high level and the same time act greedily for foods that have lower levels. The environment has sparse rewards, representing the contribution of the agent in the gathering all foods in the environment. For example, if the agent receives food with level 2, and there are another three foods with levels 1, and 3 respectively, the reward of the agent is 2/(1 + 2 2 + 0.25. The maximum cumulative reward that all agents can achieve is normalised to The environment and the controlled agent perceives other agents and foods that are located up to four grid cells in every direction. However, the modelled agents can observe the full environment state. The episode terminates when all available foods have been loaded or after 50 time steps.
food. A group of one or more agents successfully
partially-observal
Predator Prey (PP): The environment consists of four agents and two large obstacles. Three of the agents are predators and the other agent is the prey. Each agent has five navigation actions. If only one of the predators captures the prey then the predators receive reward -1 and the prey reward 1. If two more predators capture the prey, then the predators receive reward 1 and the prey reward -1. 1. In the experiments, we control the prey, while the predators sample their policies from the set of fixed policies. The environment is partially-observabiy where the prey can only observe other agents and obstacles that are within receptive field, while the predators observe all agents and obstacles the environment. The episode terminates after 50 time steps.
5

---

4.2 Baselines
Our problem of learning to adapt to different fixed policies can be viewed either as an agent modelling problem, or as single-ages paglleagenttt aasaattttttion RL problem in which we control one agent which has to learn to adapt to multiple tasks, where each one of the fixed policies defines a different task. Thus, task adaptation algorithms can be used as baselines to address our problem. We compare our method against five baselines, two of which are indicative of the upper and the lower performance of LIIM when it is evaluated against the policies from I1. All baselines are trained using the A2C algorithm.
Full Information Agent Model (FIAM): This baseline indicative of the upper performance of LIAM when it is evaluated against the fixed policies from the set II. FIAM utilises the trajectories of the modelled agent during training as well as during execution and is trained similarly to LIAM. However, the encoder is conditioned on the actual trajectory of the modelled agent, and therefore requires access to the modelled agent's 's trajectory during execution. Since FIAM has access to more information than LIAM during execution, we intuitively expect to achieve higher returns compared to LIAM.
No Agent Model (NAM): This baseline does not use an explicit agent model, and it is indicative of the lower performance of LIAM when it is evaluated against the fixed policies from the set 11. does, however, use a recurrent policy network which receives as input the observation and action of the controlled agent. By using a recurrent network we can compare returns achieved by implicit agent modelling (that is done by the hidden states of the recurrent network) with the returns achieved by explicit agent modelling that done by LIAM. NAM is similar to the single-agent task- adastttttt algorithm RL2 [Wang et al. [2166] Duan et al. 2016). In contrast to the original implementation of RI.², we reset the hidden state of the recurrent network at the beginning of each episode, and we do not use the agent's 's reward as input to the policy to ensure consistency among the tested algorithms. Since NAM does not have access to the modelled agent's trajectory during training, we intuitively expect to perform worse than LIAM in terms of achieved returns.
VariBad [Zingrraf al., 2019 This baseline trains a VAE to learn latent representations of the observation and reward Functions, as they are perceived by the controlled agent, conditioned on the observation, action, reward triplet of the controlled agent. We include VariBad for two reasons: is an algorithm that learns to adapt to different tasks, and also it uses recurrent encoder similarly to LIAM, but does not utilise the trajectory of the modelled agent. As a result, we can observe whether utilising the modelled agent's 's trajectory during training, as is done in LIAM, results in higher evaluation returns. To ensure consistency among the tested algorithms, we do not use the reward as input in the encoder of VariBad.
Classification Based Agent Modelling (CBAM): This baseline learns to reconstruct the identity of the policy that is used by the modelled agent. The policy of the controlled agent is conditioned on the reconstructed identity concatenated with the local observation of the controlled agent. CBAM consists of a recurrent network that receives as input the observation-acti agent. The output has a softmax activation function, and we train it to maximise the likelioiddoodd of the identities of the policies that are used by the modelled agent. CBAM uses the fixed policy identities during training (that LIAM does not), but does not use the trajectories of the modelled agent (that LIAM does).
sequence of the controlled
Constrastive Agent Representation Learning (CARL): Finally, we evaluate a non-reconstruction baseline based on contrastive learning [Oord et al. 2018). CARL utilises the modelled agent's s trajectories during training but during execution only the trajectories of the controlled agent are used. Details of the implementation of this baseline are included in Appendix C This baseline was included because recent works [Laskin et al., 2020 Zhang et al., 2020 in state representation found that ronnsssonsttn methods tend to perform better than reconstruction methods, especially in domains that have pixel-based observations.
4.3 Evaluation of Returns
Figure 3 shows the average evaluation returns of all methods during training in the three multi- environments. The returns of every evaluated method are averaged over five runs with different initial seeds, and the shadowed part represents 95% confidence interval (two standard errors of the mean). We evaluate the policy, learned by each method, every 1000 training episodes for 100 episodes. During the evaluation, the agent follows the stochastic policy that is outputted from the

---

Double Speaker Listener
Level-Based Foraging
Predetor Prey
-75
RAN
-233
Retums
-135
Returns
0175
5036
-225
CARL
-253
Time stops
32
62
Time steps
04
3.
0.0
10
Time steps
Figure 3: Episodic evaluation returns and 95% confidence interval of the six evaluated methods during training, against policies from II.
policy network. We found that sampling the action from the policy distribution leads to significantly higher returns compared to following the greedy policy.
First of all, we observe that in all three evaluation environment the returns of LIAM are closer to the upper baseline (FIAM) than to the lower baseline (NAM). The difference in the returns between LIAM and VariBad can be attributed to the fact that LIAM utilises the modelled agent's 's trajectories during training, leading to better representations for RL. VariBad only learns how the current observation of the controlled agent is related to the next observation and reward, while LIAM learns how the local observation is related the trajectory of the modelled agent. CARL performs worse compared to LIAM because the generated embeddings contain less information compared to the embeddings generated from LIAM. For example, in the level-based foraging environment, a specific time step t the control agent observes that the modelled agent is at a specific cell. The observation of the modelled agent also contains this type of information, and as a result, the contrastive task can easily relate that these two embeddings are the two parts of a positive pair. Therefore the representation is trained to only encode such information. On the other hand, LIAM encodes into all information that has been gathered so far in the interaction to be able to reconstruct the modelled agent's 's trajectory. We observe that LIAM achieves higher return than CBAM because identifying the agent identity is not always an informative representation. LIAM utilises the modelled agent's trajectory during training and learns how the local trajectory corresponds to the trajectory of the modelled agent. In the double wpeaker-listener environment, CBAM's returns are higher compared to the rest of the baselines and close to the returns of LIAM, because the different fixed policies are discretely different (different communication messages correspond to different colours) and the classifier learns to accurately identify the fixed policy identity. However, in the other environments the classifier is unable to separate different fixed policies, which results in lower average returns.
4.4 Model Evaluation
After comparing LIAM against five basclines with respect to the average evaluation returns, we now evaluate the encoder and the decoder of LIAM. First, we visualise the embedding space of LIAM and we evaluate how fast the encoder learns to generate informative embeddings. Finally, we evaluate whether the decoder of LIAM learns accurately reconstruct the modelled agent's actions.
We analyse the embeddings learned by LIAM's encoder. Figure 4a presents the wwo-dimensional projection, using the t-SNE method (Van der Maaten and Hinton, 2008). of the generated embeddings at the 20th time step of the episode for all policies where each colour represents a different policy in the set TI. We observe that clusters of similar colours are generated, indicating the interactions with the same fixed policy for the modelled agent result in representations that are close to each other. However, in more than one cluster. We speculate that different clusters can represent the different modalities that exist in the trajectory of the modelled agent, such the colour of the controlled agent and the message that is communicated from the modelled agent to the controlled agent.
in the double speaker-listener environment,
observe that each colour appears
Figure 4b) presents how the action reconstruction accuracy, and the accuracy of the controlled agent identifying its own colour (which is not observed by the controlled agent) in the double speaker- listener environment are changing through time. The colour of the controlled agent is represented as diree-dimensional one-hot vector in the observation of the modelled agent, and the controlled agent does not have direct access to this information during their interaction. At each time step t
7

---

LSNE Embeddings
Reconstruction
Action Reconstruction Accuracy
Accuracy
-11
Episode Time Steps
(a)
(b)
(c)
Figure 4: (a) t-SNE projection of the embeddings in the double speaker-listener environment, where different colours represent different fixed policies and different points represent different episodes. (b) Modelled agent's actions and controlled agent's colours reconstruction accuracy with respect to the episode time steps. (c) Reconstruction accuracy of the modelled agent's actions the 20th time step of the episode in the three evaluation environments (the confidence interval is not visible in the double speaker- listener bar plot).
we use the embedding 2 to reconstruct the observation and the action of the modelled agent. In the reconstructed observation we find the three dimensional vector that corresponds to the colour of the controlled agent, and we consider that the identified colour corresponds to the element of the three-dimensional vector that closer to Then, compute the percentage the times where the reconstructed colour matches the colour of that is truly observed by the modelled agent. LIAM learns to identify the modelled agent's action as well as the colour of the controlled agent with accuracy greater than 90% after 10 time steps.
The decoder of the controlled agent receives as input embedding at each time step, and predicts the observations and actions of the modelled agent. We evaluate the decoder based on action prediction accuracy. At the 20th time step of the episode, reconstruct the modelled agent's actions. Figure 4c presents the average action prediction accuracy and the 95% confidence interval in the three evaluation environments. The reduced action prediction accuracy in level-based foraging and the predator-pres prey compared to double speaker-listener can be explained by the fact that the controlled agent does not always observe the modelled agent, and result it is harder to predict their actions.
the generated embeddings and the decoder to
4.5 Ablation Study
In this section we perform ablation study to jus- tify the design choices behind the architecture of LIAM. We design and evaluate six different ablations of LIAM in the double speaker-listener environment. First, we evaluate two ablated versions of LIAM's s decoder: LIAM-No-Act- Acc--e which reconstructs only the observations of the modelled agent, LIAM- No-Obs-BE Obs-Recon which reconstructs only the actions of the modelled agent. Then, we evaluate how dif- ferent data inputs in the encoder of LIAM affect the returns in the double speaker-listener environment. We consider the model L.AM-No-Act, where the encoder of LIAM we only input the observations of the controlled agent, and the model LIAM-No-Ohh where in the encoder of LIAM we only input the ac- tions of the controlled agent. We also consider an recoder-decoded model that does not utilise the tra- jectory of the modelled agent during training. We call this model L.AM- Local, and at each time step / the decoder learns to reconstruct the next observation of and the action at conditioned on the embedding 2;- Additionally, we evaluate variational mncoder-decoder model [Kingma and Welling 2014) that performs inference to a Hidden Markov Model (HMM), and we refer to it as LIAM- VAE. LIAM-VA VAE learns prior (p(2,) (2)2))1))) that express
Double Speaker Listener
LIKM
LIKA Act Rocon
-32
LUMM-NNe-Att
-14
Returns
L1AM-Loca Local
-16
-10
-22
-24
2.5
2.0
2.5
Time steps
Figure 5: Average returns comparison of LIAM against six ablated versions of LIAM in the double gpeaker-listend environment.
8

---

Colour Reconetrution
Recondy sted Vauue
(a) 0
(b)
(c) t 20
(d)
Figure 6: (a, b. c) Snapshots of one episode of the double speaker- listener environment time steps
0, and 20 respectively. (d) Decoder's reconstruction values the colour of the controlled agent.
the temporal relationship of the latent distribution, similarly to Chung et al. 2015]. We discuss the implementation of this model in Appendix C
Figure 5 presents the average returns of LIAM and the aforementioned ablation baselines in the double wpeaker-listener environment. First, we observe that non-recoen con-reconstructing either the observation or the action of the controlled agent leads to significantly lower returns than the original implementation of LIAM. We also observe that the lack of either the observation or the action of the controlled agent from the encoder of LIAM results in significant lower returns compared full LIAM. The reason is that the model observes how the modelled agent reacts (by observing the relative position of the modelled agent that included in the controlled agent's ohservation) to different communication messages (the communication message is of the actions of the controlled agent) that the controlled agent outputs, and learns to generate informative embeddings (see also Section 4.6). 11AM-Local achieves the lowest returns compared to the other baselines. The main reason behind this result, is that LIAM-Loce does not learn the relationship between the local and the modelled agent's ' trajectory. Finally, we believe that the main reason that LIAM-VAR performs worse than LIAM is that the KL regularisation is too restrictive, which results in less informative representations compared to LIAM. To alleviate this issue, (017) or use the MMD distance [Gretton et al. 2007] for regularisation, similarly to iAE (Zhao et al. 2017). In both cases the resulting objective is not lower bound to the evidence. We did not find any important benefit of using 3 coefficient or the MMD distance compared to the simpler instantiation of LIAM that we present in this work.
can use coefficient to weigh the KL loss, similarly to Higgins al.
4.6 Understanding LIAM
To better understand the working mechanisms of LIAM, we analyse how LIAM performs agent modelling in one episode of the double speaker-listener environment (shown in Figures 6a to 6c). In this episode, the controlled agent has colour red, while the modelled agent has colour blue. The reconstruction of the colour the controlled agent can be seen as a proxy of the belief about its colour, that the controlled agent encodes into 2 each time step. Note that the reconstructed values presented in Figure 6d are not a probability distribution (can be lower than or larger than 1) due to unconstrained optimisation, but consider that the value closest to 1 represents the belief of the controlled agent about its colour. At the time step 0, all reconstructed colours have similar value close to 0.3 meaning that the controlled agent does not have any information about its colour. The controlled agent outputs different messages and observes how the modelled agent reacts to them through the episode. We observe that time step 7 the controlled agent believes that its colour is blue and starts moving toward the blue landmark. After the 10th interaction, the controlled agent has communicated several different messages the modelled agent and by observing how the modelled agent reacts to them, it understands that its belief about its colour was wrong, and it starts moving toward the correct landmark.
Conclusion
We proposed LIAM, the idea of learning models about the trajectories of the modelled agent using the trajectories of the controlled agent. L.IAM concurrently trains the model with decision policy, such that the resulting agent model is conditioned on the local observations of the controlled agent. LIAM is agnostic to the type of interactions in the environment (cooperative, competitive, mixed) and can

---

model an arbitrary number agents, Our results show that LIAM can significantly improve the episodic returns that the controlled agent achieves over baseline methods that do not use agent modelling. The returns that are achieved by A2C when combined with LIAM in some cases nearly match the returns that are achieved by the upper baseline FIAM which assumes access to the trajectories of the modelled agent during execution.
We identify two main limitations of LIAM that are left unaddressed: (a) modelling non-reactive agents, and (b) modelling agents from high-dimensional observations such images. First, a necessary requirement for successfully applying LIAM is for the actions of the modelled agent affect the observations of the controlled agent. If this assumption does not hold, LIAM will not have enough information reason about the modelled agent's 's trajectory and will just learn an average representation over the possible trajectories of the modelled agent. Second, in this work, we focus on environments with low-dimensional vector-based observations. Recent works on state representation for RL in images (Laskin et al. [2200] (Zhang et al., 2020) have shown that reconstruction-bant methods may learn low-quality representations because they weigh the reconstruction of each pixel as equally important even though the vast majority of the pixels contain no relevant information to the modelling problem. Extending LIAM to pixel-based observations can be achieved by using a different architecture for extracting representations, such CARL.
In the future, we would like to investigate how agent modelling from local information can be extended to multi-agent reinforcement learning, where several agents are learning concurrently and non-stationarity arises [Hernandez-Leal et al., 2017 Papoudakis al. 2019]. We would also like to explore notions of "safety" to handle agents that aim to deceive and exploit the agent model [Shoham
ct all 2007 Ganzfried and Sandholm 201 1 2015].
Funding Disclosure
This research was in part supported by the UK EPSRC Centre for Doctoral Training in Robotics and Autonomous Systems (G.P., F.C.), and personal grants from the Royal Society and the Alan Turing Institute (S.A.).
References
Stefano V. Albrecht and Peter Stone. Reasoning about hypothetical agent behaviours and their parameters. In International Conference on Autonomous Agents and Multi-Agent Systems, 2017.
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, pages 66-95, 2018.
Nolan Bard, Michael Johanson, Neil Burch, and Michael Bowling, Online implicit agent modelling. In International Conference on Autonomous Agents and Multi-Agent Systems, 2013.
Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs In International Conference on Machine Learning, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Machine Learning. 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi semi-supervieed learners. In Advances in Neural Information Processing Systems, 2020b.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data, arXiv preprint (al::1506.22220 2015.
Dane Corneil, Wulfram Gerstner, and Johanni Brea. Efficient model-based deep reinforcement learning with variational state tabulation. In International Conference on Machine Learning, 2018.
Finale Doshi-Velez and George Konidaris. Hidden parameter markoy decision processes: A semi- parametric regression approach for discovering latent task parametrizations. International Joint Conference Artificial Intelligence, 2016.
10

---

Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI 2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXi::661..0277, 2016.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence, 2018.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.
Sam Ganzfried and Tuomas Sandholm. Game theory- -aaoey opponent modeling in large imperfect- information games. In International Conference on Autonomous Agents and Multiagent Systems, 2011.
Sam Ganzfried and Tuomas Sandholm. Safe opponent exploitation. ACM Transactions on Economics and Computation, pages 1-28, 2015.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, 2019.
Piotr Gmytrasiewicz and Prashant Doshi. A framework for sequential planning in multi-agent settings. Journal of Artificial Intelligence Research, pages 49-79, 2005.
Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, and Theophane Weber. Temporal difference variational auto-encoder. In International Conference on Learning Representations, 2019.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-probeem, In Advances in Neural Information Processing Systems, 2007.
Aditya Grover, Maruan Al-Shedivat, Jayesh K Gupta, Yura Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International Conference Machine learning. 2018.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, 2018.
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi. and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2020.
Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In AAAI Conference Artificial Intelligence, 2004.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018.
He He, Jordan Boyd-Graber Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforce- ment learning. In International Conference on Machine Learning. 2016
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Pablo Lella, Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A survey of learn- ing in multiagent environments: Dealing with non-stationarity. arXiv preprint MxXiv:170.09138 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
11

---

Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for POMDPs. In International Conference on Machine Learning, 2018.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, 2019.
Michael Johanson, Martin Zinkevich, and Michael Bowling. Computing robust counter-strategies In Advances in Neural Information Processing Systems, 2008.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations 2014.
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart Russell, and Pieter Abbeel. Learning plannable representations with causal infogan. arXiv preprint /r0.v:/807.09:: 2018.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, 2020.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed booerative-cooppettiy environments. In Advances in Neural Information Processing Systems, 2017.
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. International Conference on Machine Learning, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, 2016.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations, In AAAI Conference on Artificial Intelligence, 2017.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018
Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V Albrecht. Dealing with non-stationarity multi-agent deep reinforcement learning. arXiv preprint arXiv: 1906.0406.77777 2019.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Neil C Rabinowitz, Frank Perbet, H Francis Song, Chiyuan Zhang, SM Eslami, and Matthew Botvinick. Machine theory of mind. In International Conference on Machine Learning, 2018.
Roberta Railcanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforeement learning. In International Conference on Machine Learning, 2018
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International Conference Machine Learning, 2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference Machine Learning, 2018.
Jürgen Schmidhuber and Sepp Hochreiter. Long short-term memory. Neural Computation, pages 1735-1788 1780, 1997.
12

---

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 2015.
02438,
Lloyd S Shapley. Stochastic games. Proceedings of the National Academy of Sciences, pages 1095-1100, 1953.
Yoav Shoham, Rob Powers, and Trond Grenager. If multi-agent learning is the answer, what the question? Artificial intelligence, pages 365-377, 2007.
Stone, G.A. Kaminka, Kraus, and J.S. Rosenschein. Ad hoc autonomous agent teams: collabora-
tion without pre-coordination. In AAAI Conference on Artificial Intelligence, 2010.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanetot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. In International Conference on Autonomous Agents and Multi-Agent Systems, 2018.
Andrea Tacchetti, Francis Song, Pedro AM Mediano, Vinicius Zambaldi, Neil C Rabinowitz, Thore Graepel, Matthew Botvinick, and Peter W Battaglia. Relational forward models for multi-agent learning. International Conference on Learning Representations, 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine
Learning Research, pages 2579-2605, 2008
Jane Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.080 1005663, 2016.
Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent represen- tations to influence multi-agent interaction. In Conference on Robot Learning, 2020.
Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learning. arXiv preprint 1804. 10689, 2018.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine Learning invariant representations for reinforcement learning without reconstruction. arXiv preprini arXiv:2006.10742, 2020.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational autoencoders. arXiv preprint ar.iv:
2017.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning, In International Conference on Learning Representations, 2019.
Luisa Zintgraf, Sam Devlin, Kamil Ciosek, Shimon Whiteson, and Katja Hofmann. Deep interactive bayesian reinforcement learning via meta-leamiigg In International Conference Autonomous Agents and Mulli-Agent Systems, 2021.
13


Reviews:
[{"review_id": "nqYhkIRPRL8", "reviewer": "Reviewer_en5v", "summary": "The authors introduce a method for learning in a multi-agent setting that centers around predicting what other agents will do in the given context. Crucially, global information is not assumed (at execution), and so the techniques introduced have a focus on agent perspective in the information used to drive behaviour.\n\nThe methods are evaluated on three domains requiring multi-agent consideration (coordination or combativeness), and the results demonstrate the merit in taking the local perspective for these partial-information domains.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\n\nThe problem setting is certainly interesting. Assumptions of full observability and perspective of all agents, by all agents, is certainly far too strong of an assumption. This work takes a reasonable step to relax that.\n\nThe one area that seems to be lacking from the related work is the growing body on embodied agents. There are a mix of papers at various conferences, but these two workshop series seem directly aligned:\n- https://embodied-ai.org/\n- https://eyewear-computing.org/EPIC_CVPR21/past-editions\n\n\n## Quality\nThe paper is well written, and the quality of the work is good. My biggest complaint on the framing of the solution (which also is an issue with the significance) is the idea that _the perspective of the other agents is available, in full, during training_. Projecting (ala Theory of Mind) during the training process (and evaluation, for that matter) seems to be a far more realistic avenue than the strong assumption that you have access to full agent trajectories when training.\n\n## Clarity\nThe paper is generally very clear. There are a few things that can be improved, however:\n\n1. Lines 45-47 are confusing, and only made sense after having read the full paper at least once.\n2. The plots are often too small to be legible (e.g., Fig 4).\n3. You should emphasize from the beginning that this is a multi-agent setting, and from the perspective of one agent. Partial observability exists in the single-agent setting, and your work is more ambitious than just that.\n4. The \"modelled agent\" used throughout is actually capturing several (all) other agents. I would suggest changing the language here, as it feels as though you are only considering 2-player games.\n\n\n## Significance\nThere are a fair number of assumptions that come into play with this work, and that affects the significance substantially. I applaud the authors for trying to move to the ego-centric setting, as it is far more realistic than the omnipotent view many multi-agent approaches assume, but the assumption of full observability during training remains a very strong one. It is clear how much this assumption helps, given the poor performance of LIAM-Local (a more realistic setup for what a controlling agent would have access to).\n\nAnother one of the assumptions is that the other agents' behaviour depends entirely on the working memory of the controlled agent. There should be elements of what the other agents do/see/experience that influence their behaviour, and in no way is something the controllable agent sees. This is touched upon by the authors as the limitation of needing to be roughly local (so some level of mutual observability is maintained), but manifests as a questionable architecture choice (having the predicted $o_t^{-1}$ based entirely on $z_t$). I get that the other agents' knowledge may be captured somehow in $z_t$, but the only vector for this is through $o_t^{1}$, which feels like a major limitation to me.\n\nThe final major limitation is with the complexity of behaviour on the other agents. There is little justification as to how realistic having pre-determined policies might be, and even with just a handful of individual policies or actions availability, the cross-product in action-space of the collective \"other agent(s)\" can quickly grow out of hand. It's unclear why you need the assumption of a fixed set of policies, rather than simply attempting to learn the function mapping the (projected) observation space of the other agents to their actions.\n\n## Post-Rebuttal Comments\nIt seems as thought my final concern on policy representation is in fact not an issue. I would recommend that the authors make it clear in the intro and section 3.1 text that the fixed policies are not something the local agent has access to.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are some substantial limitations/assumptions on the approach, and the authors address some of them. At this stage in the research, I don't believe there are any major societal impact concerns to address.", "review_text": "", "extra_scores": {"ethical_concerns": "I see no substantial ethical concerns with the paper/research.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "_pOrZCcPUEB", "reviewer": "Reviewer_5w5T", "summary": "The authors propose \"local information agent modeling\" (LIAM), i.e. using a supervised auxiliary predictive task to help learn a representation of the history which is useful to predict other agent's behaviors, and which is then used as a basis for decision making.  The task encoded by LIAM involves predicting the modeled agents' trajectories based on the controlled agent's trajectory.  LIAM is presented both as a general idea, and as a specific implementation which uses encoder-decoder models (controlled agent trajectory -> latent representation -> modeled agent trajectory).  The auxiliary learning task can be performed with supervision in an offline learning framework, where the privileged information of all agents is available.  During execution, the fully trained encoder can be used without requiring the privileged information.\n\nThe experiment section involves 3 multi-agent environments, and 5 baselines closely related to LIAM.  On top of the standard performance-based RL results, the authors include a) a model evaluation section where they analyze the learned latent representations and their ability to encode important information, and b) an ablation study which shows the empirical importance of multiple individual components of their specific LIAM implementation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "EDIT:  The author's response addresses my primary concerns regarding fairness, so I've updated my score based on their response.  (more updates may come based on the committee discussion).\n\n---\n\nThe work seems like a solid advancement in the field of multi-agent control, and the authors have positioned it well in the current literature.  The (offline self-)supervised task encoded by LIAM is fairly simple and perhaps in retrospect an obvious one (which I consider a strength!).  While it is a variant of other similar auxiliary learning tasks, LIAM does address significant drawbacks made by prior work (e.g. full online observability, or (partial) knowledge of the domain model).  In contrast, LIAM primarily makes use of a single but fairly common assupmtion (two more are stated, but it is not clear to me that they are necessary.  More on this later):  the ability to have offline training in a controlled environment where privileged information is available.  The task encoded by LIAM makes ample sense in the specific context of multi-agent RL problems which *require* some form of interaction between agents.  As a bonus, LIAM makes sense in all sorts of generic stochastic multi-agent games, i.e., cooperative, competitive, or mixed games.  The model evaluation and ablation study sections are very welcome additions which complement well the standard performance-based RL results.\n\nNonetheless, both LIAM and the evaluation performed by the authors come with some limitations and lingering questions which in my mind need to be addressed.  My current rating is based on the current submission as-is, under my current understanding and the temporary assumption that there will not be an author response;  I will gladly adjust my rating based on the author's response.\n\nIn order of importance:\n\n1) Some baseline methods use RNNs which receive observation-action sequences (CBAM), while others also receive the rewards as input (NAM and VariBad).  For some methods, it is unclear whether rewards are used or not (LIAM,FIAM,CARL).  I hope this is simply a mistake in the descriptions of the methods, but will pose my question as if the descriptions were accurate.  This setup is very problematic for multiple reasons:\n\n1.a) Such an inconsistency between the information available to each method/baseline wholly undermines the empirical evaluation.  All methods/baselines should access the same information (in principle, either all should observe the rewards, or all should not observe the rewards).  In the general case, I believe that observable rewards are a problem on their own right, as described next.\n\n1.b) \"Observable\" rewards (which are fed into the RNN and influence the choice of future actions) are IMO extremely problematic in partially observable control problems.  The rewards contain information about the underlying state which are not available from the observations alone.  This not only significantly alters our understanding of the problem, but in some cases may allow the agents to significantly bypass the need to develop information-gathering and communication strategies.  E.g., in the DSL environment, being able to observe the reward encodes the agent distances from their goal, and provides a strong signal to be able to locate the landmarks without the need to communicate.\n\n2) The assumption that other agent policies are taken from a fixed set of stationary policies which does not change over time seems to be a relatively strong one;  however, it is not clear to me that this assupmtion is intrinsically necessary.  Is there any reason why this assumption is made, aside from making the RL and LIAM tasks easier?  A natural idea would be to run evaluations where all agents are concurrently trained using LIAM;  any reason why this evaluation was not performed?\n\n3) Treating the policies and behavior of all other agents as a monolithic joint entity is perhaps the biggest limitation of the current implementation as a whole, and one which I must assume makes LIAM scale poorly to the number of agents.  Given that the evaluation environments involve only 1 or 3 other modeled agents, it certainly is not sufficient to confirm generous statements like \"LIAM [...] can model an arbitrary number of agents\".  Is it possible to factorize the predictions of modeled agent trajectories?\n\n4) Another limitation is the assumption that the modeled agent policies are reactive, i.e. that the next actions are solely chosen based on the current local observations (rather than the entire history of each agent's respective local observations).  Again, it is not clear to me that this assumption is intrinsically necessary, except for making the action-prediction problem of LIAM simpler.  Any reasons why more general history-based policies were not considered for the modeled agents?\n\n5) The NAM baseline is described as being a \"lower performance of LIAM\", and I'm not convinced that this is strictly true.  While LIAM does seem like a useful task, there is no reason to believe that it is always strictly better than no auxiliary task at all (as in NAM).  Similarly, I am not convinced that FIAM is indicative \"upper performance of LIAM\".  Again, LIAM and FIAM are similar but ultimately simply different tasks, one more centered on the ability to predict other agents, and the other on the ability to synthesize the agent's own history.  While both are relevant for decision making, claiming that the information encoded by FIAM is strictly better than that encoded by LIAM needs backing.  I would assume that broadly speaking, LIAM, FIAM, and NAM are stronger in different types of domains.  Is there any further evidence for these statements, or am I maybe reading those statements as more strongly than the authors intended?\n\n6) Why is the decoder architecture factored into the observation and policy reconstruction models, as opposed to a single model which outputs both observation and action?  (aside, perhaps, to facilitate the ablation study?).  Is the observation used in the policy reconstruction model always the one generated by the observation reconstruction model, or do you also use the ground truth observation, akin to teacher forcing?\n\nThe submission reads very well, and there are very few minor issues concerning readability:\n\n* l.220:  \"than\" -> \"then\".\n\n* l.270:  Confidence interval of what kind of uncertainty measure?  Standard deviation or standard error?  I assume standard deviation, but this should be stated explicitly.\n\n* Fig.4. (and to a lesser extent, Fig.3):  Labels and ticks are too small and should be enlargened.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The submission addresses fairly generic multi-agent RL settings, rather than specific applications.  I do not believe this work warrants any specific statements about societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "As above, the work is fairly generic and does not intrinsically raise any specific concerns which wouldn't be shared by the RL field as a whole.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "F7uDHe_tUEw", "reviewer": "Reviewer_qnRK", "summary": "This paper proposes an encoder-decoder type neural network for agent modelling in partially observable domains. The method seeks to learn a useful encoding of a controlled agent's action-observation histories to a latent vector. By useful, it is meant that the learned representation used as an input to a downstream RL method allows finding \"best response\" policies to a set of fixed opponent/other agent policies. At training time, decoders are trained to predict the observations and actions of the other agents. At test time, the decoder is discarded, and the encoder part is used to output latent vectors. The proposed approach is demonstrated in three MARL domains, showing improvement over baselines such as simply classifying other agents' policies, or discarding the agent information completely. Ablation studies confirm the effectiveness of the design choices for the proposed method.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Positives\n\n- As measured by the returns, the proposed method has good empirical performance in the domains considered. It improves on several baselines, that seem to be well chosen. It is also interesting to see that simply identifying which of the fixed policies the other agents are using (CBAM) has very limited usefulness, even in simple domains.\n\n- The ablation studies are very nice, showing that the components of the proposed method are indeed useful. The encodings learned are reasonable in the speaker-listener domain. The ablations also clearly make the point that the entire trajectory of actions and observations matter in a partially observable setting.\n\n- It is nice to see that the paper tackles the challenging partially observable setting, rarely considered in other works in agent modelling I am aware of. The paper is very well written and organized, and provides a good overview of recent agent modelling methods.\n\n# Limitations\n\n1. The set of possible opponent policies is assumed fixed. This limits the usefulness of the proposed approach somewhat, at least in competitive settings it is reasonable to expect some adaptation from the opponents.\n\n2. As acknowledged by the authors (Appendix D), the method as presented is unlikely to scale beyond low-dimensional observation/action spaces, as it requires predicting the complete actions and observations of the other agents. It is not obvious to me how representation learning methods could help here.\n\n3. In the domain studied in the paper, the proposed method works well. What could be an interesting future addition is to find some failure cases, which could point out the limits or perhaps features of the domain that make it challenging to apply this type of agent modelling.\n\n4. The proposed LIAM method can be seen as a type of extra regularization, which could also partly explain the authors' observation of its stability. The assumption of this paper is that it is not enough to learn any representation of the controlled agents' action-observation history, but it should be one that enables prediction of the other agents' actions/observations. This suggests a potential additional weakness for the proposed method; in the presences of nuisance/unnecessary information, the method might overregularize to be able to predict useless details of the other agents' actions/observations. This complements point 2. above, and can appear even in low-dimensional settings.\n\n\nOther comments:\n\n- I was initially a bit confused why the POSG is chosen as a formalism, since the paper assumes that the policies of the other agents are fixed. In fact, the problem being solved is a POMDP, specifically searching for the best response policy for the controlled agents while other agents act according to some fixed policies randomly drawn from a set of policies.\n\n- In the ablation study, the VAE method does not seem to help much. The reason stated is the restrictiveness of the KL regularization. But given that the messages received can uniquely be mapped to a specific fixed policy, it seems that there is generally not much to gain from a stochastic VAE? Did you observe the VAE to be helpful in the other domains?\n\n- Line 117 defines an independent observation model for each agent. But I did not see anything that prevents applying this method when the observation model specifies a joint distribution over all agents' observations, so this seems like an opportunity for a small additional generalization.\n\n- Line 87: remove extra \"sample\"\n\n# Post rebuttal\nI thank the authors for their response, in particular for clarifying their ideas regarding representation learning. Overall, this is a nice paper with some interesting observations and contributions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the work are appropriately addressed. Paper does not discuss societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "9gae1T61x79", "reviewer": "Reviewer_Nuzh", "summary": "This paper introduces the Local Information Agent Modelling (LIAM), which learns a latent representation of the other agents. By using an encoder-decoder structure, the latent inference could be performed only based on the ego agent's local information, without knowing the observations and actions of other agents during the test time. The learned latent state is then used to augment the observation of the ego agent for reinforcement learning.  The experiment results show the advantage of LIAM compared with other baselines tested.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The contributions of the paper, to my knowledge, are novel. The method proposed is intuitive and well-explained by the authors. The experiment results and discussions deliver convincing evidence on the advantage of the proposed method. The paper is well-written. The problem that the paper aims to address is important since, in many practical multi-agent problems, the actions and observations of other agents are unknown to the ego agent. \n\nThere are some minor concerns about the paper: \nFirst, simultaneously training the policy and the encoder where the two modules interact with each other may cause training stability issues even the policy is trained under an off-policy RL algorithm. In the paper, the author mentions that they are using different learning rates for the two modules. It would be good if there are some more studies and explanations on this issue. \nSecond, in the ablation study, the author compares the different encoders. I think maybe some move ablation studies on the decoder are also important. For example, instead of predicting both the observation and the action, how would it influence the performance of the encoder only predicts one of them? \nThird, the assumption of a fixed finite set of other agents' policies limits the application of the method and seems unnecessary. Some more extensions would be good. \nFourth, it would be great if the authors could also discuss the scalability of the method.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations discussed by the authors in the appendix are important, and the potential solutions mentioned are reasonable.", "review_text": "", "extra_scores": {"ethical_concerns": "I don't find any ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of partial observability in multi-agent learning, and takes the approach of learning the latent representations of other agents from local observations. This is shown to be beneficial during learning.
The reviewers were generally positive about this paper, which presents an interesting and effective solution to an important problem.
The reviewer scores were improved as a result of the rebuttals, and the authors should include the promised revisions in the paper.
Decision: Accept (Poster)