Local Information Opponent Modelling Using Variational Autoencoders
Authors: Georgios Papoudakis
Filippos Christianos
Stefano V Albrecht
Venue:  (2021)
Keywords: multi-agent systems
opponent modelling
reinforcement learning

Abstract: Modelling the behaviours of other agents (opponents) is essential for understanding how agents interact and making effective decisions. Existing methods for opponent modelling commonly assume knowledge of the local observations and chosen actions of the modelled opponents, which can significantly limit their applicability. We propose a new modelling technique based on variational autoencoders, which are trained to reconstruct the local actions and observations of the opponent based on embeddings which depend only on the local observations of the modelling agent (its observed world state, chosen actions, and received rewards). The embeddings are used to augment the modelling agent's decision policy which is trained via deep reinforcement learning; thus the policy does not require access to opponent observations.  We provide a comprehensive evaluation and ablation study in diverse multi-agent tasks, showing that our method achieves comparable performance to an ideal baseline which has full access to opponent's information, and significantly higher returns than a baseline method which does not use the learned embeddings.

Under review as a conference paper at ICLR 2021
LOCAL INFORMATION OPPONENT MODELLING USING VARIATIONAL AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Modelling the behaviours of other agents (opponents) is essential for understanding how agents interact and making effective decisions. Existing methods for opponent modelling commonly assume knowledge of the local observations and chosen actions of the modelled opponents, which can significantly limit their applicability. We propose new modelling technique based on variational autoencoders, which are trained to reconstruct the local actions and observations of the opponent based on embeddings which depend only on the local observations of the modelling agent (its observed world state, chosen actions, and received rewards). The embeddings are used to augment the modelling agent's decision policy which trained via deep reinforcement learning; thus the policy does not require access to opponent observations. We provide a comprehensive evaluation and ablation study in diverse multi-agent tasks, showing that method achieves comparable performance an ideal baseline which has full access to opponent's 's information, and significantly higher returns than baseline method which does not use the learned embeddings.
INTRODUCTION
An important aspect of autonomous decision-makenn agents the ability to reason about the unknown intentions and behaviours of other agents. Much research has been devoted to this opponent modelling problem [2], with recent works focused on the use of deep learning architectures for opponent modelling and reinforcement learning (RL) [20, 34, 16, 33).
common assumption in existing methods that the modelling agent has access to the local trajectory of the modelled agents [2], which may include their local observations of the environment state, their past actions, and possibly their received rewards. While certainly desirable to be able to observe an agent's local context in order to reason about its past and future decisions, in practice such an assumption may be too restrictive. Agents may only have a limited view of their surroundings, communication with other agents may not be feasible reliable [40], and knowledge of the perception system of other agents may not be available [13]. In such cases, an agent must reason with only locally available information.
We consider the question: Can effective opponent modelling be achieved using only the locally available information of the modelling agent during execution? strength of deep learning techniques is their ability to identify informative features in data. Here, we use deep learning techniques to extract informative features from a stream of local observations for the purpose of opponent modelling.
Specifically, we consider multi-agent settings in which we control a single agent which must learn to interact with a set of opponent agents (we use the term "opponent" in a neutral sense). We assume a given set of possible policies for opponent agents and that these policies are fixed (that is, other agents do not simultaneously learn, such as in multi-agent RL [32]). We propose an opponent modelling method which is able to extract compact yet informative representation of opponents given only the local information the controlled agent, which includes its local state observations, past actions, and rewards. To this end, we use encoder-decoder architecture based on variational autoencoders (VAE) [26]. The VAE model is trained to replicate opponent actions and observations from the local information only. During training, the opponent's observations are utilised reconstruction targets for the decoder; after training. only the encoder component is retained which generates embeddings using local observations of the controlled agent. The learned embeddings condition the policy of the

---

Under review as a conference paper at ICLR 2021
controlled agent in addition to its local observation, and the policy and VAE model are optimised concurrently during the RL learning process.
We evaluate our proposed method, called Local Information Opponent Modelling (LIOM), in two benchmark environments used in multi-agent systems research, the multi-agent particle environment [31, 28] and level-based foraging (LBF) [1]. Our results support the idea that effective opponent modelling can be achieved using only local information during execution: the same RL algorithm generally achieved higher average returns when combined with our opponent embeddings than without, and in some cases the average returns are comparable to those achieved by an ideal baseline which has full access to the opponent's trajectory. We evaluate the method's ' ability to predict the opponent's 's actions, and provide an ablation study on the different types of local information used by the encoder.
2 RELATED WORK
Learning Opponent Models: We are interested in opponent modelling methods that use neural networks to learn representations of the opponents. He et al. [20] proposed a method which learns a modelling network reconstruct an opponent's actions given the opponent's observations. Raileanu et al. [34] developed an algorithm for learning to infer an opponents' intentions using the policy of the controlled agent. Grover et al. [16] proposed an encoder-decoder method for modelling the opponent's 's policy. The encoder learns point- representation of different opponent trajectories, and the decoder learns to reconstruct the opponent's s policy. In addition, the authors introduced an objective to separate embeddings of different agents into different clusters. Rabinowitz et al. [33] proposed the Theory of mind Network (TomNet), which learns embedding-basee representations of opponents for meta-learning. Tacchetti et al. [42] proposed relational forward models to model opponents using graph neural networks. A common assumption in these methods, which our work aims to eliminate, is that the modelling agent has full access to the opponent's 's local information during execution, including their observations, chosen actions, and received rewards.
Opponent modelling from local information has been researched under the I-POMDP model [13] and in the Poker domain research. In contrast to our work, I-POMDPs utilise recursive reasoning [2] which assumes knowledge of the observation models of the modelled agents (which is not available in our setting). In the Poker domain, Johanson et al. [24] proposed Restricted Nash Response (RNR) for computing robust counter-strategin to opponents. Additionally, they generate a mixture-of-experts counter-stran sounter-trateggess to various opponents. During execution, the UCB1 algorithm [4] is used to adapt and select the appropriate counter- sunter--trategy out of the mixture against each specific and previously unknown opponent. Bow et al. [7] propose a method for online evaluation of an agent's 's strategy using importance sampling for reducing the variance of the estimation. Bard al. [5] combined several ideas from the aforementioned works to build complete Poker agent system. Their method creates mixture-of-experts strategies, and during execution they deploy Exp4 [3] for online adaptation (selection of the best strategy from the mixture) to each opponent. The aforementioned works do not require any access to the opponent's 's observations during exccution. The main difference between our method and these works is that the latter require number of online adaptation episodes (to select the best strategy) against each opponent. In contrast, in our work we use a single episode for adaptation.
Representation Learning in Reinforcement Learning: Another related topic which has received significant attention is representation learning in RL. Using unsupervised learning techniques to learn dimenssooio representations of the environment state has led to significant improvements in RL. Ha and Schmidhuber [18] proposed a VAE-based model and a forward model to learn state representations of the environment. Hausman et al. [19] learned task embeddings and interpolated them to solve more difficult tasks. Igl et al. [22] used VAE model for learning state representations environments. Gupta et al. [17] proposed a model which learns Gaussian embeddings to represent different tasks during meta-training and manages to quickly adapt to new task during meta-testing, Gregor et al. [14] developed a VAE-based model for long-term state predictions. The work of Zintgraf et al. [44] is closely related, where the authors proposed a recurrent VAE model which receives as input the observation, action, reward of the controlled agent and learns a variational distribution of tasks. Rakelly et al. [35] used representations from an encoder for off- off-policy meL Note that all of these methods were designed for learning representations of tasks or properties of the environment. In contrast, our approach focuses on learning representations of opponents.
in partially-obses
2

---

Under review as a conference paper at ICLR 2021
Multi-agent Reinforcement Learning (MARL): This term is used to describe the learning proce- dure of multiple agents in a shared multi-agent environment. At the beginning of the training, the agents are usually untrained and initialised with random policy. Using an opponent's 's trajectory during training but not during execution, is a common MARI. paradigm, called Centralised Training Decentralised Execution (CTDE). The opponent's ' information can be utilised during training for various reasons, such as computing a joint value function [28, 9, 41, 36], or generating intrinsic rewards [23]. However, during execution only the local information of each agent used for selecting actions in an environment. While our work draws connection with CTDE, we consider a different problem than the one that MARL addresses. In this work, we have set of fixed opponent policies, and we train a single agent to model and interact with the policies in this set, without accessing the opponent's s trajectory during execution.
TECHNICAL PRELIMINARIES
3.1 REINFORCEMENT LEARNING
We model the decision problem as a Markov Decision Processes (MDP). An MDP consists of a set of states S, a set of actions A. a transition function, P(s'|s [s. a). specifying the probability of the next state, 8', after taking action in state 8, and reward function, r(s', a, s), which returns a scalar value conditioned on two consecutive states and the intermediate action. A policy function used to choose an action in given state, which we assume can be stochastic,
(alss) Given policy T, the value function defined VT(ss) E [- 2 'r, st] and the action-value (Q-value) function Q ae) = Ex |> 7 = St, a = ar), where 0 S 7 S 1 is the discount factor and H is the finite horizon of the episode. RL methods aim to compute an optimal policy that maximises the state value functions. In this work. we will use an algorithm called Synchronous Advantage Actor-Critic (A2C) [30, 8). A2C is an on-policy actor-critic algorithm which uses parallel environment copies to break the correlation between consecutive experience samples. Assuming policy To and value network V with parameters O and Φ, respectively, the actor-critic parameters are optimised via
min BBl- A log (alass) + 2 + (V7(5) - (=( 0.6
(1)
where B is the batch of transitions, B = {(8t, (t, (11))) |= and A. r VV (x)) is the basic advantage term (in experiments we use Generalised Advantage Estimation (GAE) [38]; see Appendix D).
3.2 VARIATIONAL AUTOENCODER
Consider samples from a dataset & E X' which are generated from some hidden (latent) random variable based a generative distribution (=(=)2) with unknown parameter u. and prior distri- bution on the latent variables which we assume to be Gaussian with zero mean and unit variance, p(z) [[;; 0, f). We seek to approximate the true posterior p(=|xx) with a variational parametric distribution Iw (a(z) = . (22; μr, Σ, w). Kingma and Welling [26] proposed the Variational Autoen- coder (VAE) to learn this distribution. Starting with the Kullback-Leiblee (KL) divergence from the [Ip(2)
approximate to the true posterior, DKL(qwd (2/z)) ())))), the lower bound on the evidence log p(z)
(ELBO) is derived as:
logp(z) 2 (xlay (logg Pa (a) 2)] - (1)
(2)
Maximising the ELBO leads to minimisation of the KI. divergence from the approximate to the true posterior. The architecture consists of two neural networks: the encoder which receives sample I and generates the Gaussian variational distribution q(>|<;; w); and the decoder which receives a sample from the Gaussian variational distribution and reconstructs the generative distribution pa. (zr|z). The architecture is trained using the reparameterisation trick [26]. Zhao et al. [43] noticed that the KL divergence can lead to an uninformative posterior and proposed the use of the Maximum Mean Discrepancy (MMD) [6, 15] for forcing the posterior to be close to the prior.
DMMD (g(2) (lp())) = Ex,s'~glo (M2gggM(2222, 2 + Eg,e' [6(2, 2')] 2E [k(2, e/)]
(3)
where k(2,z') is Gaussian kernel.
3

---

Under review as a conference paper at ICLR 2021
4 APPROACH
4.1 PROBLEM FORMULATION
We control single agent which must learn interact with other agents (opponents) that use one of a fixed number of policies. We model this as a Markov game [27] which consists of N agents I= [1, .... N). a state space S, the joint action space A Ar x ... AN, transition function P: x A x S > [0. 1] specifying the transition probabilities between states given joint action, and for each agent E reward function ri S A x + R. We consider partially-observable settings, where each agent i has access only to its local observation o, C E S and reward We denote the agent under our control by 1, and the opponent agents by -1 where for notational convenience we will treat the opponent agents
single "combined" agent with joint observations 0-1 and actions a-1- We assume a set of opponent policies, IT_1 = { [k: = 1.... K), which may be defined manually (heuristic) pretrained using RI. Each opponent policy determines the opponent agent's actions as a mapping Th (o_1) from the opponent's 's local observation o-1 to a distribution over actions a-1- Our goal is to find a policy we parameterised by 0 for agent which maximises the average return against opponents from the training set II_1, assuming that each opponent policy is
initially equally probable and fixed during an episode:
H Σ
arg max (
(1(
7 Ti.
(4)
where ri, is the reward received by agent at time t, H is the episode length (horizon), and y E (0, is a discount factor.
4.2 LOCAL INFORMATION VARIATIONAL AUTOENCODERS
We denote by T-1 f0-1,; a-1, 1= an opponent trajectory where 0-1.t and a-1,t are the opponent's 's observation and action at time step in the trajectory, up to horizon H. These trajectories are generated from the opponent policies in II_1, which are represented in a latent (or embedding) Z.
space Z. Additionally, we assume that there exists unknown generative model Pu Pu(T-1l=), The latent variable 2 contains information about the trajectory of the opponent. Our approach is to solve the problem defined in Section 4.1 by performing reinforcement learning in the joint space of the observation space of our agent and the latent space of the opponent, We aim to approximate the unknown posterior, p(=/T-1), using variational Gaussian distribution N ((w Ew,) with parameters w. As a result, during execution, we can sample the latent variable from the approximate posterior N(;; pw: ).
In Sections 1 and 2, it was noted that most agent modelling methods assume access to the opponent's observations and actions both during training and execution. To remove this as- sumption during execution, we propose a VAE that uses parametric variational distribution which is conditioned on the observatioon-aciio--eewws tw (2/mi,t (01,1:t, 10,1:2-1,11,::-1))
triplet of the agent under our control; Specifically, we approximate the true posterior that is conditioned on opponent's information, with a variational distribution that only depends on local information. We start by writing the KL divergence from the approximate to the true posterior:
(5)
By following the works of Kingma and Welling [26] and Zhao et al. [43], the loss can be written as: Ez (#|m1) log Pa [T=1 + (2)))
L(T1,T-1; w. u.
(6)
From Equation we observe that the variational distribution depends only on locally available infor- mation. Since during execution only the encoder is required to generate the opponent's 's model, this approach removes the assumption that access to the opponent's observations and actions is available during execution. At each time step t, the recurrent encoder network generates latent sample 2;, which is conditioned on the information of the agent under control (01,1:t, Q1,1:-1, 71,1:-1). until time step t.
The first term of the VAE loss consists of the reconstruction loss the opponent's 's trajectory which involves the observations and actions of the opponent. The opponent's observation depends on the
4

---

Under review as a conference paper at ICLR 2021
Training
N
Execution
Figure 1: Diagram of LIOM architecture.
dynamics of the environment. The opponent's action at each time step depends on its observation and the opponent's 's identity, which is represented by the latent variable Therefore, the decoder consists of two cally-connected werward networks.
H
log Pa (T-(T112) Σ 1o = Σ pooo t=1
(7)
From Equation 7, we observe that conditioned on the latent variable, the decoder reconstructs the opponent's 's observation and the opponent's action given the opponent's observation. Intuitively, 2 encodes the type of policy used by the opponent at the current episode and its observation time step t. Figure illustrates the proposed VAE as well as the information that requires during training and execution. We refer to this method LIOM (Local Information Opponent Modelling). LIOM uses the information of both the controlled agent and opponent during training, but during execution only the information of the controlled agent used.
4.3 REINFORCHMENT LEARNING TRAINING
We use the latent variable augmented with our agent's observation to condition the policy of our agent, which optimised using RL. Consider the augmented observation space O' O 2, where O is the original observation space of our agent in the Markov game, and Z is the representation space of the opponent models. The advantage of learning the policy on O' compared O is that the policy can adapt to different algorithm (cf. Sec. 3. Note that other RL algorithms could used in place of A2C. The input to the actor and critic the local observation and the mean of the variational distribution. We do not back-pun back-propagate the gradient from the actor-critic loss (Equation 1) to the parameters of the encoder. We different learning rates for optimising the parameters of the networks of A2C and LIOM. LIOM is a VAE model, and we empirically observed that it exhibits high stability during learning, allowing us larger learning rate compared RL. Additionally, we subtract the policy entropy from the policy gradient loss to encourage exploration [30]. Given a batch B of collected trajectories in the environment, the update equation for our proposed method is the following:
Z. We optimise the policy of the controlled agent, using the A2C
min Ep 0.0
(ri + /Voloi m((')) - Vo ((=))) Å log tola
(m((((allor (8)
2
min Ea 00,04
log pm (a- la + AD (2)
(9)
The pseudocode of LIOM is given in Appendix A. Intuitively, the beginning of each episode, LIOM starts with an uninformative posterior which is equal to the prior (isotropic Gaussian) over the possible opponents. At each time step, the agent interacts with the environment and the opponent and updates the posterior over opponents based on the local information that it perceives.
5 EXPERIMENTS
We evaluate LIOM in several multi-agent tasks and compare the average returns during RL training against two baselines. We evaluate the embeddings learned by LIOM's encoder and the reconstruction accuracy of the decoder. Finally, we test the stability of LIOM with respect to different inputs in the encoder and the effect of the number of opponent policies in the returns of LIOM.
5

---

Under review as a conference paper at ICLR 2021
5.1 MULTI-AGENT ENVIRONMENTS
Speaker-listener: the environment consists of two agents, called speaker and listener, well as three designated landmarks. At the start of each episode, the listener and landmarks are generated in a random position, and are randomly assigned one of three possible colours red, green, or blue. The task of the listener is to navigate to the landmark that has the same colour as the listener. However, the colour of the listener can only be observed by the speaker, thus the speaker has to learn to communicate the correct colour to the listener. The speaker observes the colour of the goal landmark as a one-hot encoded vector and outputs a 5-bit binary communication message as an action. The listener observes the relative positions of all landmarks and the communicated message of the previous time step and can choose to navigate using the actions forward, backward, right, left, no-op. The speaker observes only the colour of the listener. The reward at each time step is the negative Euclidean distance between the listener and the correct landmark.
Double speaker-listener: the environment consists of two agents and three designated landmarks, similarly to the speaker-listener environment. The only difference is that both agents are simulta- neously speakers and listeners. Therefore, the beginning of the episode, each agent has colour that can only be observed by the other agent. Each agent must learn both to communicate a message to the other agent as well as navigate the correct landmark. The agent's observation includes the relative positions of all landmarks and the other agent as well as the communication message from the previous timestep and the colour of the other agent. Each agent performs both actions from the previous enviromment; it communicates the opponent's ' goal landmark and navigates to its own. The reward at each time step is the negative average Euclidean distance between each agent and the corresponding correct landmark.
Level-based foraging (LBF); the environment is a 20 x 20 grid- groldw consisting of two agents and four food locations. The agents and the foods are assigned random levels and positions at the beginning of an episode. The goal is for the agents to collect all foods. Agents can either move in one of the four directions or attempt to pick up food. A group of one or more agents successfully pick a food if the agents are positioned in the adjacent cells to the food and if the sum of the agents' levels is at least as high as the food's level. The controlled agent has to learn to cooperate to load foods with a high level and at the same time act greedily for foods that have lower levels. The environment has sparse rewards, representing the contribution of the agent in the gathering all foods in the environment. For example, if the agent receives a food with level 2, and there are another three foods with levels 1, 2 and 3 respectively, the reward of the agent is 21 + 2 2 + 3) = 0.25. Thus, the maximum cumulative reward that both agents can achieve is normalised to 1. The environment is where the agent observes up to four grid cells in every direction, and results, it can only perceive foods and the opponent that are within this distance. The agent's observation includes its position and level as well as the relative position and level of each other agent and food in the visible grid cells.
Images of the multi aultt- environments are provided in Appendix B. For each environment, we create ten different opponent policies which are used for training (set 1). In speaker- listener we control the listener, and we create ten policies for the speaker using different communication messages for different colours. In double speaker-listener,, we create a diverse set of opponent policies that use different communication messages similar to speaker- MADDPG algorithm [28]. The generated policies are different because they learn to both interpret and communicate different messages to the other agent. For LBF, we created four heuristic policies such as moving to the closest food closest level-compatible food, and additionally six policies using a stochastic-based RL method with different initial seeds leading to different trained policies. More details about the process of generating opponent policies are presented in Appendix C.
while they learn to navigate using the
5.2 EPISODIC RETURNS DURING TRAINING
We compare LIOM against two baselines, which are indicative of the upper and the lower performance of LIOM when it is evaluated against the opponent policies from II_,. For the upper baseline, we propose a VAE-based opponent model which is trained on the trajectories of the opponents. The encoder of the VAE receives as input the observation and the action of the opponent and infers the opponent model z. We call this baseline FIOM (Full Information Opponent Model). FIOM approximates the latent opponent identity distribution using variational distribution that is
Baselines:

---

Under review as a conference paper at ICLR 2021
Speaker-Listener
Double Speaker-Listener
Level-Based Foraging
0.4
Return
-11
-10
92
6.
24
15.
5.0
1-4
031
34
Time steps
Time steps
Time steps
FIOM
LIOM
NOM
Figure 2: Episodic return and 95% confidence interval against opponent policies from II-1.
conditioned on the trajectory of the opponent. We optimise the opponent representation along with the reinforcement learning objective similarly LIOM. Note that FIOM has access to opponent's information both during training and execution. The lower baseline that we use is called No Opponent Model (NOM). NOM uses the A2C algorithm without any model of the opponent.
Figure 2 shows the episodic returns for the three methods in all three environments. The lines shown in our results plots show the average return five with different initial seeds, and the shadowed part represents the 95% confidence interval. We evaluate the methods every 1000 training episodes for 100 episodes. During the evaluation, we compute the mean of the variational distribution at each time step, and the agent follows the stochastic policy. We found that sampling the action from the policy distribution leads to significantly higher returns compared to following the greedy policy. We observe that LIOM's 's episodic returns are closer to FIOM than NOM in all environments. This shows that LIOM can successfully learn opponent models using only locally available information. At the beginning of the training the returns achieved by LIOM and NOM are identical, because the encoder generates uninformative and noisy embeddings. After some time step, the encoder has learned to generate informative embeddings and the difference in returns increases significantly.
5.3 ENCODER EVALUATION
We analyse the embeddings learned by LIOM's encoder. Figure presents the first two principal components of the mean of the variational distribution the 20th time step of the episode for all opponents in II_1. In speaker-listener and double speaker-listence we observe that three different clusters are created, representing the three different colours that the opponent observes. We observe that the controlled agent learns to perfectly identify the underlying colour, even though this informa- tion exists only in the hidden opponent's ' observation. Note that the embeddings are not clustered based on the opponent identities but based on the selected actions and private observations, which means that different opponent policies may be embedded in the same region.
Speaker-Listener
Double Speaker-Listener
Level-Based Foraging
3.0
Second PC
0.5
First PC
First PC
First PC
Figure 3: First and second principal components of the learned opponent representations. Points represent individual episodes, colours represent the different opponent policies in II_
7

---

Under review as a conference paper at ICLR 2021
Embedding Convergence
Another interesting question that aim to answer
Euclidean Distance
Beokeriaee Lise Scadd Faegre
is how quickly the embeddings converge during an episode. At the beginning of the episode, the model does not know anything about the opponent, and it outputs uninformative posterior. As the agent in- teracts with the opponent, the posterior is updated to accommodate the information that has been gath- ered. To measure the speed of convergence, Figure 4 presents the Euclidean distance between the mean of the variational Gaussian at the each time step, and the mean of the variational Gaussian at the 25th timestep. From Figure 4, we observe that the distance to the po- sition of the last embedding decreases through time in all tasks.
Episode Time
Time steps
Figure 4: Euclidean distance of embeddings to the embedding at 25th time step (with 95% confidence interval).
5.4 DECODER EVALUATION
The decoder receives as input an embedding from the latent space, and predicts the opponent's s observation and action. We first evaluate the decoder based on action prediction accuracy. At the 20th time step of the episode, we sample an embedding and use the decoder to reconstruct the opponent's s action. Figure 5 presents the average action prediction accuracy (the percentage of the reconstructed actions that match those that were performed from the opponents) and the 95% confidence interval.
Decoder Action Prediction
LINS- Fooglng
Environment
Decoder
Opponent
Speaker-listenee
-6.6 0.6
6.33 = 0.2
Accuracy
Double speaker-listener
-14.6 + 1.4
-14.4 1
Level-based foraging
0.27 + 0.04
0.34 + 0.02
Table I:
Average episodic return and 95% confidence inter- val of the opponent using the decoder vs. opponent's 's policy to select the opponent's actions.
Figure 5: Action prediction accuracy.
The difference in prediction accuracy between LBF and the other tasks is explained by the fact that the RL-based opponent policies in LBF stochastic, and result harder to predict. Next, since the decoder is trained to imitate the opponent's policies, we can replace the opponent policies with the decoder and compare the returns achieved by the opponent, shown in Table 1. The performance of the opponent using the decoder is not significantly altered, demonstrating that the decoder learns imitate the opponent policy. The results of Figure 5 and Table I are averaged over 5000 episodes for five different seeds.
5.5 ABLATION STUDY ON LIOM INPUTS
Our full method utilises the observation, action, and reward sequence of the controlled agent to generate the opponent model. To evaluate the impact of different types of input data, we use different combinations of inputs in the encoder and compare the episodic returns. Figure presents the average episode return for four different cases: LIOM (full), LIOM using only observations and actions, LIOM using only observations and rewards, LIOM using only actions and rewards. Figure 6 shows that speaker-listener listener the most important components are the reward and the observation. In double speaker- -retter-listener listener the performance is affected by the absence of either the agent's 's actions or observations in the computation of the embeddings. The reward is the average over the distance of both agents to the correct landmarks, which is complex and cannot be effectively utilised by the encoder. In LBF, the average episodic returns are only affected by the absence of the observation in LIOM's input. In LBF, the rewards sparse and the actions that agent performs can be inferred from consecutive observations, and absence of any of those terms does not affect the returns. Generally, experiments indicate that LIOM is robust with respect to different inputs the encoder.
8

---

Under review as a conference paper at ICLR 2021
Speaker-Listener
Double Speaker-Listener
Sevel-Based Foraging
0.25
Return
5.40
-12
-38
9.25
6.00
-38
8.15
2.0
6G4
60
32
044
3.4
0.0
Time steps
Time steps
Time steps
LIOM
LIOM (Obs, Act)
LIOM (Obs, Rew)
LIOM (Act, Rew)
Figure 6: Episodic return and 95% confidence interval against opponent policies from II_1 for different combinations of input data for the encoder. (In speaker-listener, LIOM (Obs, Act) overlaps with LIOM (Act, Rew).)
5.6 NUMBER OF OPPONENT POLICIES
Finally, we evaluate the effect of number of the opponent policies in the average achieved returns of FIOM, LIOM, and NOM in the double speaker-listens listener environment. We train FIOM, LIOM, and NOM against a subset II",, of the original II_ set, where the size of II',, varies between one and ten. Figure 7 presents the average episodic return achieved at the end of training against different numbers of opponent policies. We observe that when there is single opponent policy the performance of FIOM, LIOM, and NOM is equal. This is expected as there is not need for opponent modelling. As we increase the size of IT',, we observe a steep decrease in the returns of NOM, while the returns of FIOM, and LIOM decrease at much slower rate. This natural consequence of our problem formu- lation, since at the beginning of the episode LIOM does have information about the opponent and because both the controlled agent and the opponent are relatively far from the correct landmarks, they are heavily penalised. After some interaction time steps LIOM able to identify the opponent, and outputs efficient embeddings that can be utilised by the policy network. On the other hand, NOM learns an "average" policy against the opponent policies, resulting in much lower returns.
Double Speaker- Listener
Return Against [
-10
-12
-18
-20
|M
FIOM
LIOM
NOM
Figure 7: Average episodic return
against different numbers of opponent policies.
6 CONCLUSION
We proposed new opponent modelling approach, LIOM, which jointly trains VAE-based opponent model with a decision policy for the agent under control, such that the resulting opponent model is conditioned only on the local observations of the controlled agent. LIOM is agnostic the type of interactions in the environment (cooperative, competitive, mixed) and can model an arbitrary number of opponent policies simultaneously in the set II_1. Our results show that LIOM can significantly improve the episodic return that the controlled agent achieves over method that does not use opponent modelling. Compared an ideal baseline method that has access to opponent trajectories during execution, we observed a relatively small decrease in performance of LIOM in our specific test environments. Further research on how such models could be used for non-stationary opponents would be of interest. In particular, we plan on investigating two scenarios; the first is multi-agent deep RL, where different agents are learning concurrently leading to non non-stationarity in the environment, which prevents the agents from learning optimal policies [21, 32]. Secondly, explore notions of "safety"" to handle opponents which aim to deceive and exploit the opponent model [39, 11, 12].
would like to

---

Under review as a conference paper at ICLR 2021
REFERENCES
[1] S. V. Albrecht and P. Stone. Reasoning about hypothetical agent behaviours and their parameters.
In International Conference on Autonomous Agents and Multi-Agent Systems, 2017.
[2] S. V. Albrecht and P. Stone. Autonomous agents modelling other agents: comprehensive survey and open problems. Artificial Intelligence, 2018.
[3] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In IEEE Annual Foundations of Computer Science, 1995.
[4] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 2002.
[5] N. Bard, M. Johanson, N. Burch, and M. Bowling. Online implicit agent modelling. In Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, 2013.
[6] K. M. Borgwardt, A. Gretton, M. J. Rasch, H.-P. Kriegel, B. Schölkopf, and A. J. Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 2006.
[7] M. Bowling, M. Johanson, N. Burch, and D. Szafron. Strategy evaluation in extensive games with importance sampling. In International Conference on Machine learning, 2008.
[8] P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, Y. Wu, and Zhokhov. OpenAl baselines. https: petpp::/githun /ttp//gtttttttttt//// com/ openai/ baselines, 2017.
[9] N. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[10] Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncer- tainty in deep learning. In International Conference on Machine Learning, 2016.
[11] S. Ganzfried and T. Sandholm. Game theory-basee opponent modeling in large imperfect- information games. In International Conference on Autonomous Agents and Multiagent Systems, 2011.
[12] S. Ganzfried and T. Sandholm. Safe opponent exploitation. ACM Transactions on Economics and Computation (TEAC), 2015.
[13] P. J. Gmytrasiewicz and P. Doshi. A framework for sequential planning in multi-agent settings. Journal of Artificial Intelliencee Research, 24, 2005.
[14] K. Gregor, G. Papamakarios, F. Besse, L. Buesing, and Weber. Temporal difference variational auto-encoder. International Conference on Learning Representations, 2019.
[15] A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. J. Smola. A kernel method for the pro-sample-problem. In Advances in neural information processing systems, 2007.
[16] A. Grover, M. Al-Shedivat, J. K. Gupta, Y. Burda, and H. Edwards. Learning policy representa- tions in multiagent systems. International Conference on Machine learning, 2018.
[17] A. Gupta, R. Mendonca, Liu, Abbeel, and S. Levine. Meta-reinforrcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, 2018.
[18] D. Ha and J. Schmidhuber. World models. arXiv preprint arXiv: 18011100.1111 2018.
[19] K. Hausman, J. T. Springenberg. Z. Wang. N. Heess, and M. Riedmiller. Learning embedding space for transferable robot skills. International Conference on Learning Representations, 2018.
[20] H. He, J. BoydeGraber, K. Kwok, and H. Daumé III. Opponent modeling in deep reinforcement learning. In International Conference Machine Learning, 2016.
10

---

Under review as a conference paper at ICLR 2021
[21] P. Lernandez-Leal, M. Kaisers, T. Baarslag, and E. M. de Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. arXiv preprint arXiv: 2017.
[22] M. Igl. L. Zintgraf. T. A. Le, F. Wood, and S. Whiteson. Deep variational reinforcement learning for POMDPs. In International Conference on Machine Learning, pages 2122-2131, 2018. 2018.
[23] N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, P. A. Ortega, D. Strouse, J. Z. Leibo, and N. De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. International Conference Machine Learning, 2019.
[24] M. Johanson, M. Zinkevich, and M. Bowling. Computing robust counter-strateiies. In Advances in neural information processing systems, 2008
[25] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6441..6980 2014.
[26] D. P. Kingma and M. Welling, Auto-encoding variational bayes. International Conference on Learning Representations, 2014.
[27] M. L. Littman. Markov games as framework for multi-agent reinforcement learning. In Machine Learning Proceedings 1994. Elsevier, 1994.
[28] R. Lowe, Y. Wu, A. Tamar, J. Harb, Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-competiiee environments. In Advances in Neural Information Processing Systems, 2017.
[29] A. L. Maas, A. Y. Hannun, and A. Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In International Conference Machine Learning, 2013.
[30] V. Mnih, P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning. 2016.
[31] I. Mordatch and P. Abbeel. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv:1703.049000 2017.
[32] G. Papoudakis, Christianos, A. Rahman, and S. V. Albrecht. Dealing with non-stationarity in multi-agent deep reinforcement learning. arXiv preprint 1rXiv:190600473: 2019.
[33] N. C. Rabinowitz, F. Perbet, H. F. Song, C. Zhang, S. Eslami, and M. Botvinick. Machine theory of mind. International Conference on Machine Learning, 2018.
[34] R. Raileanu, E. Denton, A. Szlam, and R. Fergus. Modeling others using oneself in multi-agent reinfoncement learning. arXiv preprint arXiv: arXiv:1802.0966 0904009440 2018
[35] K. Rakelly, A. Zhou, D. Quillen, C. Finn, and S. Levine. Efficient off-policy meta-reiiforcemment learning via probabilistic context variables. International Conference on Machine Learning, 2019.
[36] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson. Qmix: mono- tonic value function factorisation for deep multi-agent reinforcement learning. International Conference on Machine Learning, 2018.
[37] J. Schmidhuber and S. Hochreiter. Long short-term memory. Neural Computation, 1997.
[38] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 106.43355..00233, 2015.
[39] Y. Shoham, R. Powers, and T. Grenager. If multi-agent learning is the answer, what is the question? Artificial intelligence, 2007.
[40] Stone, G. Kaminka, S. Kraus, and J. Rosenschein. Ad hoc autonomous agent teams: collabo- ration without pre-coordination. In AAAI Conference Artificial Intelligence, 2010.
11

---

Under review as a conference paper at ICLR 2021
[41] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Leibo, K. Tuyls, et al. value-decomposition networks for cooperative multi- agent learning. International Conference Autonomous Agents and Multi-Agent Systems, 2018.
[42] A. Tacchetti, H. F. Song, P. A. Mediano, V. Zambaldi, N. C. Rabinowitz, T. Graepel, M. Botvinick, and P. W. Battaglia. Relational forward models for multi- learning. Interna- tional Conference on Learning Representations, 2019.
[43] S. Zhao, J. Song, and S. Ermon, Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv: 100.70002222222 022602 2017.
[44] L. Zintgraf, K. Shiarlis, M. Igl. S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In International Conference Learning Representations, 2019.
12

---

Under review as a conference paper at ICLR 2021
A PSEUDOCODE OF LIOM
Algorithm shows the pseudocode of LIOM.
Algorithm Pseudocode of the LIOM algorithm
for m 1,... M episodes do
Reset the hidden state of the encoder LSTM
Sample E opponent policies from IT_1
Create E' parallel environments and
gather initial observations
01,0, 1,00, <- zero vectors for = ...., H do
for every environment in E do
Get observations 01,t and 0-1.1
Compute the mean μ(z)) 02 -1
Sample action a1,1 ~ TT ((4)))
Sample opponent action 1- (o-10_
Perform the actions and get 01,+1, T1, d1,t end for if mod
then Gather the sequences of all E' environments in single batch B Perform gradient step to minimise (8)
Perform gradient step minimise (9)
end if end for end for
B EVALUATION ENVIRONMENTS
Figure presents instances of the three multi-agent environment that were used for the experiments.
(a) Speaker-listenee
(b) Double speaker-listens listener
(c) Level-based foraging
Figure 8: Multi-agent environments used in evaluation.
C OPPONENT POLICIES
Speaker- Liitenrr- Each policy of the speaker consists of a five-dimensional one-hot communication message that remains constant through the episode. We manually select different opponent policies that communicate different colours with different communication messages.
Double Speaker-Listener: Each the opponent's 's policy consists of two sub- aub-policies; one for the communication message and one for the navigation action. To generate diverse opponent, fixed the communication action all the ten agents to the fixed policy that was used in speaker-listence We then created five pairs of agents and we trained each pair to learn to navigate using the M. MADDPG
13

---

Under review as a conference paper at ICLR 2021
algorithm [28]. Each agent on the pair learns to navigate based on the communication message of the other agent in the pair.
Level-Based Foraging:
The opponent policies in Level-based Foraging consist of heuristic policies and policies trained with MADDPG. The heuristic agents were selected to be as diverse as possible, while still being valid strategies. We used the strategies from Albrecht and Stone [1], which are: (i) going to the closest food, (ii) going to the food which closest to the centre of visible players, (iii) going to the closest compatible food, and (iv) going to the food that closest all visible players such that the sum of their and the agent's 's level is sufficient to load it. We also trained policies with MADDPG by training multiple pairs of agents and extracting the trained parameters of those agents. In order to circumvent the instability caused by deterministic policies in Level-based Foraging, we have found that enabling dropout in the policy layers [10] both during exploration and evaluation (thereby creating stochastic policies) the agents perform significantly better.
D IMPLEMENTATION DETAILS
All feed-forward neural networks have 2 hidden layers with ReLU [29] activation function. The encoder consists of one LSTM [37] and linear layer with ReLU activation function. All hidden layers consist of 128 nodes. The latent dimension in speaker-listener and LBF is 15, while in double speaker-listener it is 20. The output of the decoder is passed through a softmax activation function to approximate the categorical opponent policy. For continuous action space, Gaussian distribution can be used. For the advantage computation, we use the Generalised Advantage Estimator [38] with AGAE 0.95. We create 10 parallel environments break the correlation between consecutive samples. The actor and the critic share all hidden layers. We use the Adam optimiser [25] with learning rates 3 10-4 and
10-4 for the A2C and the VAE loss respectively, and and we clip the gradient norm to 0.5. The multiplication factor of the MMD is À = 1 for all the environments. We subtract the policy entropy from the actor loss [30] ensure sufficient exploration. The entropy weight 3 is 10-3 We train for 10 million steps in all environments. During the hyperparameter selection, we searched: (1) learning rates in the range 10-4, 10 4] and 10 1.0 for the parameters of A2C and LIOM respectively, (2) hidden size between 64 and 128, and (3) entropy regularisation in the range of [10-5 10-2).
14


Reviews:
[{"review_id": "mATuUmnMY24", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes an opponent modelling technique for imperfect information games.  During training, a VAE is trained to encode the agent's observed trajectory to a latent space, and then decode it back to the full trajectory (including opponent observations and actions).  The encoder can be used at runtime using only information the agent observes, and if the VAE latent means are included as an input to the agent's neural net (e.g., with A2C) then the agent's performance resulting performance is enhanced.  Empirical results are presented that support the technique's effectiveness.\n\n----------\n\nPositives:\n\n- The paper is well motivated and clearly written.  I found it easy to read.\n\n- The problem is significant and the approach is a natural fit.  Of course opponent observations should not be used at execution time, and training a VAE to recover the missing information seems like a good approach.  The empirical results support the technique and I found them convincing.\n\n----------\n\nNegatives:\n\n- The paper is sometimes inconsistent in its description of techniques that need opponent observations at training time versus execution time.  The technique is often presented as \"not needing opponent observations\" while competing techniques \"need opponent observations\" (e.g., the end of Related Work:Learning Opponent Models, and the conclusion), whereas they mean (and often, but not always, clarify) that their technique *does* needs opponent observations during training, but does not during execution, and competing techniques need them during execution.  However, the authors appear unfamiliar with other opponent modeling work (I'm particularly familiar with the computer poker domain, although they cite two Ganzfried and Sandholm papers from that area) where there is a rich literature of opponent modelling being successfully used under these conditions (opponent observations needed only at training time, and never at execution).  The ongoing challenge in that community (with success against human professionals!) is to go further and perform opponent modelling without needing opponent obervations at *any* time, by using only the agent's observations at training time as well as execution time.  In general, the authors should be more careful about specifying \"at execution time\" whenever they claim their technique does not need opponent observations, and should probably be aware of more related work in this setting.\n\n- \"Opponent\" modelling.  In the intro the authors describe using 'opponent' as a neutral term to mean 'other agents in the environment', and I'm sympathetic to that as I've encountered this awkward phrasing in my own work.  However, of the 3 environments described in this work, 2 are purely cooperative (Speaker-Listener, Double Speaker-Listener) and the third is a mixed setting involving cooperation and greedy behavior.  None of these three environments are a purely adversarial (i.e., zero-sum) setting where we would properly call the other player an 'opponent', or even a positive-sum or constant-sum setting that does not involve cooperation.  This is more important than just the choice of terminology, however!  In the experiments, I can imagine the LIOM technique could possibly be inaccurate but still perform well overall, since the other player is aligned with LIOM's goals and not rewarded for exploiting any mistakes.  In an *actual* adversarial setting, even if the opponent's policy was static (i.e., not updated in response to LIOM's policy, as in the cited Ganzfried papers), the opponent could still exploit any mistakes LIOM might make through its static policy, possibly driving its performance below that of NOM.  I liked the paper and I'm half joking when I say this, but: a better empirical analysis of an opponent modelling technique would involve at least one experiment that includes an *opponent* in the strict sense, to demonstrate that the technique is effective and robust in environments where true opponents exist.\n\n- Empirical analysis.  I found this to be very good for supporting the main claims of the paper.  However, I would really have liked to see even a small investigation of robustness, even with static opponents (e.g., not with adapting opponents or worst-case opponents as in the Ganzfried and Sandholm papers cited in the conclusion, although I agree with the authors that that would be a great next step for future work).  For example, in Figure 2, if we were to generate additional holdout opponents for each game that were not part of the training set, and evaluate against them, how does the performance curve for LIOM and FIOM respond?  Do they still perform pretty well (e.g., they've usefully mapped the holdout opponent to something nearby in \\pi_-1)?  Does it revert back to NOM?  Does it do *worse* than NOM, by making painfully incorrect assumptions about the opponent?  The answers here are pretty relevant to this work, since while we might train against a population of presumed opponents, our real opponents out in the world are unlikely to share their observations with us while we re-train!  If the technique remains a bit robust if we move outside it's training, then that's much more exciting than a technique that only works with its specific training set and fails dreadfully outside of it.  Further along these lines, I'd have loved to see a graph plotting 'Size of \\pi_-1' on the x-axis and 'Average return after X steps' on the y-axis.  In such a graph, I'd really love to see: when '|\\pi_-1| == 1' does NOM perform better than in Fig 2?  How quickly does LIOM (and FIOM?) drop off as we increase the opponent population size?  Is there something special about the population sizes used here, or can the approach scale well outside of those values?\n\n- Related to the above two points: while the paper describes NOM as a lower-bound for LIOM, and it makes sense narratively to \"bound\" LIOM between FIOM and NOM, it really is not a lower bound in a mathematical sense.  FIOM might do worse than NOM in cases where 1) the other player is adversarial, or 2) the other player encountered at execution time has not been trained against, or 3) the other player was trained against, but insufficiently to create an accurate model, and overfitting to a bad model can be far worse than having no model.  In other opponent modelling work, I've seen opponent-aware agents do *far* worse than a baseline agent in those settings.\n\n----------\n\nRecommendation and Justification:\n\nI think the paper should be accepted.  The topic is relevant, the proposed solution seems like a natural fit, and the empirical results are pretty extensive (measuring reward and prediction accuracy) and convincing in their support of the technique.\n\nI have a few issues in the Issues and Suggestions section - some minor clarity points on the environments, a larger (but easily fixable) point about consistently describing when opponent observations are and aren't needed, suggestions for experiments I would have loved to see to verify basic robustness, and some notes about missed related work that also occupies this setting (use opponent info for training, never at execution).  Only one point (a line in the conclusion about this being the first successful use of opponent modelling under these conditions) is a strict concern; so long as that is addressed (or convincingly rebutted, if the authors disagree with the prior works I've cited), then I feel the paper is over the bar for acceptance.\n\nI don't have specific questions for the authors, but if they'd like to accept, address, or rebut anything I've listed in Negatives or Issues and Suggestions, I'd certainly welcome their response.\n\n----------\n\nIssues and Suggestions\n\n\n- Experiments section.  The description of the three games left me with quite a few questions (what exactly is observed by the players?  Are Speaker-Listener and Double-Speaker-Listener gridworlds, or something else? etc), and Appendix B provided helpful pictures but didn't clarify the fine details.  Since the Appendix is outside your page limit, can you be more explicit about exactly what the agents observe (pixels? If so, the whole map or a small region near them? If not pixels, how is the input formatted?  In Level-based Foraging, do agents observe their own level? The other player's level?) and what actions they take.  As a further example, for Speaker-Listener - the text says \"each agent and landmark is randomly assigned a color\".  Is the Speaker assigned a color?  It sounds like it from the text, but it wouldn't be useful for reward, so I'm guessing not?\n\n- More on the game descriptions.  Speaker-Listener is the \"Cooperative Communication\" game from Lowe 2017, right?  If so, it seems appropriate to cite its source!  If not, best to note the differences.  Are the other games original to this work, or should they be cited also?\n\n- Level Based Foraging.  It's not abbreviated as LBF while describing the game, but that abbreviation is used later, and that threw me for a bit.  I suggest abbreviating it in the description.\n\n- Speaker-Listener and Double Speaker-Listener.  This is pretty minor, but I'm a bit concerned about a specific detail of these environments (especially S-L) and A2C that might give LIOM an advantage over NOM by leaking information, without having to model the opponent well at all (although from Fig3, Fig5, and Tab1, it appears to indeed model the opponent well).  Specifically, the VAE encoder is trained with the agent's (Obs, Reward, Act) stream, so it could encode something about Reward.  Does the agent's A2C component take in only Obs as an input, or does it also include Reward and Last Action?  The original A2C paper describes only taking Obs as input, but in the practical A2C implementations in the codebase I use, the last timestep's reward and last timestep's action are often used as inputs (concatenated on after the Obs convolutional layers, before the LSTM) as these \"observations\" often aid the agent's learning and asymptotic performance.  So here's the catch.  If the VAE encodes something about the last timestep reward, and the A2C component does not (i.e., NOM doesn't see it), then LIOM might have access to a valuable feature that NOM does not, unrelated to opponent modelling altogether.  And in S-L, I can imagine a Listener successfully solving the game even with a mute Speaker, just by taking steps and seeing what direction it has to go for its reward to go up.  Could an LSTM learn that?  Possibly!  I know Speaker-Listener is a standard game, but I'd be curious to see results with either 1) an A2C implementation that observes last_reward and last_timestep to see if NOM (or LIOM (Act,Rew)) improves, or 2) a sharp version of the environment where reward is granted on reaching the goal, just to confirm that LIOM isn't winning because it's being leaked more information than NOM can observe.\n\n- In the conclusion, the statement \"To the best of our knowledge, this is the first study showing that effective opponent modelling can be achieved without requiring access to opponent observations.\" is untrue (well, the \"this is the first\" part, not the \"to the best of our knowledge part\"  :^D  ).  The authors mean (and should clarify) \"...at execution time\", but even then, there is a rich literature of prior work.  I'm most familiar with the work in the computer poker domain, where opponent information is only used (if at all) at training time and never at execution time, some approaches do not need opponent information at either training or execution time, and the techniques have been successfully deployed in real poker games to defeat top professional human adversaries.  For example, in addition to the two Ganzfried and Sandholm papers cited in this work, consider:\n\n - Computing Robust Counter-Strategies, NeurIPS 2007, Johanson et al.  Uses opponent observations at training time to compute robust counter-strategies, uses agent observations at execution time to choose which counter-strategy to play against the current opponent (who may not be one of the opponents used at training time). The UCB method described in this paper was used to compete against human professional poker players in Heads-up Limit Texas Hold'em in the 2007 Man-vs-Machine Poker Championship, which the humans narrowly won.  In the paper's experiments against artificial opponents, opponent observations were used at training time but not execution time, and some experiments involved a holdout opponent for which no opponent observations were provided at any time.  In the competition against humans, no human data was used at training time or execution time, and only agent observations were used to decide how to adapt to the opponent.\n\n - Strategy Evaluation in Extensive Games with Importance Sampling, ICML 2008, Bowling et al.  The same conditions were used as above (use opponent observations for training robust counter-strategies, use only agent observations at runtime), but adds an unbiased, low-variance method for considering all possible states the opponent could be in given what the agent observed, and selecting a counter-strategy to use in response.  This method was used to defeat top human professionals for the first time in Heads-up Limit Texas Hold'em in the 2008 Man-vs-Machine Poker Championship.  In that event, no opponent observations were used at either training time or execution time: an even purer application than is described here, although I definitely agree that the strictest constraint must be to disallow opponent observations at runtime.\n\n - Online Implicit Agent Modelling, AAMAS 2013, Bard et al.  This technique is similar to the above, in that it does not use opponent observations at runtime at all, and can be used without any opponent observations at training time either.  This technique focuses on the creation of a portfolio of useful counter-strategies for use against a broad set of opponents, such that at least one counter-strategy will be effective against any particular (and previously unknown, not trained against) opponent. The \"Implicit Opponent Model\" approach is to model opponents by how we can respond to them, instead of trying to capture (unobserved!) information about how they act at individual decision points, and is the cleanest refinement of the approach used in the previous two papers.  Again, this is rhetorically purer than the approach described in this paper, in that it never needs access to opponent information at training or execution time.\n\nThese are just some of the works that I'm most familiar with; the computer poker community has been studying this topic for a while, and under the conditions described in this paper (opponent observations available during training, but never at execution time).  I'm not listing these citations as a suggestion to cite them (they are related to this work, and the topic is much broader and longer than the papers cited by the authors, but this paper is fine without the particular citations I've listed above).  I only list them as evidence to disprove the conclusion's assertion that \"this paper is the first study showing that effective opponent modelling can be achieved without access to opponent observations (at execution time)\", as that is definitely untrue.\n\n----------------------\n\nI've read the other reviews, the author's responses, and the discussion - thank you everyone.  I'm still in favour of accepting the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WNjzYYi6bWG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers an algorithm for learning and using an opponent model that is only conditioned on an agent's local information (history of actions, observations, and rewards). A variational autoencoder (VAE) is trained to predict opponent observations and actions, given the agent's local information. While the opponent's observations and actions are required to train the VAE decoder, they're not needed at play time for the encoder. The authors show that with static opponents, the output of the encoder is informative, and conditioning the agent policy on local information plus encoder output outperforms just using the local information.\n\nThe paper was well written, and was to easy follow. Experiments seemed appropriate to demonstrate the author's claims. I had only a few specific comments.\n\n+++ Points after discussion about Meta-RL, HiP-MDPs, and MARL.\n\nLike Reviewer 4, my view of MARL is more general than requiring all agents to be learning, even if most work does have one algorithm training multiple seats. I don't think this actually changes anything in the current draft, but should hopefully not appear in future edits.\n\nThe related works should include discussion of the HiP-MDP paper.\n\nThese points fit together: there should be a consistent placement of this paper, and related works. Given fixed opponents, the multiagent problem is equivalent to a single agent problem. It doesn't seem relevant whether or not the unobserved, unknown variables correspond to different but similar environments, or different but similar opponents in a single fixed environment.\n\n+++\n\nSection 4.2 \"In Sections 1 and 2, it was noted that most agent modelling methods assume access to the opponent's observations and actions both during training and execution. To eliminate this assumption ...\"\nWeakened seems like a better choice of words than eliminated: it does still assume access to the opponent's observations during training.\n\nSection 5\nWhat is the size of the environment? Initial random placement? I'm trying to get some idea of the magnitudes of the rewards, which are based on Euclidean distance.\nAre the speaker-listener and double speaker-listener experiments using the OpenAI multiagent particle environments? If so, cite this (The MADDPG paper that is already cited for opponent algorithms?)", "extra_scores": {}, "extra_text": {}}, {"review_id": "TVaSAdtEzHH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses interactions in a multi-agent setting without access to the policies of other agents (termed opponents). Each agent has access to local observations only but is still required to cooperate with opponents (which use either heuristic or stochastic RL-trained policies). In each instance of the environment, the agent considered interacts with single opponent, drawn from a fixed set. The idea is then to learn an embedding of each opponent, which is derived from local observations of the agent, and trained by predicting opponent observations, rewards and actions which are made available at training time.\n\nOverall, I find this to be a well-executed study of learning opponent models with a VAE. The write-up is clear and easy to follow, and the experiments show good performance of the method, along with a base- and topline. Ablations on VAE training targets are provided, as well as analysis regarding the learned opponent embeddings and VAE decoder performance. I'm however a bit surprised by the poor performance of the NOM baseline on the speaker-listener environment. The reward is dense, so it seems the main task for the listener (mapping the the available 5-bit message to the respective goal location) should be solvable as-is? \n\nSome suggestions for further improvement:\n- I found Figure 1 confusing as I understand that the actual opponent policies do not have access to the latent variable Z, and that three items are predicted from the latent space: the opponent's actions, observations and rewards. \n- For LBF in Figure 3, I would suggest that the authors expand upon the fact that there is no discernible structure in the opponent embeddings. It would have been interesting to see how well the method works if agent and food locations were randomized in the LBF environment so that agents also have to cooperate in exploration. \n- I would also suggest to clearly state the dimensionality of the learned embeddings in Appendix D -- from the current text I would assume it is 128?\n- In the conclusion, you state: \"LIOM is agnostic to the type of interactions in the environment (cooperative, competitive, mixed) and can model an arbitrary number of opponents simultaneously.\" The method presented appears to only support a single opponent per environment instance, so it would be good to clarify this statement.", "extra_scores": {}, "extra_text": {}}, {"review_id": "EyVVROw2Qu-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this paper study opponent modelling in partially observable Markov games, following the _centralised training, decentralised execution_ paradigm. In particular, during training the assume access to all agents' (both ego and other agents) local observations and actions, while at test/execution time they control the ego agent with only local information. The local ego information, i.e., ego local observations, actions and rewards are used to infer unobserved (sufficient) statistics for others' observations and actions. The authors show empirically that their method perform competitively in a set of standard multi-agent environments.\n\n**Questions and Concerns**:\n1. The claims about novelty and comparison to literature is questionable. The authors **do** use non-local information during trained to train their VAE, an idea that is very much common in the \"centralised training, decentralised execution multi-agent paradigm, i.e., Dec-POMDPs\".\n2. How does the proposed method do compared to standard Dec-POMPD solvers, e.g., [1-4]?\n\n**References**\n\n[1] Foerster, J., Farquhar, G., Afouras, T., Nardelli, N. and Whiteson, S., 2017. Counterfactual multi-agent policy gradients. arXiv preprint arXiv:1705.08926.\n\n[2] Rashid, T., Samvelyan, M., De Witt, C.S., Farquhar, G., Foerster, J. and Whiteson, S., 2018. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:1803.11485.\n\n[3] Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W.M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J.Z., Tuyls, K. and Graepel, T., 2017. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296.\n\n[4] Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D.J., Leibo, J.Z. and De Freitas, N., 2019, May. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning (pp. 3040-3049). PMLR.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submitted paper is well written and easy to follow and also the idea of using VAEs for making inferences about the opponents on which a policy can be conditioned on is sensible. Also the reported performance in comparison to two baselines is good (although I have concerns about the selection of the baselines—see below). Acceptance of the paper was suggested by 3 of the reviewers and rejection by one of the reviewers. While I don’t share all concerns of the negative reviewer, I also suggest to reject the paper.

My suggestion to reject the paper is mainly based on seeing concerns of the positive reviewers more critical as these reviewers themselves and some concerns I have on my own. In particular, I don’t think that all MARL approaches can simply be discarded for comparison—no matter whether the opponents are learning or not. Regarding the evaluation, I think that an environment with real opponents must be considered and that robustness is a key property that should be studied (otherwise an approach with a fixed set of best response policies and inference about the opponent might perform as well). In that regard I also find the selection of baselines insufficient—the minimum I would expect is to consider a NOM baseline using an RNN (which as far as I can tell is not the case) such that it could make inferences about the opponent.

I want to acknowledge that the paper improved quite a lot during the rebuttal period in which the authors extended their discussion of related work on opponent modeling.

In summary, the paper could be improved substantially by an extended empirically study (more environments + baselines + "mismatch" settings). If the currently observed performance gains also hold in these settings, this can become a good paper but in its current form I think the paper is not demonstrating that the proposed approach performs favorably over natural baselines and works well against real opponents.

Decision: Reject