Variational Autoencoders for Opponent Modeling in Multi-Agent Systems
Authors: Georgios Papoudakis
Stefano V. Albrecht
Venue:  (2020)
Keywords: reinforcement learning
multi-agent systems
representation learning

Abstract: Multi-agent systems exhibit complex behaviors that emanate from the interactions of multiple agents in a shared environment. In this work, we are interested in controlling one agent in a multi-agent system and successfully learn to interact with the other agents that have fixed policies. Modeling the behavior of other agents (opponents) is essential in understanding the interactions of the agents in the system. By taking advantage of recent advances in unsupervised learning, we propose modeling opponents using variational autoencoders. Additionally, many existing methods in the literature assume that the opponent models have access to opponent's observations and actions during both training and execution. To eliminate this assumption, we propose a modification that attempts to identify the underlying opponent model, using only local information of our agent, such as its observations, actions, and rewards. The experiments indicate that our opponent modeling methods achieve equal or greater episodic returns in reinforcement learning tasks against another modeling method.

Under review as a conference paper at ICLR 2020
VARIATIONAL AUTOENCODERS FOR OPPONENT MODELING IN MULTI-AGENT SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-agent systems exhibit complex behaviors that emanate from the interac- tions of multiple agents in a shared environment. this work, we are interested in controlling one agent in multi-agent system and successfully learn to inter- act with the other agents that have fixed policies. Modeling the behavior of other agents (opponents) is essential in understanding the interactions of the agents in the system. By taking advantage of recent advances in unsupervised learning, we propose modeling opponents using variational autoencoders. Additionally, many existing methods the literature assume that the opponent models have access opponent's s observations and actions during both training and execution. To eliminate this assumption, we propose a modification that attempts to identify the underlying opponent model, using only local information of our agent, such its observations, actions, and rewards. The experiments indicate that our oppo- nent modeling methods achieve equal greater episodic returns reinforcement learning tasks against another modeling method.
INTRODUCTION
In recent years, several promising works (Mnih et al., 2015 Schulman et al. 2015a, Mnih et al., 2016) have arisen in deep reinforcement learning (RL), leading to fruitful results in single-agent scenarios. In this work, we are interested in using single-agent RL in multi-agent systems, where we control one agent and the other agents (opponents) in the environment have fixed policies. The agent should be able to successfully interact with a diverse set of opponents as well as generalize to new unseen opponents. One effective way to address this problem is opponent modeling. The opponent models output specific characteristics of the opponents based on their trajectories. By successfully modeling the opponents, the agent can reason about opponents' behaviors and goals and adjust its policy to achieve the optimal outcome. There is rich literature of modeling opponents in the multi- agent systems (Albrecht & Stone 2018).
Several recent works have proposed learning opponent models using deep learning architectures (He cet al. 2016) Raileanu et al. 2018; Grover et al., 20t Rabinowitz et al.] (018). In this work, we focus learning opponent models using Variational Autoencoders (VAEs) (Kingma & Welling 2014). This work is, to the best of our knowledge, the first attempt to use VAES in multi-agent scenarios. VAE are generative models that are commonly used for learning representations of the data, and various works use them in RL for learning representations of the environment (Igl et al.) 2018 Ha & Schmidhuber] 2018 Zintgraf et al. 2019). We first propose VAE for learning opponent representations in multi-agen systems based on the opponent trajectories.
shortcoming of this approach and most opponent modeling methods, as will be presented Sec- [ion is that they require access to opponent's ' information, such as observations and actions, during training as well as execution. This assumption is too limiting in the majority of scenarios. For exam- ple, consider Poker, where each agent never has access to the opponent's 's observations. Nevertheless, during Poker, humans can reason about the opponent's behaviors and goals using only their local observations. For example, an increase in the table's pot could mean that the opponent either holds strong cards or is bluffing. Based on the idea that an agent can reason about an opponent's 's model using its observations, actions, and rewards in recurrent fashion, we propose second VAE-based architecture. The encoder of the VAE learns to represent opponents' models conditioned on only local information removing the requirement to access the opponents' information during execution.

---

Under review as a conference paper at ICLR 2020
To summarize contribution, in this work, we explore VAEs for opponent modeling in multi- agent systems. We are not interested in VAEs as generative models but as methods for learning representations. We evaluate our proposed methodology using toy example and the commonly used Multi-agent Particle Environment (Mordatch & Abbeel 2017). We evaluate the quality of the learned representations, and the episodie returns that RI. algorithms can achieve. The experiments indicate that opponent modeling without opponents" information can perform the same or even better in RL compared models that access the opponent's is information.
2 RELATED WORK
Learning Opponent Models. In this work, we are interested in opponent modeling methods that use neural networks to learn representations of the opponents. He et al. (2016) proposed an opponent modeling method that learns modeling network to reconstruct the opponent's actions given the op- ponent observations. RRaileauu et al. (2018) developed an algorithm for learning to infer opponents' goals using the policy of the controlled lldd agent. Grover et al. (2018a) proposed encoder-decoder method for modeling the opponent's policy. The encoder learns a point-based representation of dif- ferent opponents' trajectories, and the decoder learns to reconstruct the' opponent's policy given samples from the embedding space. Additionally, Grover et al. 20) introduce an objective to separate embeddings of different agents into different clusters.
dl d(z4,2_,2) e1 la- 22-2 1a)2
(I)
where 2+ and are embeddings of the same agent from two different episodes and embedding is generated from the episode of a different agent. |Rabioowiz et al. (2018) proposed the Theory representations of opponents for meta-
of mind Network (TomNet), which learns embedding-basee learning. [Tacchetti et al.] (2018) proposed RFM to model opponents using graph neural networks. A common assumption among these methods, that this work aims to eliminate, is that access to opponents trajectories is available during execution.
Representation Learning in Reinforcement Learning. Another topic that has received signifi- cant attention, recently, is representation learning in RL. Using unsupervised learning techniques to learn low dimensional representations of the MDP has led to significant improvement in RL. Ha & Schmidhuber (2018) proposed a VAE-based and forward model to learn state representations of the environment. |Haummnn et al. (2018) learned tasks embeddings and interpolated them to solve envi-
harder tasks. [gl et al. 20188) used VAE for learning representation in partially-observat ronments. Gupta et al. 2018) proposed MAESN, which learns Gaussian embeddings to represent different tasks during meta-training and manages to quickly adapt to new task during meta-testing. The work of Zintgraf et al. (019) is closely related, where Zintgraf et al. proposed a recurrent VAE model, which receives input the observation, action, reward of the agent, and learns variational distribution of tasks. Rakelly et al. (019)) used representations from an encoder for off-policy meta- RL. Note that, all these works have been applied for learning representations of tasks or properties of the environments. On the contrary, our approach is focused on learning representations of the opponents.
BACKGROUND
3.1 REINFORCEMENT LEARNING
Markov Decision Processes (MDPs) are commonly used to model decision making problems. An MDP consists of the set of states S, the set of actions A, the transition function, P(s'|s, a), which is the probability of the next state, s', given the current state, s, and the action, a, and the reward function, r(s', a, s), that returns a scalar value conditioned on two consecutive states and the intermediate action. A policy function is used to choose an action given state, which can be stochastic a ~ π (a(s) or deterministic a = μ(s). Given policy π, the state value function is defined as V(st) = E, D "ryss = *.] and the state-action value (Q-value) Q(s,, ax) EI "r--r,,1 is the finite horizon of the episode. The goal of RL is to compute the policy that maximizes state value function V, when the transition and the reward functions are unknown.
Sa.a az), where 0 < > 1 is the discount factor and H
2

---

Under review as a conference paper at ICLR 2020
There is large number of RL algorithms; however, in this work, focus two actor-critic algorithms; the synchronous Advantage Actor-Critic (A2C) (Mnih et al.]
2016
Dhariwal et al.
2017) and the Deep Deterministic Policy Gradient (DDPG) (Silver et al.
2014
Lillicrap et al,
2015). DDPG an off-policy algorithm, using an experience replay for breaking the correlation between consecutive samples and target networks for stabilizing the training (Mnih et al. 2015). Given an actor network with parameters 0 and critic network with parameter Ø, the gradient updates are performed using the following equations.
min B + Quarger, of (s', Mrarget, (s')) Qs(s.
2 min - E (s, 0
On the other hand, A2C is an on policy actor-critic algorithm, using parallel environments to break the correlation between consecutive samples. The actor-critic parameters are optimized by:
min 1-sl-Alop mo(als) 2 + >Vo's') Vo(s)) 0,6
(3)
where the advantage term, A, can be computed using the Generalized Advantage Estimation (GAE) Schulman al. 20155).
3.2 VARIATIONAL AUTOENCODERS
Consider samples from a dataset E X that are generated from some hidden (latent) random vari- able z based on generative distribution (a(z))) with unknown parameter u and prior distri- bution on the latent variables, which we assume is Gaussian with 0 mean and unit variance p(z) = (2; 0, .). We are interested in approximating the true posterior p o(z x))x with a varia- tional parametric distribution qu (z|x)) = N<; μ4, Σ, w). Kingma & Welling 2014) proposed the Variational Autoencoders (VAE) to learn this distribution. Starting from the Kullback-Lein
(KL.) divergence from the approximate the true posterior DKL(gw (2 (=)()))))))) the lower bound on the evidence log p(x) is derived as:
logp(x)) 2 (x)e (log (m//²] (1m
The architecture consists of an encoder which receives sample æ and generates the Gaussian vari- ational distribution p(z æ;; w). The decoder receives a sample from the Gaussian variational distri-
bution and reconstructs the initial input æ. The architecture is trained using the reparameterization trick Kingma & Welling (2014). Higgins et al. (2017) proposed /- GAV, where a parameter / 2 0 used control the trade-off' between the reconstruction loss and the KL-divergence.
L(x; W. v) (x)a)
(lg Pu (z]=))
B3DKL (z|m)]lp(=))
(5)
4 APPROACH
4.1 PROBLEM FORMULATION
We consider a modified Markov Game (Littman 1994). which consists of N agents (1,2, 2, ..., N). the of states S, the set of actions A A A-1 the transition function P: A X + IR and the reward function settings, where each agent i has access only to its local observation o, and reward r;. Additionally,
S x A X + R We consider partially observable
two sets of pretrained opponents are provided T = [T- [ )==M and G [T- 1,m ( m== which are responsible for providing the joint action A-1- Note that by opponent we refer to I_,m which consists of one or more agents, independently from the type of the interactions (cooperative, mixed or competitive). At the beginning of each episode, we sample a pretrained opponent from the set T during training or from G during testing. Our goal is to train the agent 1 using RL, maximize the average return against opponents from the training set, T, and generalize to opponents sampled from the test G. Note, that when we refer to agent we drop the subscript.
max E, EE[ Σ Arell
(6)
3

---

Under review as a conference paper at ICLR 2020
4.2
VARIATIONAL AUTOENCODER WITH ACCESS TO OPPONENT's INFORMATION
Latent Space
0 111
Encoder
Z
W
Opponent Policy
a_1,1 1,1
a.1.1
0-11
Figure 1: Diagram of the proposed VAE architecture
We assume number of K provided episode trajectories for each pretrained opponents j E T, EGi (T-1 (_(,)))/__ where ((i {0-1.t. a-1,t and 0-1.1: Q-1,1 are the observations and actions of the opponent at the time step in the trajectory. These trajectories are generated from the opponents set T. which are represented the latent space from the variable and for which we assume there exists an unknown model Pu (T-112), Our goal is to approximate the unknown posterior, p(=[T_ T--1 using a variational Gaussian distribution (j, E; w) with parameters w. We consider using 3-VAE for the sequential task:
T(T-1; w. Ez (x/ Po 12)] (ogg
3DRL (2|T=
We can further subtract the discrimination objective (equation D that was proposed by Grover al. 201 Since the discrimination objective is always non-negative, we derive and optimize lower bound as:
L(T-1; 2
(*/ (og
(8)
3DL (aw (2 (=| l))()))) (d)
The discrimination objective receives as input the mean of the variational Gaussian distribution, pro- duced by three different trajectories. Despite the less tight lower bound, the discrimination objective will separate the opponents the embedding space, which could potentially lead higher episodic returns. At each time step t, the recurrent encoder network generates a latent sample which conditioned on the opponent's trajectory 1,1 until this time step. The KL divergence can be written as:
H Σ (x (p(z))
DKL (x)T))=))))
The lower bound consist of the reconstruction loss of the trajectory which involves the observation and actions of the opponent. The opponent's action, at each time step depends on its observation and the opponent's policy, which represented by the latent variable 2. We use a decoder that consists of fully-connected layers, however, a recurrent network can be used, if we instead assume that the opponent decides its actions based on the history its observations. Additionally, the observation at each time step depends only on the dynamics of the environment and the actions of the agents and not on the identity of the opponent. Therefore, the reconstruction loss factorizes as:
log (T
P. (
(10)
log pa-_:aa-.:l--1...2
From the equation above, we observe that the loss the reconstruction of the opponent's policy given the current observation and sample from the latent variable. Overall, our proposed VAE takes the form of a Conditional VAE (Sohn et al.] (015).. Figure illustrates the diagram of the VAE. The full pseudocode of the method is provided in the Appendix
4

---

Under review as a conference paper at ICLR 2020
4.3
VARIATIONAL AUTOENCODER WITHOUT ACCESS TO OPPONENT'S INFORMATION
Latent Space
0,
a
Encoder
Z
N
r1-1
Opponent Policy
a_1,1
d
0-1,1
Figure 2: Diagram of the proposed VAE architecture using local information
In Sections and was noted that most agent modeling methods assume access to opponent's observations and actions is available both during training and execution. To eliminate this assump- tion, we propose a VAE that uses parametric variational distribution which is conditioned on cntiratioo-actiooneewaa triplet of the agent that we control and a variable d indicating whether the episode has terminated; aw (=|T = (o, a, d)). More precisely, our goal is to approximate the true posterior that is conditioned on opponent's 's information, with variational distribution that only depends on local information. The use this local information in a recurrent fashion has been successfully used in me settings (Wang et al. 2016 Duan al. 2016). the KL divergence between the two distributions:
We start by computing
(Kw (2| (p((( = (2 Iw r) - -
(11)
By following the works of Kingma & Welling (2014) and Higgins et al. (2017 and using the Jensen inequality, the VAE objective can be written
L(T, w. v)
3DKL (x()))(()))))
(12)
The reconstruction loss factorizes exactly similar to equation 10 From equation 12 can be seen that the variational distribution only depends on locally available information. Since during exe- cution, only the encoder is required to generate the opponent's 's model, this approach removes the assumption that access to the opponent's 's observations and actions available during execution. Figure presents the diagram of the VAE.
4.4 REINFORCEMENT LEARNING TRAINING
We use the latent variable z augmented with the agent's observation condition the policy of our agent, which is optimized using RL. Consider the augmented observation space O' = O x Z, where O is the original observation space of the our agent in the Markov game, and Z is the representation space of the opponent models. The advantage of learning the policy on O' compared to O is that the policy can adapt different Z.
After training the variational autoencoder that was described in Section 4.2 we use it to train our agent against the opponents in the T. We use the DDPG (Lillicrap et al. 2015) algorithm for this task. We did not manage to optimize the representation jointly with the policy. neither with DDPG or A2C. At the beginning of each episode, we sample an opponent from the set The agent's 's input is the local observation and a sample from the variational distribution. We refer to this as OMDDPG (Opponent Modeling DDPG), and the full pseudocode provided Appendix [
We optimize the second proposed VAE method jointly with the policy of the controlled agent. We use the A2C algorithm, similarly to the meta-learning algorithm RL (Wang et al. 2016 Duan et al, 2016). In the rest of this paper, refer to this SMA2C (Self Modeling A2C). The actor's 's and the critic's input is the local observation and a sample from the latent space. We back- prokapro the gradient from both the actor and the critic loss to the parameters of the encoder. Therefore, the encoder's 's parameters are shaped to maximize both the VAE's objective as well as the discounted sum of rewards. The full pseudocode is provided in Appendix
5

---

Under review as a conference paper at ICLR 2020
5 EXPERIMENTS
5.1 TOY EXAMPLE
We will first provide a toy example to illustrate SMA2C. We consider the classic repeated game of prisoner's dilemma with constant episode length of 25 time steps. We control agent, and the other agent is selected randomly between two possible opponent policies. The first opponent always defects, while the second opponent follows tit-for-tat policy. At the beginning of the episode, one of the two opponents is randomly selected. We train SMA2C against the two possible opponents. The agent that we control has identify the correct opponent, and the optimal policy, can achieve, is to defect against opponent one and collaborate with opponent two. Figure 3 shows the payoff matrix, the embedding space at the last time step of the episode, and the episodic return that SMA2C and A2C achieve during training. Note that, based on the payoff matrix, the optimal average episodic return that can be achieved is 24.5.
Embeddinge Visualization
Episodic Reward
AUE
Secord Dimerston
(2, 2)
(-1,3)
Reward
D
(3,-1)
(0,0)
Pirst Dimansion
Time steps
(a) Payoff matrix
(b) Embedding visualization
(c) Episodic return
Figure 3: Payoff matrix, embedding visualization and episodic return during training.
5.2 EXPERIMENTAL FRAMEWORK
To evaluate the proposed methods in more complex environments, we used the Multi-agent Particle Environment (MPE) (Mordatch & Abbeel 2017). which provides several different multi-agent en- vironments. The environments have continuous observation, discrete action space, and fixed-length episodes of 25 time steps. Four environments are used for evaluating the proposed methodology: the speaker-listener, the double-speaker listener, the predator-prey, and the spread. In Appendix [A] descriptions of the different environments provided. During the experiments, evaluated the two proposed algorithms OMDDPG and SMA2C as well as the modeling method of Grover et al. (018a) combined with DDPG (Lillicrap et al. 2015).
In all the environments, we pretrain ten different opponents, where five are used for training and five for testing. In the speaker-listener environment, we control the listener, and create ten dif- ferent speakers using different communication messages for different colors. In the double speaker- listener, which consists of agents that have to be both listener and speaker simultaneously, we control the first agent. We create a diverse set of opponents that have different communication mes- sages similar to speaker-listener while they learn to navigate using the MADDPG algorithm (Lowe) et al., 2017), with different initial random seeds. In the predator-prey environment, we control the prey and pretrain the three other agents the environment using MADDPG with different initial parameters. Similarly, in spread, we control one of the agents, while the opponents pretrained using MADDPG.
We use agent generalization graphs (Grover et al. 2018b) to evaluate the generalization of the pro- posed methods. We evaluate two types of generalizations in this work. First, we evaluate the episodic returns against the opponents that used for training, T, which Grover al. (2018b) call "weak generalization". Secondly, we evaluate against unknown opponents from set G, which is called "strong generalization". A figure of agent generalization graph is provided in Appendix

---

Under review as a conference paper at ICLR 2020
5.3 EVALUATION OF THE REPRESENTATIONS
To evaluate the representations created from our models we will estimate the Mutual Information (MI) between the variational distribution (q(z/7) or g(2(T-1) and the prior on the opponents' identities, which is uniform. This is a common method to estimate the quality of the representation (Chen al., 2018, Hjelm al., 2019). To estimate the MI, we use the Mutual Information Neural Estimation (MINE) (Belghazi ct al. 2018). Note that, the upper bound of the MI, the entropy of the 1.61. We gather 200 trajectories against each opponent
uniform distribution, in our experiments in T, where 80% of them are used for training and the remaining for testing. The visualization of the embedding space, for the predator-prey environment, is provided in Appendix
Table I: MI estimations using MINE in the double speaker-listener and the predator-prey environ- ment of the embeddings at the 15th, 20th and 25th time step of the trajectory.
Double speaker- listener
lpetter-listener
Predator-pres prey
Algorithm
15
20
25
15
20
25
OMDDPG
0.86
0.81
0.86
0.87
0.86
0.85
SMA2C
0.73
0.72
0.7
1.27
1.04
1.39
Grover al. (2018a)
1.16
1.21
1.20
1.29
1.38
1.41
From T we observe that the method of Grover al. (2018a) achieves significantly higher val- ues of MI. We believe that the main reason behind this is the discrimination objective that implicitly increases MI. This is apparent in the MI values of OMDDPG as well. SMA2C manages create opponent representations, based only on the local information of our agent, that have information about the opponent identities. Additionally, based on Figure 4 we observe that the value of MI is not directly related to the episodic returns in RI. tasks. In Appendix B we demonstrate that when we detach the encoder's parameters from the policy optimization, the MI decreases.
5.4 REINFORCEMENT LEARNING PERFORMANCE
We evaluate the proposed opponent modeling methods in RL settings. In Figure the episodic returns for the three methods in all four environments are presented.
Speaker- Lidenner
Double Speaker- idener
Predetoe-Prey
Spread
Speate-Listeree
Double Speaker L Listener
Predator Pray
Spread
Rewerd
steps
Time staps
Tim stops
Time steps
SMA2C
OMDDPG
Grover et al. (018a)
Figure 4: Episodic return during training against the opponents from T (top row) and G (bottom row). Four environments are evaluated; speaker-listener (first column), double speaker-listener (sec- ond column), predator-prey (third column) and spread (fourth column). For the double speaker- listener, predator-pin -rrey and spread environments the x-axis is in logarithmic scale.
Every line corresponds to the average return over five runs with different initial seeds, and the shad- owed part represents the 95% confidence interval. We evaluate the models every 1000 episodes for
7

---

Under review as a conference paper at ICLR 2020
100 episodes. During the evaluation, sample embedding from the variational distribution each time step, and the agent follows the greedy policy. The hyperparameters for the experi- ments in Figure were optimized weak generalization scenarios, against opponents from set T. Details about the implementation and hyperparameters that were used for generating the figures are presented in Appendix [
OMDDPG an upper baseline for SMA2C achieving higher returns in all environments during weak generalization. However, OMDDPG, well as Grover et al. (2018a), tend to overfit and perform poorly during strong generalization in the speaker-listener and double speaker-listener en- vironment. SMA2C achieves higher returns that Grover et al. (2018a) in more than half of the scenarios. Below, in the Section we perform an ablation study on different inputs in the encoder of SMA2C. Appendix [B, the encoder, in SMA2C, affects the episodic returns.
we evaluate whether back- back- propkpaatpgnnngg the RI. loss to the parameters of
5.5 ABLATION STUDY ON SMA2C INPUTS
We perform an ablation study assess the performance requirements of the SMA2C. Our proposed method utilizes the observation, action, reward, and termination sequence to generate the opponent's model. We different combinations of these elements in the encoder and compare the average episodic returns. In Figure [5] the average episode return is presented for three different cases; SMA2C with all inputs, SMA2C with only observation and action as inputs and SMA2C with only observation as input; for all four environments.
Spaske-LListenee
Double Speaker-Littn Litteer
Prodalor-Prayy
Reward
SistkerListener
Double ListnerListener
Predatio-
Spread
Roward
Roward
Time steps
Time steps
Time steps
Time steps
SMA2C Full
SMA2C Observation-AAcion
SMA2C Observation
Figure 5: Ablation on the episodic returns for different inputs in the VAE of SMA2C for weak (top row) and strong (bottom row) generalization in all four environments.
5.6 ABLATION STUDY ON THE DISCRIMINATION OBJECTIVE
Another element of this work is the utilization of the discrimination objective of the Grover et al. (2018a) in the VAE loss. To better understand how the opponent separation in the embedding space is related to RL performance, below, Figure 6 shows the episodic return during the training for the OMDDPG with and without the discrimination objective is presented. Using the discrimination objective has a significant impact on the episodic returns in the speaker-listener listener and the double speaker-listener environment.
8

---

Under review as a conference paper at ICLR 2020
SpsokenListener
Double Listter
Predittor Prey
Spread
Reward
Spaaker-Liike Lstener
Double Spesker-
Proditoo-Prey
Sprad
18 stips
18
18
00
18
staps
staps
staps
OMDDPG
OMDDPG No Discrimination
Figure 6: Ablation study on the importance of the discrimination objective. Episodic returns in all four environments for weak (top row) and strong (bottom row) generalization.
CONCLUSION
To conclude this work, we proposed two methods for opponent modeling in multi-agent systems using variational autoencoders. First, we proposed OMDDPG a VAE-based method that uses the common assumption that access to opponents' information is available during execution. The goal of this work is to motivate opponent modeling without access to opponent's 's information. The core contribution of this work is SMA2C, which learns representations without requiring access to op- ponent's information during execution. We performed a thorough experimental evaluation of the proposed methodology. We evaluated the quality of the representations produced by our models as well as the episodic return that can achieve in RL tasks. The experiments conclusively indicate that access to the opponent's 's information not necessary during execution, eliminating a long-standing assumption of the prior work. Additionally, MI and the RL performance is not apparent. In the future, we would like to research how these models can be used for non-stationary opponents. Particularly, there are scenarios worth inves- tigating; the first is multi-agent deep RI., where different agents are learning concurrently leading -tatiioiion in the environment, which prevents the agents from learning optimal policies. Secondly, we would like to explore whether the proposed models can deal with opponents that try deceive it and exploit the controlled agent (Ganzfried Sandholm 2011 2015).
provided evidence that the relationship between the
REFERENCES
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-8 020666695550 2018.
Mohamed Ishmacl Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint 1oX1v:1801.044 2018.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders, In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: 2017.
//githull wtw. http://enaaibaaeeiiiee
Yan Duan, John Schulman, Xi Chen, Peter Bartlett, Ilya Sutskever, and Pieter Abbeel. RI 2; Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.02779, 2016.

---

Under review as a conference paper at ICLR 2020
Sam Ganzfried and Tuomas Sandholm. Game theory-basee opponent modeling in large imperfect- information games. In The 10th International Conference on Autonomous Agents and Multiagent Systems- Volume 2, pp. 533-540. International Foundation for Autonomous Agents and Multia- gent Systems, 2011.
Sam Ganzfried and Tuomas Sandholm. Safe opponent exploitation. ACM Trunsactions Eco- nomics and Computation (TEAC), 3(2):, 2015.
Aditya Grover, Maruan Al-Shedivat, Jayesh K Gupta, Yura Burda, and Harrison Edwards. Learning policy representations in multiagent systems. International Conference on Machine learning, la.
Aditya Grover, Maruan Al-Shedivat, Jayesh K Gupta, Yuri Burda, and Harrison Edwards. Evaluating generalization in multiagent systems using agent-interaction graphs. In International Conference on Autonomous Agents and Multiagent Systems, 2018b.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, pp. 5302-5311, 2018.
David Ha and Jürgen Schmidhuber. World models. arXiv preprini 1803. 2018.
2018.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposium Series, 2015.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. 2018.
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep rein- forcement learning. In International Conference on Machine Learning, pp. 1804-1813, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. International Conference on Learning Representations, 2(5): 6, 2017.
Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchilnrr, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. International Conference on Learning Representations, 2019.
Maximilian Igl. Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps. In International Conference on Machine Learning, pp. 2122- 2131, 2018.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzatoon with gumbel-softmax, arXiv preprint arXiv: 1.0/ 144, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations, 2014.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (oXiv:/50900277/ 2015.
Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In Machine Learning Proceedings 1994, pp. 157-163, Elsevier, 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cosperative-ompetitie environments. Advances in Neural Information Processing Systems, pp. 6379-63990 2017.
10

---

Under review as a conference paper at ICLR 2020
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In Internation Conference on Machine Learning, volume 30, pp. 3, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, pp. 529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-1937, 2016.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv: 703.04908, 2017.
Neil C Rabinowitz, Frank Perbet, H Francis Song, Chiyuan Zhang, SM Eslami, and Matthew Botvinick. Machine theory of mind. International Conference on Machine Learning, pp. 4218- 2018.
Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. arXiv preprint arXiv: 1802.09640, 2018.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy metttreeinfore learning via probabilistic context variables. International Conference Ma- chine Learning, pp. 5331-5340, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.0244 02438, 2015b.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministie policy gradient algorithms. In International Conference Machine Learning, 2014.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, pp- 3483-3491, 2015.
Andrea Tacchetti, H Francis Song, Pedro AM Mediano, Vinicius Zambaldi, Neil C Rabinowitz, Thore Graepel, Matthew Botvinick, and Peter W Battaglia. Relational forward models for multi- agent learning, arXiv preprint arXiv: /044, 2018
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint 1=Xx::6611.007633 2016.
Luisa Zintgraf, Maximilian Igl, Kyriacos Shiarlis, Anuj Mahajan, Katja Hofmann, and Shimon Whiteson. Variational task embeddings for fast adaptation in deep reinforcement learning. Inter- national Conference on Learning Representations Workshop on Structure Priors in Reinforce- ment Learning, 2019.

---

Under review as a conference paper at ICLR 2020
A DETAILS OF THE EXPERIMENTAL ENVIRONMENT
A.1 SPEAKER-LISTENER ENVIRONMENT
The speaker- listener environment consists of two agents, called speaker and listener as well as three designated landmarks that each one has a different color, red green or blue. At the beginning of the episode, the listener is assigned a color, which can be red, green, or blue. The task of the listener is to navigate to the landmark that has the same color. However, the color of the listener can only be observed by the speaker. So the speaker has to learn to communicate the correct color to the listener. The listener should be able to understand the generated message of the speaker and to navigate to the correct color. The observation space of the listener has 13 dimensions, which consists of the position of the listener, the positions of the three landmarks in the 2D environment and the 5-dimensional communication message of the speaker and its action space has five dimensions. The speaker has a 3-dimensional observation space, which is a vector assigned to the color of the listener, and 5- dimensional action space. We use our method of different speakers, pretrained speakers that use different communication messages. Both speaker and listener share the same reward, which is the negative Euclidean distance between the listener and the correct landmark. In Figure 7a an instance from the speaker-listent enviromment is presented.
train the listener be able understand a set
A.2 DOUBLE SPEAKER -ISTENR ENVIRONMENT
The double speaker- environment consists of two agents and three designated landmarks that each one has a different color, red green or blue, similarly to the speaker-listende environment. The only difference that both agents are simultaneously both speakers and listeners. Therefore, at the beginning of the episode, each agent has a color that can only be observed by the other agent. Each agent must learn both to communicate a message to the other agent as well as navigate the correct landmark. The observation space of each agent has 16 dimensions, where 13 of them are the same as the listener's the previous environment, and the other three are the vector assigned to the color of the opponent, while the action space is 5-dimensional for the navigation actions and 5-dimensional for the communication message as well. The reward is the average of the negative Euclidean distances between each agent and the correct landmark. This environment is significantly more difficult compared to the speaker-listenee because our agent has to infer both the color that the other agent observes as well as communicate the correct message that the opponent expects. In Figure 7b an instance from the double speaker-listener environment is presented.
A.3 PREDATOR- ENVIRONMENT
This environment consists of one prey agent and three predator agents. At the beginning of the episode, the prey and the predators are randomly placed on a 2D map. The goal of the prey is to avoid heing caught by predators. In the environment, there additionally two large black obstacles in order to block the agents. The advantage of the prey compared to the three predators is that it can move faster compared to the three adversaries. This environment, compared to the previous two, is competitive. We deliberately chose this environment in order to prove that our method is agnostic to the environment setting. In our work, we will apply the proposed algorithm in the prey agent in order to examine whether it can avoid a large number of different pretrained predator agents. The observation of each agent has 14 dimensions representing the agents' positions as well as the obstacles in the 2D space, while their action space consists of 5 actions. In Figure 7c an instance from the predator prey environment presented.
A.4 SPREAD ENVIRONMENT
The spread environment consists of three large agents and three landmarks as well. At the beginning of the episode, the three agents and the three landmarks are spread randomly in the 2D space. The goal of the agents is to navigate to three different landmarks without colliding. The reward is the negative distance of each agent from the landmark. In the case of collision, there is an additional negative reward. The reward is the same for all agents. All the agents have the same observation space, which consists of 18 dimensions, while their action space consists of 5 actions. In Figure 7d an instance from the spread environment, is presented.
12

---

Under review as a conference paper at ICLR 2020
(a) Speeker-listener
(b) Double SI.
(c) Predator-pre prey
(d) Spread
Figure 7: Instances of different MPE environments
B ABLATION STUDY ON RL BACK-PROPAGATION
We evaluate SMA2C without back-pree pronddapppggtnngggg the gradients of the RL loss the parameters of the encoder. Therefore, the encoder is only trained based on ELBO.
Speaker-Listener
Double Speaker-Listener
Reward
Reward
090
077
100
143
115
Time steps
Time steps
SMA2C SMA2C No Back-Propagation
Figure 8: Ablation study on back-propagation of the RL loss the encoder's parameters for weak generalization.
Figure 8 verifies that not performing back pack-propagation does not significantly affect the episodic returns that SMA2C achieves during weak generalization. Additionally, we compute the MI between the embeddings and the opponents' identities similarly to 5.. the double speaker-listener environment. We observe that the MI decreases when we do not perform back-propagation to the embeddings of the encoder.
Table 2: MI estimations using MINE in the double speaker-listener environment of the embeddings at the 15th, 20th and 25th time step of the trajectory.
Double speaker listener
Algorithm
15
20
25
SMA2C
0.73
0.72
0.7
SMA2C No Back-propagation
0.61
0.62
0.58
C EMBEDDING VISUALIZATION
Figure visualizes the embedding space for the three different evaluated algorithms in the predator- prey environment. Note that the embeddings were generated using interactions between the oppo- nents from the set T and the trained agent that control.
13

---

Under review as a conference paper at ICLR 2020
Predator
Predelor Proy
Prodater
Secone Prandove Compps Comporeen
Frr Princppll Component
Find Phreipal Componee
Find Phiniped Component
(a) OMDDPG
(b) SMA2C
(c) Grover et al.
2018a)
Figure Embedding visualization from the double speaker-listener environment. The figures present 200 embeddings against each one of the five opponents and have been projected in the two first principal components from the initial ten dimensions. The embeddings presented here are from the last time step of the episode.
D IMPLEMENTATION DETAILS
The pseudocode for training the VAE from Section 4.2 is provided below. We consider that 1000 trajectories are provided for each one of the opponents. which are generated against trained agents. We train the VAE for 1000 epochs, using Adam Kingma & Ba 2014) with 10 learning rate.
Algorithm Pseudocode of the proposed VAE algorithm
for 1 M do
Sample an episode and compute the embedding = < g(sample(F.))
Sample different episode and compute the embedding 24 < (ssample
for = 1 M do
if === then
continue
Sample an episode and compute the embedding z_ f- ((sample((E)) Update VAE parameters by maximizing
We train OMDDPG for million steps in all the experiments. Since the encoder of the VAE has an LSTM layer and has to be trained on sequential data, we use modified experience replay, that en- ables sampling of whole episodes, which has also been used by Hausknecht & Stone (2015). DDPG algorithm requires continuous action space. However, since our experimental environments have discrete action space, we use the Gumbel-Softmax trick (Jang al. 2016) to create differentiable samples from a discrete distribution. Additionally, we regularize the actor loss by adding the squared logits order to prevent them from getting large values. The pseudocode OMDDPG is presented below] 2
Algorithm 2 Pseudocode of the OMDDPG algorithm
for e = 1 K episodes do
opp <- sample(opponents)
while episode is not fished do
Get the observation of our agent and the observation of the opponent o-1
Compute the action of the opponent a-1
Get sample from the encoder +- q(2 g(2(0_1,0_1)
Compute the action of the agent using exploration
Perform the actions the environment and get new observations and rewards
Store the sequences of both agents in the experience replay
1%update ffrquuencyy == 0 then
Sample batch of sequences from the experience replay
Update the actor-critic parameters using 2 where f- concat(o,
Update the target networks
14

---

Under review as a conference paper at ICLR 2020
All neural networks have 2 hidden layers with ReLU (Maas al. 2013) activation function. We use the Adam optimizer (Kingma Ba 2014) for all experiments. The target networks are updated with 0.01. The hidden dimenstons for all VAE layers is 100. The parameter in the VAE loss (equation 8 is always 1. We perform gradient updates every 50 time steps with a batch 100 episodes. Table summarizes
Iummarizz the rest of the hyperparameters.
Table 3: Hyperparameters of OMDDPG, for speaker- listener (SL), double speaker-listener (DSL), predator-prey (PP) and spread (SP).
SL
DSL
PP
SP
Hidden dim RL.
64
128
64
Embedding dim
2
10
Actor Ir
0.001
0.001
0.001
0.001
Critie Ir
0.001
0.001
0.01
0.001
0.01
0.01
0.05
0.1
Policy reg
0.01
0.001
0.001
0.001
We train A2C for million steps in the speaker-lisener environment, 5 million in the double speaker- listener, 10 million in the predator-prey and 15 million in the spread environment. A2C policy algorithm is significantly less sample-efficient compared to DDPG, and as a result, more training steps are required. The pseudocode of SMA2C is presented below 3 We subtract the policy entropy from the actor loss (Mnih et al. 20166 to ensure sufficient exploration. The loss that SMA2C minimizes can be written as:
on-
min 6,0,0, 2
Esl(r + +V's - Vo(s))² Å Alog Ta(a)s)
(13)
ò log T + ( - -b7(TT)
Algorithm 3 Pseudocode of the SMA2C algorithm
Create D parallel environments
for = 1 K episodes do
Sample D opponents opp + sample(ppponents) while episode is not finished do
for every environment in do
Get the observation of our agent and the observation of the opponent o-1
Get a sample from the encoder +- q(2(o, a, d)
Compute the action a of the agent using exploration
+ +++
Perform the actions in the environment and get new observations, rewards and done 1%update d"oupddatefirequenee .frequenccc == 0 then
Gather the sequences from all environments to single batch
Update the actor-critic and VAE parameters using 13 where +- concat(o,
For the advantage computation, we use the GAE (Schulman et al. 2015b) with \GAE 0.95. We create 10 parallel environment to break the correlation between consecutive samples. The actor and the critic share all hidden layers in all the environments except the double speaker-listends We use the Adam optimizer (Kingma & Ba 2014), and we clip the gradient norm to 0.5. summariies the rest of the hyperparameters.
15

---

Under review as a conference paper at ICLR 2020
Table 4: Hyperparameters of SMA2C, for speaker-listener (SL), double speaker-listener (DSL),
predator-prey (PP) and spread (SP).
SL.
DSL
PP
SP
Hidden dim
64
128
128
64
Embedding dim
10
0.0005
0.0005
0.0005 1
0.0003
0.1
0.5
0.1
0.1
0.1
0.1
0.1
b
0.1
0.001
0.01
0.001
train frequency
25
5
5
E
AGENT GENERALIZATION GRAPHS
Figure 10 presents the agent generalization graph that was used in all experiments in this paper.
Weak Generalization
Strong Generalization
Controlled Agent
Figure 10: Agent Generalization graph. weak generalization, strong generalization
16


Reviews:
[{"review_id": "rkgGaeaAFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a reasonable and natural way of modeling\nopponents in multi-agent systems by learning a latent space\nwith a VAE. This latent space will ideally learn the strategy\nthat the opponent is playing to inform the agent's policy.\nWith fixed opponents, the results across many tasks are convincing.\n\nMy one concern with this modeling approach is that it will start\nbreaking down if the opponents are *not* fixed as this\npotentially makes the agent more exploitable.\nThe opponents could learn to send adversarial sequences to\nthe opponent model that make it appear like they are playing\none strategy but then they could change strategies at\na critical point where it is too late for the agent to recover\nor perform optimally.\nThis type of exploitability has been explored in the game\ntheory community in [1,2] and the references therein.\n\n[1] Ganzfried, S., & Sandholm, T. Game theory-based opponent modeling in large imperfect-information games. AAMAS 2011.\n[2] Ganzfried, S., & Sandholm, T. Safe opponent exploitation. TEAC 2015.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJllyhLAYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a variational autoencoding (VAE) framework for agent/opponent modeling in multi-agent games. Interestingly, as the authors show, it looks like it is possible to compute accurate embeddings of the opponent policies without having access to opponent observations and actions. The paper is well written, the methods are simple yet still interesting/informative, but there are a few questions that I find necessary to be addressed.\n\n\nMethods:\n\n1. I find the idea of learning to infer embeddings of the opponent policies from the agent's own local observations quite interesting. Intuitively, it makes sense -- since the opponent's policy effectively specifies the dynamics of the environment (from the agent's perspective), opponent's behavior must be reflected in the agent's observations. Comparing figures 1 and 2, the proposed encoder architecture also uses information about the reward (and episode termination). How critical is this information for opponent identification? Would it work without r_{t-1} and d_{t-1}?\n \n2. Sec. 4.2: \"We assume a number of K provided episode trajectories for each pretrained opponent\" -- how exactly are these trajectories obtained? Similarly, how exactly are the opponents pretrained? (Self-play, hardcoded, or something else?)\n\n3. As the authors mention, the triplet loss that discriminates between the opponents loosens the lower bound. Since the regularized objective is still a lower bound, I wonder if the triplet loss can be re-interpreted/expresses as a specific prior on the opponent model?\n\n\nExperiments:\n\n1. Sec. 5.1: to understand the effect of opponent modeling, it would be nice to see how baselines perform in this setup against a randomly picked opponent (otherwise, the curve in Fig. 3-c is not informative). I suggest the following baselines: tit-for-tat (hardcoded), a couple of classical learning algorithms for iterated games (e.g., policy hill-climbing, WoLF), an agent that learns using policy gradients but without opponent embeddings. Without any baselines, Sec. 5.1 seems like a sanity check which just shows that the implementation works unless I am missing something.\n\n2. Sec. 5.3: (1) Why is mutual information between the approximate posterior q and the prior p makes sense as the policy embedding quality metric here? (2) Could you intuitively (or formally) justify the fact that the triplet loss degrades MI metric? Right now, this is stated as a fact but not justified. (3) It looks like Grover et al. (2018a) used deterministic trajectory encoders; how exactly is MI measured in that case?\n\n3. If I understand correctly from Fig. 4, SMA2C (which uses local information) underperforms as compared to the methods that use opponent trajectories in 6/8 cases. To me, this somewhat confirms the point opposite to what the authors claim -- local observations, while containing some information about the opponent, are still inferior. Also, having baselines that do not use opponent embeddings on the charts of Fig.4 would help understand the contribution of opponent modeling.\n\n----\n\nI acknowledge reading the author's response, which addressed some of my questions/concerns to some extent. However, I believe that while estimating accurate embeddings of the opponent behavior from the agent's observations only is interesting, the approach has limitations, and I feel those are not studied in-depth enough (e.g., as a reader, I would like to understand if and when I should use the proposed approach and expect it to work). My assessment of the paper stays the same.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlAsasptH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to use VAEs to model fixed-policy opponents in a reinforcement learning setting. They use these models to augment existing RL algorithms in situations where the environment can be factorized into opponents.\n\nI really fail to see the point of this paper. All the techniques presented in the paper are standard and the way they are put together is not particularly original. I found no specific claims about the benefits the presented approach offers over alternatives. The experiments are described from a technical perspective but I did not understand what they are actually supposed to show.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The present work addresses the problem of opponent modeling in multi-agent learning settings, and propose an approach based on variational auto-encoders (VAEs). Reviewers consider the approach natural and novel empirical results area presented to show that the proposed approach can accurately model opponents in partially observable settings. Several concerns were addressed by the authors during the rebuttal phased. A key remaining concern is the size of the contribution. Reviewers suggest that a deeper conceptual development, e.g., based on empirical insights, is required.
Decision: Reject