Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a reasonable and natural way of modeling
opponents in multi-agent systems by learning a latent space
with a VAE. This latent space will ideally learn the strategy
that the opponent is playing to inform the agent's policy.
With fixed opponents, the results across many tasks are convincing.

My one concern with this modeling approach is that it will start
breaking down if the opponents are *not* fixed as this
potentially makes the agent more exploitable.
The opponents could learn to send adversarial sequences to
the opponent model that make it appear like they are playing
one strategy but then they could change strategies at
a critical point where it is too late for the agent to recover
or perform optimally.
This type of exploitability has been explored in the game
theory community in [1,2] and the references therein.

[1] Ganzfried, S., & Sandholm, T. Game theory-based opponent modeling in large imperfect-information games. AAMAS 2011.
[2] Ganzfried, S., & Sandholm, T. Safe opponent exploitation. TEAC 2015.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors propose a variational autoencoding (VAE) framework for agent/opponent modeling in multi-agent games. Interestingly, as the authors show, it looks like it is possible to compute accurate embeddings of the opponent policies without having access to opponent observations and actions. The paper is well written, the methods are simple yet still interesting/informative, but there are a few questions that I find necessary to be addressed.


Methods:

1. I find the idea of learning to infer embeddings of the opponent policies from the agent's own local observations quite interesting. Intuitively, it makes sense -- since the opponent's policy effectively specifies the dynamics of the environment (from the agent's perspective), opponent's behavior must be reflected in the agent's observations. Comparing figures 1 and 2, the proposed encoder architecture also uses information about the reward (and episode termination). How critical is this information for opponent identification? Would it work without r_{t-1} and d_{t-1}?
 
2. Sec. 4.2: "We assume a number of K provided episode trajectories for each pretrained opponent" -- how exactly are these trajectories obtained? Similarly, how exactly are the opponents pretrained? (Self-play, hardcoded, or something else?)

3. As the authors mention, the triplet loss that discriminates between the opponents loosens the lower bound. Since the regularized objective is still a lower bound, I wonder if the triplet loss can be re-interpreted/expresses as a specific prior on the opponent model?


Experiments:

1. Sec. 5.1: to understand the effect of opponent modeling, it would be nice to see how baselines perform in this setup against a randomly picked opponent (otherwise, the curve in Fig. 3-c is not informative). I suggest the following baselines: tit-for-tat (hardcoded), a couple of classical learning algorithms for iterated games (e.g., policy hill-climbing, WoLF), an agent that learns using policy gradients but without opponent embeddings. Without any baselines, Sec. 5.1 seems like a sanity check which just shows that the implementation works unless I am missing something.

2. Sec. 5.3: (1) Why is mutual information between the approximate posterior q and the prior p makes sense as the policy embedding quality metric here? (2) Could you intuitively (or formally) justify the fact that the triplet loss degrades MI metric? Right now, this is stated as a fact but not justified. (3) It looks like Grover et al. (2018a) used deterministic trajectory encoders; how exactly is MI measured in that case?

3. If I understand correctly from Fig. 4, SMA2C (which uses local information) underperforms as compared to the methods that use opponent trajectories in 6/8 cases. To me, this somewhat confirms the point opposite to what the authors claim -- local observations, while containing some information about the opponent, are still inferior. Also, having baselines that do not use opponent embeddings on the charts of Fig.4 would help understand the contribution of opponent modeling.

----

I acknowledge reading the author's response, which addressed some of my questions/concerns to some extent. However, I believe that while estimating accurate embeddings of the opponent behavior from the agent's observations only is interesting, the approach has limitations, and I feel those are not studied in-depth enough (e.g., as a reader, I would like to understand if and when I should use the proposed approach and expect it to work). My assessment of the paper stays the same.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The authors propose to use VAEs to model fixed-policy opponents in a reinforcement learning setting. They use these models to augment existing RL algorithms in situations where the environment can be factorized into opponents.

I really fail to see the point of this paper. All the techniques presented in the paper are standard and the way they are put together is not particularly original. I found no specific claims about the benefits the presented approach offers over alternatives. The experiments are described from a technical perspective but I did not understand what they are actually supposed to show.
review_title: Official Blind Review #3

Meta-review: The present work addresses the problem of opponent modeling in multi-agent learning settings, and propose an approach based on variational auto-encoders (VAEs). Reviewers consider the approach natural and novel empirical results area presented to show that the proposed approach can accurately model opponents in partially observable settings. Several concerns were addressed by the authors during the rebuttal phased. A key remaining concern is the size of the contribution. Reviewers suggest that a deeper conceptual development, e.g., based on empirical insights, is required.
Decision: Reject