Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method, ODITS, to enable effective ad hoc cooperation without requiring to predefine a static set of teammate types. This is done by training encoder networks to extract latent variables which capture the "situation type", which is trained to be a sufficient statistic for predicting the joint action value. This enables the agents to adapt in online fashion (it doesn't assume static other agents), and additionally the methodology is able to deal with partially observable environments.
main_review: Just as a forewarning, I'm not highly familiar with this problem so what I say should be taken with a grain of salt, but here are my observations nonetheless.

### Originality

It seems like the main claims to originality verge on the fact that ODITS:
1) doesn't require pre-defining agent types
2) is able to adapt in an online fashion to changing and unseen agents/situations
3) is able to deal with partially observable environments

Two works which are not cited, seem to potentially be related (and check at least some of the boxes above) are:
- Deep Interactive Bayesian Reinforcement Learning via Meta-Learning. From the paper: "A popular way of modelling other agents is type-based modelling which assumes that the other agent has one of several types that can be pre-defined by an expert, or learned from data [2, 5, 47]. In this paper, we learn a latent variable model, in which the latent variables can be seen as a continuous representation of agent types learned in an unsupervised way via interaction.". My understanding is that this would check boxes 1 and 2, and might be extendible to also achieve 3.
- Georgios Papoudakis and Stefano V Albrecht. Variational autoencoders for opponent modeling in multi-agent systems. My understanding is that in this paper they focus on partial observability (3) potentially at the expense of adaptation (2).

In any case, it would be helpful to add these to the related work and address what the main differences are. If particularly related, they could also constitute additional baselines for the method. MeLIBA in particular seems like a potentially natural choice of baseline to the method presented.

### Clarity

The paper is well written and overall quite clear. Some minor suggestions/questions:
- Figure 1: it's unclear what the groups on the right hand side are supposed to represent. For example, why are there 3 green agents and 5 yellow ones? Is this just supposed to represent the proportions with which such agents are sampled?
- In the "Marginal Utility" heading, it might be useful to briefly give a definition of Marginal Utility, rather than just referring to other work. I'm not sure how common this term is in this subfield – it might be standard enough for this to be reasonable.
- In various places throughout the text, you use the expression "key insight". It might be useful to vary this expression or use it more sparingly for the thing that is actually most "key" – especially the usage in the introduction seems quite underwhelming.
- Figure 3: I would separate the legend (containing "Teammate", "Other agent", and "Coins") from the training and testing types. I found the illustration quite confusing.
- Section 5.1. Partial observability means that the main agent might not see what coins the teammate has taken. I would describe qualitatively what the main agent does in order to address this? Do they try to find the teammate as soon as possible and then follow them? Do they scan the entire grid until they are certain of what coins the teammate took? Neither?

I appreciated the use of ablations in Sec. 5.5 showing the relative importance of the various components of the architecture.
summary_of_the_review: I don't feel qualified to comment on the method's limitations in the broader context of previous work. Outside of the method's quality, the experiments seem well executed and the work is relatively clear. It seems like the method presented does better than the baselines considered, although these might be limited (as mentioned in "originality"). Overall, I'll mostly remit to other reviewer's judgement on the method, and wouldn't be opposed to this paper being accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on the important problem of building agents that can function as useful teammembers in a ad-hoc team where agents can change behavior over an episode. The paper propose a new framework named ODITS that allows training an agent for ad-hoc teaming applications without strong assumptions on observability or pre-defined roles and categories. It does so my learning a latent variable representation of teammate behaviors in a data-driven fashion during policy training. The framework also adds an information-based regularizer to allow learning these variables from partial local observations of an agent (using all available information during training in a CTDE manner). The paper performs experiments on three different environments to demonstrate their outperformance.

main_review: **Strengths**

Ad-hoc teaming is an important problem from the perspecitve of real world applications. Removing some of limitations of required prior knowledge is quite important to allow full data-driven learning of such behavior. Paper's clear focus on this problem leads to integration of many ideas from latent-variable generative modeling in the generative modeling community with multi-agent RL solution techniques in the ODITS framework. The general idea of distilling high-level semantics of teammate behavior into a latent vectors makes sense and has been explored in some of the related works, but doing that in an online manner remained unexplored. Other ideas about using a hypernetwork for integration (as done by QMIX) and utilizing the CTDE assumption to its fullest while learning a proxy encoder for decentralized execution are all well utilized. There is a clear train-test agent separation protocol in the expriments and comparisons with multiple baselines and different numbers of agents and types.

**Weaknesses**

Fig 3 fill color seems to suggest something is wrong with AATEAM. In RL the failures are seldom so consistent that unlearning would have such a low spread around the mean. Techniques using some form of bi-level optimization or diversity should be relevant related works [1,2].
It's understandable that environments with more complicated dynamics were used given the number of training iterations required for the experiments (which would have been slower too because of additional computation added), it would be useful to add a limitations section to the paper. The experiments were still with relatively small teams and with agent behavior largely coming from pre-trained RL policies. How different the RL policies were might be hard to really understand but it might be useful to report action distribution divergences on pre-selected trajectories along with the t-SNE plot. Overall it's still difficult to say if the selected policies for testing are representative enough as say doing human evaluation where the style of play might be completely different.

[1] https://openreview.net/forum?id=Bkl5kxrKDr
[2] https://arxiv.org/abs/1907.03840



summary_of_the_review: Overall it's a good combination of exisiting ideas in latent-variable modeling and CTDE MARL to demonstrate effective ad-hoc teaming behavior.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper the authors tackle the challenge of ad hoc teamwork with unknown teammate types. Unlike most previous work, the proposed framework, ODITS, does not assume some finite set of teammate types or full observability, and adapts to current teammates via the utilization of a teammate situation encoder-decoder framework, which learns a latent representation of the teammate configuration the agent is currently observing (this observation can be partial). The authors subsequently show this approach outperforms state of the art baselines such as AATEAM on a several domains, namely a modified version of coin game, predator prey, and save the city.
main_review: Overall this is a pretty cool paper. The core insight, which is learning a flexible and adaptable way of utilizing teammate observation in ad hoc teamwork via latent representations seems to be a very useful approach, and the overall architecture seems sound. I do have some comments and questions, however:

- The relationship between the proxy encoder-decoder and the fully-observable encoder-decoder is not entirely clear to me, and it's not entirely obvious to me how robust the current formulation is. There's also the concern that the entire framework might be very difficult to train for even moderately sized environments such as half field offense.

- To be fair, in practice, most ad hoc teamwork approaches, even if they formally assume a finite set of teammate types, handle variable or novel teammates by learning a distribution over previously modeled types (this can be framed as a k experts problem, or a multi-armed bandit, among other approaches). While AATEAM arguably does this implicitly via the attention mechanism it would still be of interest to consider a simpler baseline whose situational encoder is a distribution over existing learned representations per type - how much does the encoder-decoder approach actually buys you?

- It is unclear to me that the approaches utilized in Section 5.2 to engender variability among teammate types actually gets you this - what concrete measures are there to show that there's real variability in single agent policies learned by varying seeds in QMIX, for instance? One would argue that in the very least a separate effort should be put into maximizing variability, or at least promoting it actively.

- I concede that I did not fully understand the take home message in Section 5.4, is it meant to show that simply following single agent QMIX policy works poorly for an ad hoc agent? That does not seem like a very strong result to show. I would argue that a better experiment would be using  a single agent learning algorithm (say PPO or A3C) and showing that it does worse than the ad-hoc policy learning approach (I would very much expect it to, but at least I understand what that experiment aims to show).

summary_of_the_review: Overall this is a reasonably solid paper which presents a novel approach for ad-hoc teamwork policy learning via learning latent embeddings of partial teammate observations.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses the problem of ad hoc teamwork, where an agent learns how to coordinate with a team of unknown teammates. Previous works typically assume that the teammates belong to one of a finite set of possible (known) types; at test time, the ad hoc agent uses the observed teammate behavior to identify their corresponding type and act accordingly.

The proposed method, instead, builds a predictor of the ad hoc agent's marginal utility (which depends on the teammates) that relies on local information alone. The predictor takes as input the ad hoc agent's current observations, $b_t$, and a latent vector, $c$, representing the current "teamwork situation". The teamwork situation representation, $c$, roughly plays the role of "teammate types" in previous works. To train the predictor, the proposed method makes use of an encoder-decoder model that, at training time, learns the "teamwork situation" representation $c$ using full state-action information. The output of the model is used in an "integration network" that estimates the $Q$-function for the underlying MMDP from $c$ and the agent's estimated marginal utility, $u_t$. Finally, since at test time the agent cannot access the full state-action information (and hence cannot estimate $c$ using the aforementioned encoder-decoder model), during training it learns an proxy encoder that builds an estimate $z$ for $c$ using only local information. The overall architecture thus includes two encoder-decoder models jointly trained so that they estimate a coherent representation for the "teamwork situation"---the first using only local information, and the second using full state-action information. At test time, the agent can use the estimated representation, $z$, as input to its marginal utility predictor and act accordingly.

The paper tests the proposed approach in 3 domains (modified coins environment, predator-prey, save the city) showing positive results over existing architectures and a well-established MARL approach (QMIX), and also presents a brief ablation study establishing the relevance of using the two encoder-decoder models to the overall performance of the method. 
main_review: Overall, I liked the paper. The proposed approach strikes me as novel in the context of ad hoc teamwork, although the idea of building multiple encoder-decoder models that learn a common latent representation from diverse sources of information by joint training through the addition of mutual-information loss terms has been explored in the context of multimodal representation learning (see, for example, references [a, b] below). It is also clearly different from the "main trend" on ad hoc teamwork, which mostly disregards partial observability.

At the same time, there is one aspect about the paper that does not fully convince me and which could be further discussed. My main objection is the assumption that---during training---the agent has full access to the other agent's and the environment's state. I'm fully aware that previous approaches (such as PLASTIC or ConvCPD) assume a "training phase" whereby the ad hoc agent is allowed to learn about the teammates. However, the interaction between the ad hoc agent and teammates is the same during training and testing---something which, in my view, is key to the ad hoc teamwork formulation: the ad hoc agent builds on its interaction with previous teams to quickly adjust to the current (unknown) team. The notion that the ad hoc agent, during training, is allowed full access to the teammates seems more in line with other settings---such as MARL or, perhaps, transfer in MARL---than with ad hoc teamwork.

Refs.

[a] M. Suzuki, K. Nakayama, Y. Matsuo. "Joint multimodal learning with deep generative models." arXiv:1611.01891, 2016.
[b] H. Yin, F. Melo, A. Billard, A. Paiva. "Associate latent encodings in learning from demonstrations." AAAI, 2017. 
summary_of_the_review: I believe the paper is interesting and provides a novel contribution in terms of ad hoc teamwork (addressing partial observability). It would be good if additional discussion could be made regarding the assumption of "centralized training".
review_title: null

Meta-review: The paper presents a method for cooperative ad-hoc collaboration by learning latent representations of the teammates. The method is evaluated in three domains. All the reviewers agree that the method is novel and adds an interesting contribution to the important and difficult problem of the ad-hoc collaboration, making fewer assumptions about the team and the teammates.

The next version of the paper should comment:

- On the societal impact of the centralized training.
- Wang et al, CoRL 2020, https://arxiv.org/abs/2003.06906, which addresses the cooperative tasks in the ad-hoc teams without privileged knowledge and assumptions about the teammates.
Decision: Accept (Poster)