Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of group-to-group coordination in multi-agent scenarios, where agents need to coordinate with newly arrived teammates. The proposed CSP approach (1) simulates diverse teams, (2) autoencodes teams' behavior trajectories, and (3) clusters trajectory representations into groups, each of which learns a separate sub-policy. Finally, given a new team to coordinate with, its behavior trajectory is observed and assigned the closest cluster and sub-policy. On four multi-agent cooperative environments, the paper demonstrates zero-shot coordination generalization to unknown teams.
strength_and_weaknesses: ## Strengths
- The problem of group-to-group coordination requiring generalization to new teammates is interesting and seems novel.
- The idea of separately probing a newly introduced team makes sense, as a probing policy can explore the parts of the state which bring out coordination schemes. Then, the resultant trajectory would be a good representation of what the team can do in different scenarios. Thus, while solving the task, the policy can look into the potential "future" coordination behavior (assumed to be deterministic though) through the trajectory representation. This eliminates the non-Markovian nature of the problem originating due to the unknown team of agents.
- On careful inspection and understanding, the method and the different proposed stages make sense as a solution to the proposed problem.
- The ablation study is crucial and tests the different variations of the method, albeit in 2 environments.

## Weaknesses
- **Paper writing**:
    + Overall, the paper was hard to follow as someone unfamiliar with the details of ad hoc teamwork. Ideally, the paper could have been written in a way that anyone with a working knowledge of RL and some Multi-agent RL should be able to follow the build-up of the paper. But there were many specialized terms (ad hoc, coordination scheme, etc) introduced without enough details, which made it hard to follow the flow of writing. It took me 2 full reads to even understand what the paper was trying to convey.
    + It is difficult for a reader to distinguish the problem and the solution in the paper. For example, "coordination scheme" is used sometimes as part of the problem formulation and sometimes as part of the approach (e.g. the unsupervised discovered latents are coordination schemes).

- **Unclear definitions**
    + *Coordination scheme definition*: Section 3 just defines coordination schemes as "different ways to solve the task". This is very open-ended and not a concrete definition, which makes it hard to understand the central point of the paper. It leaves many unanswered questions, such as:
        * What is a coordination scheme $c \in C$ exactly? Does it fully define all the sets of optimal behaviors for each agent in the task?
        * Why is the set of coordination schemes C discrete?
        * What does it mean for a policy $\pi^c$ to be derived by a scheme $c \in C$? Is this defined for deterministic policies only?
    + *Problem Formulation*: The formulation of cumulative reward is not explained well. For instance, what does $(\pi^c)^{-1}$ come from and what does it mean? What is the surrogate that is being replaced here?
    + *Assumptions*: The method just lists down the process that CSP follows, but does not first list down the assumptions that the problem setup makes. This makes it hard to understand what an ideal baseline would be. For instance,
        * Why is it assumed that the training can be split into such different stages with full control over the teams that can be sampled and tested?
        * Are the agents at test time also new or only the team compositions are new and expect generalization?
        * Why can't the coordination schemes be different from the training clusters? What if a new coordination scheme is required at test time that none of the clustered policies actually solve well directly, but interpolation is better (i.e. coordination schemes lie on a continuous plane)? In this case, in fact, the POP(z) or CSP(z) ablations should surely work better?

- **Empirical Evaluation**
    + The first (generating diverse team populations and training MARL) and second stages (learning scheme probing policy) of CSP require interaction with the environment. Are these extra environment steps accounted for while plotting the results in Figure 3? It seems from the description in Section 5 that the baselines PBT, FCP, and ODITS do not require Stage 2 interactions; and LIAM, and FIAM do not require Stage 1 or 2 interactions. If that is correct then for a fair comparison of learning efficiency, Figure 3 should account for the extra steps taken by CSP and baselines in stages 1 and 2. I am also curious how many timesteps are currently being dedicated to these stages for the 6 scenarios.
    + While I appreciate the ablation study, it is a bit little unsettling that the POP(z) ablation suffers so much. It could be that the current set of environments and coordination schemes are too simple and discrete that they can be modeled by separate sub-policies. However, for more complex and realistic environments, intuitively the context-based approach, i.e., POP(z) or CSP(z) should be better, as they can model the continuity in coordination schemes and interpolate in its latent representations accordingly. I am curious as to what the authors' thoughts are on this.

## Clarifications
- The introduction mentions: "The information for identifying different teams is collected by the same policy that aims to maximize coordination performance" will suffer from the exploration-exploitation dilemma. Is this validated by comparison to any baseline?

clarity,_quality,_novelty_and_reproducibility: Being in a niche domain, the paper is understandably hard to write and explain. Thus, it was tough to read and understand properly. So, its clarity was subpar.

The method and experimental setup seem to be good quality and novel. Especially, all the necessary ablations I could think of are present.

The code is provided, so it should be reproducible.
summary_of_the_review: The paper has a good amount of strengths but lacks several details, which limit its understandability, and consequently, a proper evaluation is hard. The empirical evaluation does not yet seem to incorporate the potentially special assumptions made by the paper. If these issues are addressed or clarified, I would be happy to engage in a discussion and raise my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address two weaknesses in current multi-agent coordination methods - they tend to focus on a single agent's ability to coordinate with teams in a zero-shot setting, which ignores team to team coordination and limits the ability to adapt to teammates. The paper addresses these shortcomings using a three step framework:
1) Learn coordination schemes from a diverse set of trained teams by maximizing a Soft-Value Diversity objective, or the variance of softmax normalized value estimates for all policies in a population. 
2) Train a module to embed information about teammates' policies using an action autoencoder. The data for this module is collected from one episode of team interaction using a separate probing policy, trained to maximize reconstruction loss of the autoencoder, effectively searching for uncertain states. 
3) Classify the teammate embedding within coordination strategies using k-means and learn a meta-policy to select among coordination schemes during evaluation. 
strength_and_weaknesses: Strengths:
* The paper addresses an important question of team-to-team coordination and fast adaptation of the policy. 
* The method generally matches or exceeds state of the art on all tasks. 
* The paper is generally well written and polished. 

Weaknesses:
* This paper seems to be solving a "simple" problem - finding a coordination policy for a particular team - by first solving the much harder problem of finding all possible coordination policies. It's not clear that the SVD objective presented in this paper would be effective in uncovering enough diversity in policies to coordinate well with a general team, as the partner agents are all drawn from the same (evaluation) population and trained with the same method, while prior works have shown that the choice of algorithm can result in substantially different zero-shot coordination ability. It would be interesting to see how the method performs when drawing teams from different populations or partially-trained populations. I would also like to see an ablation of the SVD outer objective during training.

* While there are six different, coordination tasks explored in this paper, most based on prior work, they range of coordination strategies is relatively limited. In the Fork environment for example, there are essentially only two choices strategies, up or down, while in Overcooked Ring the choices are clockwise or counterclockwise. On the opposite extreme, there is no particular strategy to learn for 1c3s5z, to the best of my knowledge. I would be interested in seeing some environments with less clearly defined coordination strategies, such as Hanabi. 

Questions:
* Are baselines trained and evaluated on the same populations, trained with the same SVD objective? 
* Is the algorithm used QMIX (Section 4.1) or VDN (Section 5)? 

Minor:
* Section 3/Problem Formalization - "we" -> "We"
* Section 4/ Meta-Policy Learning - "reset" -> "rest"
* Appendix C/PBT - "tixing" -> "fixing"(?)
clarity,_quality,_novelty_and_reproducibility: Quality:

The paper is generally of a high quality. The related works is sufficiently comprehensive. The paper runs thorough experiments on six scenarios across four different environments, and compares against five state of the art baselines.

Clarity:

The paper is well written and organized, with few typos. 

Originality:

The paper combines several existing methods in a novel way to address an under-explored area of research. Furthermore, it introduces some new methods and objectives, namely the Soft-Value Diversity objective and the team-dynamics reconstruction method. 


summary_of_the_review: Overall, this paper proposes an interesting but fairly complex method. While there are some weaknesses, the thorough experiments and results provide evidence for the effectiveness of the method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper focuses on the ad-hoc teaming aspect of coordination. Unlike most previous works it focuses on coordination of different groups of agents rather than a single-agent with different teams. Moreover they show adaption ability in the context of within an episode as well in various multi-agent cooperative scenarios. Their solution has three major components: 1) a bilevel optimization formulation for obtaining a _diverse_ population of teams, 2) a disentangling coordination scheme probing module to classify new teammates with limited episodic data with its own policy optimized for probing actions, and 3) hierarchical policy that chooses the appropriate sub-policy for distinguished coordination scheme to coordinate with other agents. 
strength_and_weaknesses: **Strengths**

This is a fantasic paper and it's great to see it go all the way towards a practical solution for this problem (albeit computationally expensive). The shared code is also fairly easy to follow and would definitely make it easier to try alternative ideas in this space. The three steps make total sense. While individual one of these may have been tried, this paper bring it together as a strong pipeline for a very practically useful class of problem. SMAC Fork turns out to be surprsingly effective understandable environment as well.

**Weaknesses**

Only unsatisfying part of the solution is the separate step of clustering/grouping which the paper shows was important for effective training. In principle it should be possible to force such clustering by the probing module or be figured out by the conditioned policy.

Given the bilevel framing, it's surprising that PSRO [1] and $\alpha$-PSRO [2] don't even see a mention. Similarly would be useful to mention [3] and [4] when it comes to conditioned policies for multi-agent adaption and their generalization evaluation. 

[1] https://arxiv.org/abs/1711.00832

[2] https://openreview.net/forum?id=Bkl5kxrKDr

[3] https://proceedings.mlr.press/v80/grover18a.html

[4] https://www.ifaamas.org/Proceedings/aamas2018/pdfs/p1944.pdf
clarity,_quality,_novelty_and_reproducibility: While elements of the ideas have been present in the community, the paper beautifully brings them together to make an effective pipeline. The paper is very clearly written and does a fantastic job of experiments to understand what is going on and what parts are important. The shared codebase was quite easy to follow and hopefully will be released for exploration by the community. 
summary_of_the_review: The paper comes up with a very effective pipeline for solving a super practically important problem of ad-hoc teaming.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Instead of following zero-shot coordination adopted in existing work, the proposed method CSP improves group adaptation using limited interaction data. CSP first trains a set of diverse teammate policies, then train a scheme probing policy to detect the patterns of teammates. By interacting with the teammates, the embeddings of the trajectories of the scheme probing policy encode the information of teammates' patterns (coordination schemes). By clustering the embeddings, CSP obtains the possible coordination schemes. Then CSP trains a meta-policy, which uses each sub-policy to deal with each coordination scheme. In execution, CSP first acts scheme probing policy to interact with the new teammates and then chooses the sub-policy with the closest embedding distance from the scheme probing policy.
strength_and_weaknesses: ## Strength

* Coordinating with new teammates is an important problem in multi-agent reinforcement learning.
* The proposed method seems reasonable.
* The experiments are performed on a variety of multi-agent tasks.

## Weaknesses
### minor issues 
* In problem formulation, the agents under control are denoted by $G^1$ and the teammates are denoted by $G^{-1}$. In Stage 1, do you train the team population of $G^{-1}$ or $G^1 \cup G^{-1}$? I think in multi-agent environments, it is not reasonable to train $G^{-1}$ only, but Stage 1 in Figure 2 is confusing. And I suggest distinguishing $G^1$ and $G^{-1}$ clearly in the method section.

* Interacting with different teammates, the intrinsic reward $L_{pred}$ of scheme probing policy is different. Would the non-stationary reward damage the learning of the scheme probing policy?

* I am curious about whether the team policies in Stage 1 are diverse enough or just collapse to a few modes. The paper claimed that "the number of clusters k will be approximately equal to the number of environmental coordination schemes if $\Pi_{train}$ already covers all possible schemes. " It is important to run experiments with different M (size of $\Pi$) and show the relationship between k and M.

* The intrinsic reward of the scheme probing policy $L_{pred}$ is an important part of CSP. Ablation studies about $L_{pred}$ are necessary to verify the effectiveness.

### major issues
* The problem setting is similar to the meta-learning problem. Thus, meta-learning baselines should be included for comparison, e.g., PEARL [1].
*  The scheme probing module is used to differentiate the embeddings of the trajectories when playing with different teammates. Thus, policy representation methods should also be included as baselines, e.g., [2]. 
* The proposed method may not be scalable. It seems all coordination schemes should be covered in the training set. Otherwise, the generalization to unseen schemes is not guaranteed. In the current experiments, the settings are essentially two-agent cases (even for SMAC where one agent controls 4 units and another controls 5). What about the case of more than two agents? This should be included as least empirically.

[1] Rakelly et al., Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables, ICML 2019.

[2] Grover et al., Learning Policy Representations in Multiagent Systems, ICML 2018.
clarity,_quality,_novelty_and_reproducibility: * Although the paper is well-written, the procedure of the method is complex and unclear, and it is hard to get the main points. A brief algorithm is necessary in the main pages. 
* Moreover, many details are not quite clear, e.g., how $\pi_{sp}$ is learned, and MARL($\pi_{sp}$) does not give sufficient information.
* Two-agent settings for the experiments can only be found in Appendix. Such information should be made clear in the main pages. 
* For novelty, it seems not quite novel, considering the proposed method is essentially policy representation in two-agent settings.   
 
summary_of_the_review: More baselines and experiment settings as mentioned above are required. The difference between the proposed method and existing work including policy representation and meta-learning methods should be justified. In its current form, the paper is below the bar of ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper motivates the relatively under studied problem of group-to-group coordination. The proposed solution (CSP) contains 3 key stages into an overall system highly praised and supported by one reviewer, but questioned by others. Most notably because CSP utilizes additional compute during its second stage, making comparisons to other baseline approaches in the original submission inconclusive. Furthermore, preliminary results offered by the authors during the discussion period show that by increasing the training time of baselines some perform similarly to CSP when considering variance (e.g. ODITS 3x on Overcooked.) Further exploration of the relative sample efficiency of CSP and (if necessary) improvements to its sample efficiency would significantly improve the algorithmic contribution.
justification_for_why_not_higher_score: Outstanding concerns regarding the fairness of comparison to baselines in the paper and significance of the additional compute used by the proposed method
Decision: Reject