Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors propose to use equivariant networks to handle the symmetry-breaking problem in multi-agent coordination settings. To generate an equivariant history-dependent policy, the authors present the equivariant LSTM structure and validate its performance in Hanabi games. 
strengths_and_weaknesses: Strengths:

 - I found the work interesting and believe the main technical contributions are (1) extending an equivariant MLP to an equivariant LSTM structure, (2) leveraging the permutation group while prior work mainly leverages the SO(2), (3) extending the equivariant net to game applications.

Weaknesses:

Originality:

 - Although the extension to the equivariant LSTM structure is interesting, I found the paper does not have an analysis tailored to LSTM (see the first two questions in the clarification part). Thus the contribution of this paper is not significant to me, but I look forward to the answers to the clarification questions below.

Clarification:

 - I found the Section 3 Method hard to follow. It could be improved by listing several subsections instead and highlighting the main contribution as extending the equivariant neural net to the LSTM setting.

questions: About contribution and soundness: 
- In section 3 line 147, the authors proposed a modification for LSTM layers, which is interesting. However, how is the modification here related to propositions 3 and 4 in the following line 155 and line 160? I assume that propositions 3 and 4 are not tailored for LSTM nets. 
 - A follow-up question is how the modifications in line 147 affect the equivariance of the proposed LSTM network?
 - What is the effect of the size of the network on the asymptotic performance? I assume that with a larger network and adequate data, the OP may have better performance. The equivariant net for sure helps improve the data efficiency but not necessarily the asymptotic performance. 

About writing and clarification:
 - The network modification is mysterious. Does the modification to LSTM only include the one mentioned in line 147?
 - In option 1 in line 177, the authors mention a mini-batch, is the batch size equal to 1? What is the effect of the group element batch size? 
 - How is the network parameter selected in this paper?

Minor issues:
 - Figure 1 is not referred to in the main content.


limitations: The authors addressed one limitation which is the prior knowledge of the group symmetry, which I think is crucial.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work approaches the zero-shot-coordination (ZSC) problem by explicitly restricting the learnable (or act-able) policies to only those that do not rely on known asymmetric convention in the environment. They do this by using a 'symmetrizer' that can be used to make any regular trained policy equivariant post-training or be used during training to train a network to act more equivariant (although a symmetrizer can still be used with this network on inference as well). Specifically, this work seeks to outperform prior symmetry-aware ZSC algorithms on the benchmark environment Hanabi.
strengths_and_weaknesses: Strengths
- This paper extends equivariance to recurrent neural networks
- This work is well written, especially considering the complexity of the material discussed
- This work does seem like a further mathemtical advance on eliminating symmetries similar to how Other-Play approaches ZSC

Weaknesses
- The original models for the basline are freely available, but this work chooses to train their own models. This adds another difference that makes it harder to compare to the baseline and calls into question the accuracy of the results.
- This paper seems to present an incrementally better solution towards dealing with explicitly defined symmetries in an environment, but more recent work (such as OBL [21]) does not require this explicit definition.
- I am uncertain of any empirical advantages it provides with the current state-of-the-art in ZSC. For example, Off-Belief-Learning [21], like this work, is also shown to eventually converge to the same policy regardless of the initialization, but achieves better results (20.85-23.76 cross-play) than this work without needing to explicitly define symmetries [21]
- This work chooses the same set of symmetries as Other-Play, and seems to train the neural networks the same or very similar as Other-Play to be invariant to these symmetries.
- In my understanding, this work trains its own versions of the baseline models, which seem to get very different scores than the original baseline papers report. However, these original papers publish already trained models that could be used (more details in questions)
questions: - This paper reports results for the baseline in Section 4.1 that are significantly different than the reported results in the baseline papers [19] and [20]. For example, the cross-play scores of Other-Play [20] in this work are 11.12, in the original OP paper they are 15.32 or 22.07 (with or without the auxilliary task, respectively) [20] and in this work SAD cross-play is reported as 9.43 in Table 2 but is reported as 2.52 in the original paper[20] (Table 1). What is the reason for these differences?

- How do you construct group actions suitable for cell state and hidden state i the LSTM as discussed in Proposition 2 in Section 3? In my understanding, the different dimensions in these states do not normally correspond to known observations or actions.

- If possible, I would appreciate more of a walkthrough in why the $\pi^{\text{equiv}}$ policy in section 2.5 will always choose the 0.9 lever.

- On page 5, what are the differeneces between the strategy of using "G-OP" and using Other-Play? Is it that Other-Play trains with more permutations?
limitations: Yes

Post-rebuttal change:
Increased score from 3 to 4 to reflect improvement of paper in rebuttal. However, as discussed elsewhere, I still have concerns on the magnitude of the changes in the revision and remaining uncertainty on experiment methodology and comparisons.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In cooperative MARL, following training with a given set of partners, it is important that the train agents be able to zero-shot coordinate with novel partners. 
A common failure mode, entitled symmetry breaking, happens when the trained agents learned on out of many equivalent (thanks to the game's symmetries) but mutually incompatible policies, yielding high performance with training partners but very low performance with novel partners.

To solve this failure mode, this paper proposes to build equivariant network architectures (assuming agents are modelled with deep neural networks, it means that "symmetric changes to their observation cause a corresponding change to their output"), in order for agents in Dec-POMDPs to learn policies that do not break symmetries.

Previous papers, like Other-Play which proposes an optimization algorithm that trains "agents to maximize expected return when randomly matched with symmetry-equivalent policies of their training time partners" via data augmentation, do not have equivariance guarantees.

Instead, this paper proposes Equivariant Coordinator (EQC) which has the following features:
- it comes with theoretical equivariance guarantees - the paper provides mathematical and numerical evidences; 
- it encodes the constraint of equivariance over the game's symmetries "not in the data, but in the network itself".
- it can be applied in two ways, either during training time (via a tractable approximation) or can be used as a policy-improvement operator at test-time (without approximation).

Comparing EQC with state-of-the-art and baseline approaches in zero-shot coordination (like OP, SAD, VDN) shows better performance, independantly of the way it is applied.
strengths_and_weaknesses: # Originality :
To my knowledge, the paper is the first to propose to build equivariant network architectures to encode environment's symmetries, in the context of zero-shot coordination.

# Quality : 
The quality of the paper is high, the maths seems very sound to me, and the experiments are compelling and well-rounded, as far as I can see.

Reproducibility seems strong to me.

Nevertheless, there is one quality/soundness issue around the strength of the numerical evidence:
--> line 287 :  Given the size of the standard deviations, I would appreciate that the authors perform a more thorough investigation using statistical tests to have a quantitative measure of the extent with which the hypothesis is actually supported by the results presented. For instance, using two-sample Kolmogorov-Smirnov tests between each condition's distribution, with the alternative argument set as 'less' or 'greater' [1].

[1] : SciPy's two-sample Kolmogorove-Smirnov test : https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html


# Clarity :
I have found the paper to be mainly well-written and easy to follow.
That being said, I would invite the authors to address the following, mainly to increase accessibility and completeness of the paper:

--> line 102 & 140 : Clarity : I feel I need more details on the nature of $\Psi$, why defining it as a **class** of symmetries rather than a set of symmetries, for instance? It may be obvious but I am missing the point, I am sorry.

Could this choice be influenced by the fact that $L_g$ and $K_g$ /$K_g^{-1}$are denoted as"permutation matrices" rather than functions? (line 140)
Their usage in Eq.2 would invite for a functional definition, if I understand correctly. as they operate on $x,\psi(x)\in (\mathbb{R}^d)^2$ ...

--> line 157: Clarity : I would invite the authors to provide an easier reading experience by detailing that proof:

$$
\begin{align} 
\pi ^ 1(a | \tau ) &= \pi^1( \phi\circ\phi^{-1}(a) | \tau ) & \text{from } id=\phi\circ\phi^{-1} \\
&= \pi^1(K_{\phi}\circ\phi^{-1}(a) |\tau) & \text{by identification}\\
&= \pi^1( \phi^{-1}(a) | L_{\phi}(\tau) ) & \text{from Eq.2}\\
&= \pi^1(\phi^{-1}(a) | \phi(\tau) ) & \text{by identification}\\
&= \pi^2(a|\tau) & \text{from }\pi^1=\phi(\pi^2)
\end{align}
$$

--> line 200 : Clarity : the denomination "first approach" is ambiguisly used, maybe? Isn't G-OP the first approach, while the second approach is to use the *symmetrizer* $S$ on SP-trained $\psi$ and deploy $S(\psi)$?

--> line 220-221 : Clarity/Soundness : I am not sure I can agree with the last sentence: I do not think it is a fair description to say that the present method does not require G to be fully known, as the symmetrizer relies on it to be fully known, no?

I agree that, in practice, the paper shows numerical results that a subset of all the environment symmetries ***may*** be sufficient, but the theoretical evidence requires full knowledge of them, if I understand correctly.
Therefore I would invite the authors to rephrase here.

--> Table 1 : missing result in Table 1, for S5-equivariant agents, maybe? Line 257 seems to refer to it but those results are not in the table.


# Significance :
I think the paper is rightfully placed in the literature and very significant to the field of machine learning as zero-shot/human coordination is a very important problem for the field going forward. 
While the theoretical evidence are, in my viewpoint, very significant, I must confess that I am left disappointed with the numerical results not providing, at first glance, a very significant improvement to the state-of-the-art.
Thus, I am hoping that the statistical analysis that I am inviting the authors to perform will provide a compelling statistical significance argument to the numerical results, hopefully.
If not, then at least readers will know that there is still legitimately some work to do here.


questions: Q0 : Maybe the most important point for me, could you provide statistical analysis of the significance of the numerical results, possibly using two-sample KS tests as detailed above, please?

Addressing this would allow me to raise up my appreciation of the paper.

Q2 : Could you rephrase around line 220 in order to clearly differentiates the theoretical needs and guarantees from the practical ones, please?

Q1 : Could you clarify the denotation of "permutation matrices" as opposed to functions, maybe, as detailed above, please?
limitations: I believe the authors have mainly adequately addressed the limitations and potential negative societal impact.
Nevertheless, I would welcome the paper from investigating and/or discussing further the possible limitations that are the root cause of the numerical results falling short of solving the benchmark, maybe?


## Post-Rebuttal Changes:

I am raising my rating of the paper from 7 to 8.
Indeed, following the revisions made by the authors, I find the theoretical guarantees to be now both impactful and also sufficiently numerically evidenced.
review_title: null

Meta-review: metareview: This paper proposes a test-time algorithmic modification to address a multi-agent coordination problem where agents choose incompatible strategies due to symmetries in the environment, showing that it is applicable to ZSC tasks like Hanabi. The proposed symmetrizer is applied to LSTM recurrent policies.

Reviews were mixed for this paper, and I really valued the in-depth discussion amongst reviewers and between reviewers and authors. I am recommending acceptance primarily based on the interesting discussion and new experimental results surfaced between reviewer Y7M6 and the authors:
1. The feedback from reviewer Y7M6 and authors increased my confidence in the statistical correctness of scores wrt baseline, and it is clear that the authors ensured that the baselines were computed fairly. 
2. The rebuttal-revised version of the paper shows that the symmetrizer improves upon any self-play algorithm, such as OBL level 4 and 5. Reviewer Y7M6 points out "I think it would be better to claim that this work can improve state-of-the-art competitive models to become better instead of claiming this work achieves the new state-of-the-art ZSC performance in Hanabi. ". I believe the authors have shown that with their experiments on OBL L5.

The primary critique is that many of the scores were changed late in the review process, which does call into question whether the empirical rigor should be re-examined via another round of review. The authors have gone to great lengths to improve the rigour in the rebuttal. Beyond the specifics of empirical evaluation, I think the idea presented in the paper is interesting and worth sharing with the community: this paper presents an "explicit symmetrizer", whereas to my knowledge, most of MARL focuses on "implicit symmetrizers" like OBL, well-tuned MAPPO, data augmentation, neural fictitious play-like schemes. Furthermore, the technique appears to be quite complementary and can be combined with existing MARL algorithms. 

Some asks: 
- this paper uses a combination of training and inference time symmetrizers, so please make it clear in the final version of the paper how each of these contribute to the stated performance gains. 
- it would be helpful to mention some of the statistical comments around "In our experience, a pool of 12 policies achieving 2.52 is particularly unlucky and is the result of several seeds that happen to completely misalign with all other policies." in an appendix, even if it's sort of anecdotal. 


confidence: Less certain
Decision: Accept