Equivariant Networks for Zero-Shot Coordination
Authors: Darius Muglich
Christian Schroeder de Witt
Elise van der Pol
Shimon Whiteson
Jakob Nicolaus Foerster
Venue: NeurIPS 2022 Accept (2022)

Abstract: Successful coordination in Dec-POMDPs requires agents to adopt robust strategies and interpretable styles of play for their partner. A common failure mode is symmetry breaking, when agents arbitrarily converge on one out of many equivalent but mutually incompatible policies. Commonly these examples include partial observability, e.g. waving your right hand vs. left hand to convey a covert message. In this paper, we present a novel equivariant network architecture for use in Dec-POMDPs that prevents the agent from learning policies which break symmetries, doing so more effectively than prior methods. Our method also acts as a "coordination-improvement operator" for generic, pre-trained policies, and thus may be applied at test-time in conjunction with any self-play algorithm. We provide theoretical guarantees of our work and test on the AI benchmark task of Hanabi, where we demonstrate our methods outperforming other symmetry-aware baselines in zero-shot coordination, as well as able to improve the coordination ability of a variety of pre-trained policies. In particular, we show our method can be used to improve on the state of the art for zero-shot coordination on the Hanabi benchmark.

Equivariant Networks for Zero-Shot Coordination
Darius Muglich University Oxford wtw.ummm970yyaooooo com
Christian Schroeder de Witt FLAIR, University of Oxford
Elise van der Pol Microsoft Research A14Science
csêrobots. ox ac. uk
evanderpol wtw.rpoott.irroooo.ooooo com
Shimon Whiteson
Jakob Foerster
University of Oxford
FLAIR, University of Oxford j jakob. fegg ox. uk
shimon. shiteson@cs ox ac .uk
Abstract
Successful coordination in Dec-POMDPs requires agents to adopt robust strategies and interpretable styles of play for their partner. A common failure mode is symme- try breaking, when agents arbitrarily converge one out of many equivalent but mutually incompatible policies. Commonly these examples include partial observ- ability, e.g. waving your right hand left hand to convey covert message. In this paper, we present novel equivariant network architecture for use in Dec-POMDPs that prevents the agent from learning policies which break symmetries, doing so more effectively than prior methods. Our method also acts as a "coordination- improvement operator" for generic, pre-trained policies, and thus may be applied at test-time in conjunction with any self-play algorithm. We provide theoretical guarantees of our work and test on the AI benchmark task of Hanabi, where we demonstrate our methods outperforming other symmetry-avee shot coordination, as well as able improve the coordination ability of pre-trained policies. In particular, we show our method can be used to improve on the state of the for zero-shot coordination on the Hanabi benchmark.
baselines in zero- variety
1 Introduction
A popular method for learning strategies in partially-observable cooperative Markov games is via self-play (SP), where a joint policy controls all players during training and test time [2 37 45
While SP can yield highly effective strategies, these strategies often fail in zero-shor coordination (ZSC), where independently trained strategies are paired together for one step at test time. common cause for this failure is mutually incompatible symmetry breaking (e.g. signalling 0 to refer to a "cat" vs. a "dog"). Specifically, this symmetry breaking results in policies that act differently in situations which are equivalent with respect to the symmetries of the environment
To address this, we interested in devising equivariant strategies, which we illustrate in Figure I Equivariant policies are such that symmetric changes to their observation cause a corresponding change their output. In doing so, we fundamentally prevent the agent from breaking symmetries over the course of training.
In earlier work, the Other-Play (OP) learning algorithm 22 addressed symmetry in this setting by training agents to maximize expected return when randomly matched with symmetry-equivalenr policies of their training time partners. This learning rule is implemented by independently applying a random symmetry permutation to the action-observatton history of each agent during each episode. OP, however, has some pitfalls, namely: 1) it does not guarantee equivariance; 2) it poses the burden of learning environmental symmetry onto the agent; and 3) it is only applicable during training time and cannot be used as a nopicy-improvemmnt operator.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Observation
Action
Observation
Action
Hint 2
Hint Green 2
Hint 2
Hint Red 2
Hint 3
Hint Yellow 3
Hint 3
Hint Purple 3
Color Invariant Policy
Color Equivariant Policy
Figure I: Illustrating different kinds of symmetry-robust policies, where the symmetries in this example are colors. Policies make actions based on the number of circles and the color. Invariant policies act irrespective of the change in color (i.e. only the number of circles matters), while equivariant policies act correspondence with the change in color (i.e. changing the color will cause corresponding change to the action).
To address all of this, our approach to encode symmetry not the data, but in the network itself.
There are many approaches to building equivariant networks in the literature. Most work focuses on analytical derivations of equivariant layers 49 ST 52). which can prove time-consuming to engineer for each problem instance and inhibits rapid prototyping. Algorithmic approaches to constructing equivariant networks are restricted layer-wise constraints [TO] T7
18 33], which require computational overhead to enforce. Furthermore, for complex problems such as ZSC in challenging benchmarks such as Hanabi [3), these approaches are insufficient because they are not scalable; and 2) they usually do not allow for recurrent layers, which are the standard method for encoding action-observation histories partially observable settings.
In this paper, we propose the equivariant coordinator (EQC). which uses our novel, scalable approach towards equivariant modelling Dec-POMDPs. Specifically, our main contributions are:
1. Our method, EQC, that mathematically guarantees symmetry- eqquvvriiicc of multi-agent policies, and can be applied solely test time as a coordination -imprtttinnimpoooemmett operator; and
2. Showing that EQC outperforms prior symmetry-robust baselines on the Al benchmark Hanabi.
2
Background
This section formalises the problem setting 2.1 provides brief primer group-theorete notions 2.2
introduces equivariance (2.3 provides background to prior multi-agent work on
symmetry (2.4 and offers small, concrete example help tie these notions together and facilitate
intuition 2.5
2.1 Dec-POMDPs
We formalise the cooperative multi-agent setting decentralized partially-observale Markov decision process (Dec-POMDD (32) which is a 9-tuple (S,N, {AA) {O'};_, T, R, { T, 7), for finite sets S,N, (At [A [O') denoting the set states, agents, actions and observa- tions, respectively, where a superscript denotes the set pertaining to agent E N (1,... n) (i.e., A and O' are the action and observation sets for agent i, and a² A' and of O are specific action and observation of agent i). We also write A = x A and O x,O', the sets of joint actions

---

and observations, respectively. E S is the state at time and b== { k. where sy is state feature of st. a, E A is the joint action of all agents taken at time t. which changes the state according to the transition distribution S1+1 [ss++1 8t, at). The subsequent joint observation of the agents 0++1 E O, distributed according to 0c+1 U(o++1 | where U x,Lf' observation
features of 0 E O' are notated analogously to state features; that is, o'+1 = [ Ja- The reward T++1 E IR is distributed according to rt-1 ~ R(r++1 $1+1, a.). T is the horizon and y E [0, is the discount factor.
Notating oं = (ai, oi oi.....,.j............. a) for the action-observation history of agent i, agent acts according policy of ~ (a r²). The agents seek to maximize the return, ie., the expected discounted sum of rewards:
JT = Eerrr) Σ
re].
EST
where 72 = (8o, do; 01, ......, 1, Ot, Tt, 8e) is the trajectory until time I.
2.2 Groups
A group G is a set with associative binary operation on G. such that, relative to the binary operation, the inverse each element in the set is also in the set, and the set contains an identity element. For our purposes, each element E G can be thought of as automorphism over some space X (i.e. an isomorphism from X to X), and the binary operation is function composition. The cardinality of the group is referred to as its order. If a subset K of G is also a group under the binary operation of G (a.k.a a subgroup), we denote it K < <G
A group action is function G X - X satisfying er = x (where is the identity element) and (g- ha. = g (hz)). for all g. h E by + gr. can be thought to be "relabeling" the elements of X, which aligns with our definition for Dec-POMDP symmetry introduced Section
particular, for fixed g. we get a function ag X + X given
Groups have specific mathematical properties, such as associativity, containing the inverse of each element, and closure (closure is implicit of the binary operation being over G itself) which are necessary for proving Propositions | and and ensuring equivariance.
2.3 Equivariance
Let G be group. Consider the set of all neural networks & of a given architecture whose first and ultimate layers are linear; that is, V {1b 1o h(v(/))), where S h are linear functions, and @ non-linear function. We say network 1/2 E is equivariant (with respect a group G, or G-equivariant) if
(2) where Ly is the group action of E G on inputs, Kg is similarly the group action on outputs, and is the dimension of our data. satisfies Equation 2 then we say vi E L equiv. In the context of reinforcement learning, one may consider ub as the neural parameterization of a policy of so a policy is G-equivariant if
K (x)(x) = (Lgx), for all E G.
x (K2(0) () T(a (T)), for all g E G and for all
(3)
If for any choice of g have that K2 I, the identity function//atrix, then we say л (or v/) is invariant to g.
2.4
Other-Play and Dec-POMDP Symmetry
The OP learning rule [22 of joint policy, т (n² ,²)), is formally the maximization of the following objective (which we state for the two-player case for ease of notation):
arg max E Eg J (()²))
(4)
'To disambiguate between actions of mathematical groups and actions reinforcement learning agents, we always refer the former as "group actions". " We use ti and for this reason more or less interchangeably.
3

---

where @ is the class of equivalence mappings (symmetries) for the given Dec-POMDP, so that each & is an automorphism of S, O whose application leaves the Dec-POMDP unchanged up to relabeling (Φ is shorthand for db { AA (o)). The expectation in Equation is taken with respect to the uniform distribution on +
Each Φ E & acts on the action-observation history as
((+)) - (o(ab), ((0
(5)
and acts on policies as
A= ((T) = =(Ø (a) ((T)))))))))))))))))))))))))() x(a r).
(6)
Policies n, i in Equation correspond with Equation we have La - {é 6o) and K = Φ
6 are said to be symmetry-equivvalntt one another with respect to Φ. To
2.5 Idealized Example
Consider a game where you coordinate with an unknown stranger. Each of you must pick lever from set of 10 levers. Nine of the levers have written on them, and one has 0.9 written. you and the stranger pick the same lever then you are paid out the amount that is written on the lever, otherwise you are paid out nothing.
We have G == Sq < 4, where S9 is the symmetric group over elements (all possible ways to permute elements), since the nine levers with written them are symmetric. If you (using policy r²) pick the 1-point lever ( and the stranger (using policy x²) picks the 1-point lever 12, then n' and ? would be eymmetry-equivaler with respect any Φ E that permutes the lever choices (c.g. 6(41) = 12. or equivalently (41) = (44)) l2. as the levers are both observations and actions). If, as is likely, 4 + 122 you and the stranger leave ampty-handed illustrating symmetry breaking. If during training you and the stranger constrained yourselves to the space of equivariant policies whilst trying to maximize payoff, you would both converge to the policy that picks the 0.9-poin lever (which is not only equivariant, but invariant to all á E G), and so the choice of using equivariant policies allows symmetry to be maintained and payout to be guaranteed.
Method
We now present our methodology and architectural choices, well as several theoretical results.
3.1 Architecture
We first consider the symmetrizer introduced in (35). which functional that transforms linear maps to equivariant linear maps. Inspired by this we present generalisation of the symmetrizer that maps from to the equivariant subspace, Vequiv namely S L Veguiv which we define
5(v)
1
ΣΣ K (L,),
G
9EG
so that the first layer of db denoted f, being linear, is composed with each Lg. and the ultimate layer, denoted h, is similarly composed with K, (i.e. K KKt K h(b(s o Lg)), where = h()()).
Below we establish properties of that were analogously shown for the linear symmetrizer case Propesition 1. (Symmetric Property) S(b) T equiv: for all 1a L; that is, S maps neural networks to equivariant neural networks.
35
Proposition 2. (Fixing Property) V equi Ran(S); that is, the range of S covers the entire equivari- and subspace.
The proofs of these propositions can be found in the Appendix. The proof of Proposition | reelies on properties of the group structure (such closure of the group); it is thus critical for the symmetrizer to sum over permutations that together form a group structure, and not any random collection of permutations.

---

Propositions and 2 show in combination that the architecture of (( is indeed equivariant, and that optimizing over the class RSn(S) (that is, keeping the permutation matrice and K for all y frozen while updating the other weights) is equivalent to optimizing over V "quiv- The sum over each g E G of ((t) highly parallelizable (19], which allows for maintaining tractability. this way we algorithmically enforce equivariance, mitigating the need to design equivariant layers by hand, as is typical
12 48
49
51 52].
We note that for a network containing LSTM [0 layers, S will still map the network to an equivariant one no matter the values used for the hidden and cell states; indeed, Propositions || and 4
all hold regardless of how the hidden and cell states are treated. However, how the hidden and cell states are treated matter for the overall performance of the network (equivariance is not a sufficient condition itself for success in ZSC, consider the uniformly random policy). We ergo propose the following: for each hidden state higg and cell state ot.g produced by the LSTM layer over input (I 14-1, we compute h2 |al Egec hi.g and e₁ = |G Egec Ct.g. and proceed inductively. This non- obvious design choice, since each higg and Cr.g is the output choice would be to choose h, = ft..c and c Ct.e. where E G' is the identity element, but to ensure symmetry-awand play we experimented with this averaging scheme to "symmetrically adjust" the short and long-term memories. Since all the permutations over the input computed in parallel with matrix multiplication, this design choice retains tractability. We test this choice empirically in See the Appendix for further experimentation surrounding this design choice.
non-linear function; a more obvious
Section 4l
3.2 A Theory for Symmetry And Coordination
The following propositions provide guarantees for EQC's ability prevent all symmetry breaking: Proposition 3. If n¹ and ? are symmeety-eqqiivllen policies with respect to G and x! is G-equivariant, then πt ==
Proof. We have n5 )(T²) for some & E G. meaning म (a- <(T)) = 2a r). Identifying L& {ox, )o) } and K3 óAA. by equivariance (Equation observe that π (a 7) π (Ko
(a) r) x r'KK'' (a TT).
6 (a) [L(T)) = x ((r)). Thus n (a r) л (a) ((())
Proposition 4. a and ?t are symmetry-equiivllen policies with respect to é E G, then S(x*) = S(m²).
Proof. This is a corollary of the proof of Proposition D| such that the symmetrizer architecture is fixed under any Φ E G.
G-equivariance thus reduces cross pray to self-play between policies that are symmetry-equivalee with respect to any & E G. This is critical, as this means equivariance merges policies of large classes together, such that symmetry-equivaate another way, the set of policies that are symmetry-equivalent with respect to any & E G form an equivalence class, and G-equivariance reduces the cardinality of each equivalence class 1, thus allowing fluent coordination of policies within an equivalence class (as it merely becomes self-play).
policies now indeed become equivalent in coordination. Put
3.3 Algorithmic Approach
Based on Propositions T and 2 a "naive approach" to obtaining a G-equivariant agent, for a given group G, might then be follows: 1) initialize a random seed vi E v; 2) map 1/ to S(v); 3) train (()) using any self-play algorithm, updating the weights while keeping the permutation matrices L, and K- for all frozen; 4) S( is ready for inference. This naive approach is sound, but time and memory complexity inhibit scaling to large choices of G. For this reason we present the following two options (that together summarize EQC):
Any group can be viewed as a group of permutations, or as group of "permutation matrices": this is known as the permutation representation of group [9]. Also, the terminology "permutation matrix" alludes to the implementation of group actions matrix products.
5

---

1. As a first option, we introduce "G-OP", where during training we sample random subsets of G for each mini-batch (i.e. we select a random g G for each x in the mini- so that group-theoretically acts on x, and update the weights of <b) and at test time, we use all the permutations of by deploying S(1)).
2. As second option, we train under any self- self-play algorithm and at test time deploy S(b))
With Option we train over the permutations of G' at training time. With Option 2 we train under standard SP and use S(.) to obtain (( at test time, which can be viewed as a coordination- improvement operator. We empirically validate both options in Section From here on, refer to ((b) as "symmetrizing v²".
We expect compatibility between the G-OP trained agents and the symmetrizer at test time, since at training time, these agents were trained over these permutations. For Option 2, where we have network & not trained with permutations, it is worthwhile to inspect S(Ø); while we know S()) is equivariant, we know little else regarding the conventions and style of play it would use. While symmetry is critical component of ZSC, other factors (such as choosing robust actions that are likely to succeed under minimal assumptions 2 [3] )) are also significant. The following points justify why S maps any network to one that is not only equivariant, but overall stronger at ZSC: 1) the network averages over how z would act under each E G. thus ensuring play that accounts for multiple symmetries; 2) the group closure property fixes the architecture under any permutation in the group (see the proof of Proposition D forcing consistent play across different symmetries. We verify the empirical efficacy of Options and in Section
What remains is the choice of the group for our G-equivariant agents. One might consider the choice of G 4, but this in general scales factorially with the size of the Dec-POMDP (e.g. the automorphism group over an environment with m symmetric features grows asymptotically with the number of permutations over m elements), and quickly becomes infcasible. Hence, we treat G as a hyperparameter, and in Section we analyze how different choices of G affect ZSC, In any case, picking G subgroup of Φ may be motivated in and of itself, when enumerating all possible symmetries of an environment is intractable, or if an environment is obfuscated such that some symmetries cannot be known (as is the case in many large, realistic domains). In this way, our method can relax the assumption of requiring access to all environment symmetries by flexibly choosing equivariance with respect subgroup.
Experiments
The primary test bed for our methodology is the Al benchmark task Hanabi [5] Hanabi is a unique and challenging card game that requires agents to formulate informative implicit conventions in order to be successful. Hanabi has served as the primary test bed for many algorithms designed for ZSC due to its representation introduction to Hanabi).
22 23 28
formidable Dec-POMDP task (see the Appendix for an
To make sense of the ensuing results, recall that a perfect score in Hanabi is 25, and that "bombing out" refers to expending all three fuse tokens and ending the game with a score of O. Agents thus must be cautious not to expend all fuse tokens.
In Hanabi, the class of environment symmetries are the permutations of the card colors, because assuming there is no supplemental information, a permutation of the colors of the cards leaves the game unchanged. We therefore have & S5, the symmetric group over 5 elements (all possible permutations over 5 elements). 5 here represents the number of different colors. |Ss| 5! = 120.
In this section analyze differences in play and the relationship to environmental symmetry between our equivariant agents and OP agents [). and evaluate various pre- pre-trained agents before and after symmetrization (4.2). These subsections evaluate Options and 2, respectively (see Section B
For further details of the training setup and hyperparameters used in the following experiments, please refer to the Appendix.

---

Table 1: Comparing OP agents with our equivariant agents (cach G-equivariant agent is trained using G-OP then symmetrized at test time) for their respective cross-play (XP) abilities. Each agent is trained with a different seed. Each pair of agents was evaluated over 5000 games, with the total averages compiled here. The error bars are the standard error of the mean.
XP Stats
OP
C3- Cg-Equivariant
Diu-Eon Puu-Equiarriant
Average Scores
15.32 + 0.65
15.45 = 0.47
16.08 = 0.42
Average Bombout Rate
30.8%
24.3%
19.7%
4.1 Symmetry-Robust Agents
Here we compare the 12 OP agents from [2] with our equivariant agents. Specifically, we conduct the following ablation study: we fix two subgroups of S5 Cs and D10. which are the cyclic group of order and the dihedral group order 10, respectively; the geometric and/or algebraic interpretations of these groups is not important, for us groups purely serve as structured collections of permutations D.. Next, we train 10 agents with the CG-OP learning rule (see Section Bb. then map these 10 agents to equivariant ones using the -ysymmmetrizer. Similarly, we train 10 agents with the D1o-OP learning rule, then map them with the D Doo-symmetrizer Finally, we compare the ZSC performance between these three pools of 10 agents. This is an ablation, because we control for the order of the group by considering both C3 and D10 (in fact, C3 is a subgroup of D10) so to observe the effect of group order. Each agent uses Recurrent Replay Distributed Deep Q-Networks (R2D2) [25] and is trained
using value decomposition networks (VDN) (43) and the simplified action decoding (SAD) algorithm
(2T] The average self-play scores achieved by the OP agents is 23.93 + 0.02, of the Cg-equivariant agents is 23.96 + 0.03, and the Dio-equivariant agents is 23.95 + 0.03.
Table shows the findings of our ablation analysis. We see that both equivariant pools outperform the OP agents, in spite of the OP agents being trained over far more permutations than any of our equivariant agents: |Cb] 5, |o 10, |Sp| 120 (the OP agents are trained on permutations from Sz). This is consistent with earlier work where equivariant networks outperform data augmen-
tation approaches 36 47 49 50 52 55). We conduct Monte Carlo permutation test [15] 16 to compare the P1o-equivarian agent with the OP agents, for which we bound the derived p- value in a 99% binomial confidence interval.
We also observe that the Dio Dio-equivariant agents outperform the Cs-equivariant agents, which we largely attribute to Cs < Dıo and Propositions 33 and g if we have group G < $. and choose & uniformly at random, then P(ś E G) [
and so for two groups G, H with |G| S [H) we
have p( E G) & P( E H).
4.2 Equivariance as a colicy-Improvement Operator
Here we consider the extent to which test-time symmetrizing improves ZSC performance across diverse policy types: we train 10 IQL agents [44). and use the 12 OP agents and 12 SAD agents from (22) and the agents from 23
(level 5), and examine the effect of transforming each of these self-play trained agents with our equivariant architecture at test time. This selection of agent types diverse in terms of both overall performance level as well as the style of play. Furthermore, we consider agents that are both trained for cross-play (OP and OBL) and trained for the self-play setting (SAD and IQL), which is an important dimension to account for in consideration of EQC's applicability. All agents use R2D2 25]. and the SAD and OP agents also use VDN 43) The average self-play scores achieved by the SAD agents is 23.97 + 0.04, the IQL agents is 23.15 0.02, the OP agents is 23.93 + 0.02, and the OBL agents is 24.20 + 0.01.
Table summarizes our findings over the policy types, where we consider symmetrization with both the cyclic group of order 5, Cs. and the dihedral group of order 10, D10- We find that mapping any of these networks through EQC with either choice of group improves its cross-play ability with novel agents at test time. We conduct Monte Carlo permutation tests [1] [6] to compare the Dio -o-smmmmetrized agents with their unsymmetrized counterparts for each policy type, for which we bound each derived p-value in a 99% binomial confidence interval. The p-values for the respective one-tailed tests for each of the SAD, IQI., OP and OBL agents are found to be within (0, 0.00028), (0, 0.00028), (0, 0. 0.00028), and (0.0825, 0.0973] respectively, making the differences significant at
7

---

Table 2: Cross-play (XP) scores and bombout rates of various diverse policy types symmetrized at test time. Each agent is trained with a different seed. Each pair of agents was evaluated over 5000 games, with the total averages compiled here. The error bars are the standard error of the mean.
XP Stats
SAD
IQL
OP
OBL-L5
W/o symmetrizer
2.52 + 0.34
10,53 + 0.78
15.32 + 0.65
23.77 + 0.06
C3 symmetrized
3.61 + 0.38
13.57 0.66
16.07 = 0.59
23.77 + 0.05
D 10 symmetttri
3.61 = 0.39
13.62 + 0.65
16.48 + 0.53
23.89 + 0.04
Bombout Rate Stats
SAD
IQL
OP
OBL-L5
W/o symmetrizer
87.2%
39.4%
30.8%
0.56%
Cn-symmetrized
81.9%
33.3%
26.1%
0.33%
D Dio-symmetrized
82.0%
33.3%
24.6%
0.22%
the level a = 0.01 for the SAD, IQL, and OP agents, and at the level a = 0.1 for the OBL agents. Therefore, the improvement in coordination found by symmetrizing agents is highly statistically significant.
In particular, see that OP agents improve in 7SC when symmetrized. In the proof for Proposition 2 we showed that equivariant networks are fixed under the symmetrizer. Since these OP agents trained under the same permutations they are being symmetrized with, yet still improve upon symmetrization, this suggests that the data augmentation of OP does not lead to full equivariance, and so encoding environmental symmetry the model level is necessary to realise crucial properties such as Propositions 3 and g
as well as to improve overall performance.
It is especially significant that EQC maps networks not trained with permutations (SAD, IQL, OBL) to ones that are overall better at ZSC; we justify the mechanism by which it accomplishes this in demonstrate that EQC zero-shot coordination on the Hanabi benchmark at the time of writing.
Section 3 In particular,
improve on OBL, the state of the art for
Figure shows the conditional action matrices (i.e. P(a a(-1)) of a symmetrized and unsym- metrized IQL agent (see the Appendix for the plots of other agents). This plot informs differently the agent responds to possible actions of its partner, where large differences indicate specialized play, and therefore likely discrepancy between the conventions used by this agent and potential partners, which would be detrimental to -lays We see that the IQL agent has learned specialized conventions (for example, hinting the fourth color means for the partner to discard their first card), but that upon symmetrization its actions and responses become more consistent.
how
Another observation is that the Pro-symmetrize does not always outperform the Cs-symmetrizer: namely for the case of the SAD agents terms of bombout rate. The combination of C5 D10 and Propositions D and implies that the Dio-symmetrizer corrects for symmetry breaking better than the Cs does, so the worse bombout rate may only be attributed to the kind of play the symmetrized policy encourages, with a likely explanation as follows: if a policy is such that it acts very differently in each symmetry, then symmetrizing this policy over a large number of symmetry permutations will result in policy with higher entropy (i.c. averaging over large number of very different distributions results in one that is approximately uniform (see the Appendix)). But behaving "randomly" is not in general conducive to effective coordination, where one should try to use unambiguous conventions to facilitate play. Thus, when symmetrizing an arbitrary network, we tacitly assume some degree of regularity of the network across symmetries (we control this regularity when we opt for the proposed G-OP learning rule). Nonetheless, both choices of Cs, and D10 result in better ZSC than without any symmetrization for all the tested policy types (even with SAD, which learns notoriously specialized play [2] 23). suggesting that policies in practice do generally satisfy these regularity requirements. This phenomenon, however, should be kept in mind and motivates future research into deriving the optimal choice of G for G-equivariant agents.
5 Related Work
Zero-Shot Coordination. Zero-shot coordination is the problem of constructing independent agents that can effectively engage with one another in cross-play at test time. Many directions to this
8

---

Cyclic Symmetrized IQL
Dihedral Symmetrized IQL
action R5R4 C5
1.00
actioon
action
0.75
0.50
0.25
0.00
P1
him
22
hint
23
hin
response
response
response
Figure 2: Conditional action matrices of IQL, i.e. P(a o/-1). unsymmetrized (left) and sym- metrized at test time (middle is Cg-symmetrized and right Pro-symmetrized), The y-axis represents the action taken at timesetep t and the x-axis shows the proportion each action as response at timestep I 1. The matrices show the interactions between color/rank hinting and play/discarding. C1-5 and R1-5 mean hinting the 5 different colors and ranks respectively, and P1-5 and D1-5 mean playing and discarding the 1st-5th cards in the hand. We selected random agent, and each plot is thereby computed by running 1000 episodes of self-play with the agent to compute the statistics.
problem have been considered: 23 explore controlling the cognitive-reasoning depth so avoid formation of arbitrary conventions that can complicate cross-play. However, [14) is computationally exacting, requiring days of GPU running time, and 23 requires simulator (environment) access. (8) offers modular approach to separately learn rule-dependent and convention-dependee
behaviors to facilitate coordination, but their method comes the expense of scalability, with experiments limited to bandit problems and 1-color Hanabi. [4 uses repeated eigenvalues of the Hessian during training find policies suited for 7SC, but again inhibited by scalability, with experiments focused on lever games. 31 investigate using life-long learning for ZSC, but their approach requires access to pool of pre-trained policies. (22). which we have referred at length throughout this paper, takes to data augmentation to train symmetry- robust agents. EQC can be used along with any of these approaches (even with [2 23), as we demonstrated in Section so to correct for symmetry breaking and to encourage robust play.
Ad-Hoe Teamwork. Similar zero-shot coordination is the ad-hoe teamwork setting 6 30
where an agent is pitted with an existing group of agents, and is tasked with learning to coordinate with this group during interaction. While ZSC and ad-hoc teamwork are closely related problems, they differ in that ad-hoc team play seeks policies that do well with arbitrary agents (often an undefined problem setting), while ZSC agents only need to perform well with agents that are optimized for ZSC using a common high-level approach. In principle EQC can also be used to improve performance in ad-hoc teamplay, which we leave for future work.
Equivariance. Equivariant networks are neural networks with symmetry constraints built in, such that for a transformation the input, the output is appropriately transformed well Equivariant policies have shown improved data efficiency in single agent reinforcement
49
52
learning
54 In the multi-agent case, prior works use permutation symmetries to ensure agent anonymity 8 24 27
29
33
43] or rotational equivariance over multiple agents (36). While these approaches consider equivariance in multi-agent settings, they not tackle the problem ZSC. In terms of architecture, the symmetry ensemble of (39] is related to ours, but we propose adaptations that tailor recurrent layers to multi-agent coordination, thus suiting our design not just for Markov Decision Processes (MDPs) but for more general POMDPs and Dec-POMDPs.
6 Conclusion
In this work we proposed equivariant modelling to solve symmetry breaking. To this aim we presented EQC, which we validated via mathematical proof and experimentation over the complex Dec-POMDP task of Hanabi. We showed that EQC greatly improves on [22) by guaranteeing conmetry-equivariinee of policies, and can used as a policy-improvement operator to prevent symmetry breaking and promote robust play for diversity of agents. In this way, we also showed the extent to which symmetrizing improves overall performance in coordination. Furthermore, we showed that EQC is "symmetry efficient", such that it does not require access all environment

---

symmetries in order to perform well. In addition, we empirically validated that agents perform well under our proposed architectural choices.
There are many directions for future work. One is to consider how to derive optimal choices of the group G' for G-equivariant agents effective 7SC, relative given policy type and task. Another direction is exploring how to efficiently uncover environment symmetries from a domain when they not given nor assumed be known. While our work focused on symmetry breaking, future work could explore other fundamental aspects of coordination.
Acknowledgements
The experiments were made possible by a generous equipment grant from NVIDIA. Christian Schroeder de Witt is generously funded by the Cooperative Al Foundation. We thank Jacob Beck for his constructive criticism of the manuscript.
10

---

References
[]] Farzad Abdolhosseini et "On Learning Symmetric Locomotion". In: ACM SIGGRAPH
Motion, Interaction, and Games. 2019.
[2] Marina Agranov et al. "Beliefs and endogenous cognitive levels: An experimental study". In: Games and Economic Behavior 75.2 (2012), pp. 449-463
[3] Brandon Anderson, Truong Son Hy, and Risi Kondor. "Cormorant: Covariant molecular neural
networks". In: Advances in neural information processing systems 32 (2019).
[4] Karl Astrom. "Optimal control of Markov decision processes with incomplete state estima- tion". Im: J. Math. Anal. Applic. 10 (1965), pp. 174-205.
[5]
Nolan Bard et al. "The hanabi challenge: A new frontier for ai research". In: Artificial Intelli-
gence 280 (2020), p. 103216.
[6] Samuel Barrett, Peter Stone, and Sarit Kraus. "Empirical evaluation of ad hoc teamwork in the pursuit domain". In: The 10th International Conference on Autonomous Agents and Multiagent Systems-Volus 2. 2011, pp. 567-574.
[7] Richard Bellman. "A Markovian decision process". In: Journal of mathematics and mechanics (1957), pp. 679-684.
[8] Wendelin Böhmer, Vitaly Kurin. and Shimon Whiteson. "Deep coordination graphs". In: International Conference on Machine Learning. PMLR. 2020.
[9] Arthur Cayley. "VII. On the theory of groups, as depending on the symbolic equation On= 1". In: The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 7.42 (1854), pp. 40-47.
[10] Gabriele Cesa, Leon Lang, and Maurice Weiler. "A Program Build E (N)-Equivariant Steerable CNNs". In: International Conference on Learning Representations. 2021.
[11] Taco Cohen and Max Welling. "Group equivariant convolutional networks". In: International conference machine learning. PMLR. 2016, pp. 2990-2999.
[12] Taco Cohen and Max Welling. "Steerable cuns". In: arXiv preprint arXiv: (2016).
[13] Miguel Costa-Gomes and Vincent P Crawford. "Cognition and behavior in two-person guessing games: An experimental study". In: American economic review 96.5 (2006), pp. 1737- 1768.
[14] Brandon Cui et al. "K-level Reasoning for Zero-Shot Coordination in Hanabi". In: Advances in Neural Information Processing Systems 34 (2021).
[15] Meyer Dwass. "Modified randomization tests for nonparametric hypotheses". In: The Annals of Mathematical Statistics (1957), pp. 181-187.
[16] T Eden and F Yates. "On the validity of Fisher's z test when applied to an actual example of non-norma (With five text-figures.)" In: The Journal of Agricultural Science 23.1 (1933), pp. 6-17.
[17] Marc Finzi, Max Welling, and Andrew Gordon Wilson. "A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups". . In: International Conference Machine Learning. PMLR. 2021, pp. 3318-3328.
[18] Jonathan Gordon et al. "Permutation equivariant models for compositional generalization in language". In: International Conference on Learning Representations. 2019.
[19] Maurice Herlihy and Nir Shavit. The Art of Multiprocessor Programming, revised first edition.
Morgan Kaufmann, 2012.
[20] Sepp Hochreiter and Jürgen Schmidhuber. "Long short-term memory". In: Neural computation
9.8 (1997), pp. 1735-1780.
[21] Hengyuan Hu and Jakob N Foerster. "Simplified action decoder for deep multi-agent reinforce- ment learning". In: arXiv preprint arXiv: 1912.. 02288 (2019).
[22] Hengyuan Hu et al. ""Other-Play" for Zero-Shot Coordination". In: International Conference Machine Learning. PMLR. 2020, pp. 4399 4410.
[23] Hengyuan Hu et al. "Off-belief learning". In: International Conference on Machine Learning.
PMLR. 2021, pp. 4369-4379.
[24] Jiechuan Jiang et al. "Graph convolutional reinforcement learning". In: International Confer- ence on Learning Representations. 2020.
[25] Steven Kapturowski al. "Recurrent experience replay distributed reinforcement learning".
In: International conference learning representations. 2018.
11

---

[26] Diederik P Kingma and Jimmy Ba. "Adam: A method for stochastic optimization". In: arXiv preprint /41. 6980 (2014).
[27]
Iou-Jen Liu, Raymond A. Yeh, and Alexander G. Schwing. "PIC: Permutation invariant critic
for multi-agent deep reinforcement learning". In: Conference on Robot Learning. 2019.
[28] Andrei Lupu al. "Trajectory diversity for zero-shot coordination". In: International Confer- ence on Machine Learning. PMLR. 2021, pp. 7204-7213.
[29] Arnab Kumar Mondal, Pratheeksha Nair, and Kaleem Siddigi. "Group equivariant deep reinforcement learning". In: arXiv preprint arXiv:2007.034. (2020).
[30] Darius Muglich et al. "Generalized Belief's for Cooperative Ar". In: International Conference Machine Learning. PMLR. 2022, pp. 16062-16082.
[31] Hadi Nekoei al. "Continuous coordination as a realistic scenario for lifelong learning". In: International Conference on Machine Leurning. PMLR. 2021, pp. 8016-8024.
[32] Frans A Olichock. "Decentralized pomdps". In: Reinforcement Learning. Springer, 2012,
pp. 471-503.
[33]
Jung Yeon Park et al. "Learning Symmetric Representations for Equivariant World Models". In: (2021).
[34] Jack Parker-Holder al. "Ridge rider: Finding diverse solutions by following eigenvectors of the hessian". In: Advances in Neural Information Processing Systems 33 (2020), pp. 753-765.
[35] Elise van der Pol et al. "MDP homomorphic networks: Group symmetries in reinforcement learning". In: Advances in Neural Information Processing Systems 33 (2020), pp. 4199-4210.
[36] Elise van der Pol et al. "Multi-Agent MDP Homomorphic Networks". In: International Conference on Learning Represenlations (2022).
[37] Arthur L Samuel. "Some studies in machine learning using the game of checkers". In: IBM Journal of research and development 3.3 (1959), pp. 210-229.
[38] Andy Shih et al. "On the critical role of conventions in adaptive human-AI collaboration". In: arXiv preprint arXiv:2104. 02871 (2021).
[39] David Silver et al. "Mastering the game of Go with deep neural networks and tree search". In: nature 529.755 75877 (2016), pp. 484 489.
[40] Gregor NC Simm et al. "Symmetry-awarc actor-critic for 3d molecular design". In: Interna- tional Conference on Learning Representations (2021).
[41] Peter Stone et al. "Ad hoc autonomous agent teams: Collaboration without pre-coordination. In: Twenty-Fourth AAAI Conference on Artificial Intelligence. 2010.
[42] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. "Learning multiagent communication
with backpropagation" In: Advances in Neural Information Processing Systems. 2016.
[43] Peter Sunehag et al. "Value-decomposition networks for cooperative multi-agent learning". In: arXiv preprint prXivv/706.000 05296 (2017).
[44] Ming Tan. "Multi-agent reinforcement learning: Independent vs. cooperative agents". In:
Proceedings of the tenth international conference on machine learning. 1993, pp. 330-337,
[45] Gerald Tesauro. "TD-Gammon, self-teaching backgammon program, achieves master-level play". In: Neural computation 6.2 (1994). pp. 215-219.
[46] Nathaniel Thomas et al. "Tensor field networks: Rotation-and translation-cquivariinn neural networks for 3d point clouds". In: arXiv preprint 1802.0880..083 (2018).
[47] Dian Wang, Robin Walters, and Robert Platt. "SO (2)-equivariant reinforcement learning". In: The Tenth International Conference Learning Representations. 2022.
[48] Maurice Weiler and Gabriele Cesa. "General e (2)-equivariant steerable enns". In: Advances in Neural Information Processing Systems 32 (2019).
[49] Maurice Weiler et al. "3d steerable cnns: Learning rotationally equivariant features in volumet- ric data". In: Advances in Neural Information Processing Systems 31 (2018).
[50] Marysia Winkels and Taco Cohen. "3D G-CNNs for pulmonary nodule detection".. In: arXiv
preprint arXiv: / 1804 04656 (2018).
[51] Daniel Worrall and Max Welling. "Deep scale-spaces: Equivariance over scale", In: Advances in Neural Information Processing Systems 32 (2019).
[52] Daniel E Worrall et al. "Harmonic networks: Deep translation and rotation equivariance". In: Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition. 2017,
pp. 5028-5033
12

---

[53] Chao Yu et al. "The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games", In arXiv preprint arX1v:2103.01955 (2021).
[54] Xupeng Zhu et al. "Equivariant Grasp learning In Real Time". In: (2021).
[55] Xupeng Zhu et al. "Sample Efficient Grasp Learning Using Equivariant Models". In: arXiv preprint aurXiv:22022 2204.022000 (2022).
13

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Sections and
(b) Did you describe the limitations of your work? (Yes] See Sections
S and
(c) Did you discuss any potential negative societal impacts of your work? Yes)
See the
Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] In Section and the Appendix
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes] Section 4 the Appendix, and the supplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section +
and the Appendix
(c) Did you report error bars (c.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] In the supplement material
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable
information or offensive content? [N/A] We use rollouts of Hanabi games played by Al agents.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "bH_s3PLaQ7r", "reviewer": "Reviewer_Y7M6", "summary": "This work approaches the zero-shot-coordination (ZSC) problem by explicitly restricting the learnable (or act-able) policies to only those that do not rely on known asymmetric convention in the environment. They do this by using a 'symmetrizer' that can be used to make any regular trained policy equivariant post-training or be used during training to train a network to act more equivariant (although a symmetrizer can still be used with this network on inference as well). Specifically, this work seeks to outperform prior symmetry-aware ZSC algorithms on the benchmark environment Hanabi.", "questions": "- This paper reports results for the baseline in Section 4.1 that are significantly different than the reported results in the baseline papers [19] and [20]. For example, the cross-play scores of Other-Play [20] in this work are 11.12, in the original OP paper they are 15.32 or 22.07 (with or without the auxilliary task, respectively) [20] and in this work SAD cross-play is reported as 9.43 in Table 2 but is reported as 2.52 in the original paper[20] (Table 1). What is the reason for these differences?\n\n- How do you construct group actions suitable for cell state and hidden state i the LSTM as discussed in Proposition 2 in Section 3? In my understanding, the different dimensions in these states do not normally correspond to known observations or actions.\n\n- If possible, I would appreciate more of a walkthrough in why the $\\pi^{\\text{equiv}}$ policy in section 2.5 will always choose the 0.9 lever.\n\n- On page 5, what are the differeneces between the strategy of using \"G-OP\" and using Other-Play? Is it that Other-Play trains with more permutations?", "limitations": "Yes\n\nPost-rebuttal change:\nIncreased score from 3 to 4 to reflect improvement of paper in rebuttal. However, as discussed elsewhere, I still have concerns on the magnitude of the changes in the revision and remaining uncertainty on experiment methodology and comparisons.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This paper extends equivariance to recurrent neural networks\n- This work is well written, especially considering the complexity of the material discussed\n- This work does seem like a further mathemtical advance on eliminating symmetries similar to how Other-Play approaches ZSC\n\nWeaknesses\n- The original models for the basline are freely available, but this work chooses to train their own models. This adds another difference that makes it harder to compare to the baseline and calls into question the accuracy of the results.\n- This paper seems to present an incrementally better solution towards dealing with explicitly defined symmetries in an environment, but more recent work (such as OBL [21]) does not require this explicit definition.\n- I am uncertain of any empirical advantages it provides with the current state-of-the-art in ZSC. For example, Off-Belief-Learning [21], like this work, is also shown to eventually converge to the same policy regardless of the initialization, but achieves better results (20.85-23.76 cross-play) than this work without needing to explicitly define symmetries [21]\n- This work chooses the same set of symmetries as Other-Play, and seems to train the neural networks the same or very similar as Other-Play to be invariant to these symmetries.\n- In my understanding, this work trains its own versions of the baseline models, which seem to get very different scores than the original baseline papers report. However, these original papers publish already trained models that could be used (more details in questions)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ui0iFgAOFWC", "reviewer": "Reviewer_VbG9", "summary": "The authors propose to use equivariant networks to handle the symmetry-breaking problem in multi-agent coordination settings. To generate an equivariant history-dependent policy, the authors present the equivariant LSTM structure and validate its performance in Hanabi games. ", "questions": "About contribution and soundness: \n- In section 3 line 147, the authors proposed a modification for LSTM layers, which is interesting. However, how is the modification here related to propositions 3 and 4 in the following line 155 and line 160? I assume that propositions 3 and 4 are not tailored for LSTM nets. \n - A follow-up question is how the modifications in line 147 affect the equivariance of the proposed LSTM network?\n - What is the effect of the size of the network on the asymptotic performance? I assume that with a larger network and adequate data, the OP may have better performance. The equivariant net for sure helps improve the data efficiency but not necessarily the asymptotic performance. \n\nAbout writing and clarification:\n - The network modification is mysterious. Does the modification to LSTM only include the one mentioned in line 147?\n - In option 1 in line 177, the authors mention a mini-batch, is the batch size equal to 1? What is the effect of the group element batch size? \n - How is the network parameter selected in this paper?\n\nMinor issues:\n - Figure 1 is not referred to in the main content.\n\n", "limitations": "The authors addressed one limitation which is the prior knowledge of the group symmetry, which I think is crucial. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n - I found the work interesting and believe the main technical contributions are (1) extending an equivariant MLP to an equivariant LSTM structure, (2) leveraging the permutation group while prior work mainly leverages the SO(2), (3) extending the equivariant net to game applications.\n\nWeaknesses:\n\nOriginality:\n\n - Although the extension to the equivariant LSTM structure is interesting, I found the paper does not have an analysis tailored to LSTM (see the first two questions in the clarification part). Thus the contribution of this paper is not significant to me, but I look forward to the answers to the clarification questions below.\n\nClarification:\n\n - I found the Section 3 Method hard to follow. It could be improved by listing several subsections instead and highlighting the main contribution as extending the equivariant neural net to the LSTM setting.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Ohn3gFpFFLA", "reviewer": "Reviewer_ef8A", "summary": "In cooperative MARL, following training with a given set of partners, it is important that the train agents be able to zero-shot coordinate with novel partners. \nA common failure mode, entitled symmetry breaking, happens when the trained agents learned on out of many equivalent (thanks to the game's symmetries) but mutually incompatible policies, yielding high performance with training partners but very low performance with novel partners.\n\nTo solve this failure mode, this paper proposes to build equivariant network architectures (assuming agents are modelled with deep neural networks, it means that \"symmetric changes to their observation cause a corresponding change to their output\"), in order for agents in Dec-POMDPs to learn policies that do not break symmetries.\n\nPrevious papers, like Other-Play which proposes an optimization algorithm that trains \"agents to maximize expected return when randomly matched with symmetry-equivalent policies of their training time partners\" via data augmentation, do not have equivariance guarantees.\n\nInstead, this paper proposes Equivariant Coordinator (EQC) which has the following features:\n- it comes with theoretical equivariance guarantees - the paper provides mathematical and numerical evidences; \n- it encodes the constraint of equivariance over the game's symmetries \"not in the data, but in the network itself\".\n- it can be applied in two ways, either during training time (via a tractable approximation) or can be used as a policy-improvement operator at test-time (without approximation).\n\nComparing EQC with state-of-the-art and baseline approaches in zero-shot coordination (like OP, SAD, VDN) shows better performance, independantly of the way it is applied.", "questions": "Q0 : Maybe the most important point for me, could you provide statistical analysis of the significance of the numerical results, possibly using two-sample KS tests as detailed above, please?\n\nAddressing this would allow me to raise up my appreciation of the paper.\n\nQ2 : Could you rephrase around line 220 in order to clearly differentiates the theoretical needs and guarantees from the practical ones, please?\n\nQ1 : Could you clarify the denotation of \"permutation matrices\" as opposed to functions, maybe, as detailed above, please?", "limitations": "I believe the authors have mainly adequately addressed the limitations and potential negative societal impact.\nNevertheless, I would welcome the paper from investigating and/or discussing further the possible limitations that are the root cause of the numerical results falling short of solving the benchmark, maybe?\n\n\n## Post-Rebuttal Changes:\n\nI am raising my rating of the paper from 7 to 8.\nIndeed, following the revisions made by the authors, I find the theoretical guarantees to be now both impactful and also sufficiently numerically evidenced.\n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Originality :\nTo my knowledge, the paper is the first to propose to build equivariant network architectures to encode environment's symmetries, in the context of zero-shot coordination.\n\n# Quality : \nThe quality of the paper is high, the maths seems very sound to me, and the experiments are compelling and well-rounded, as far as I can see.\n\nReproducibility seems strong to me.\n\nNevertheless, there is one quality/soundness issue around the strength of the numerical evidence:\n--> line 287 :  Given the size of the standard deviations, I would appreciate that the authors perform a more thorough investigation using statistical tests to have a quantitative measure of the extent with which the hypothesis is actually supported by the results presented. For instance, using two-sample Kolmogorov-Smirnov tests between each condition's distribution, with the alternative argument set as 'less' or 'greater' [1].\n\n[1] : SciPy's two-sample Kolmogorove-Smirnov test : https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html\n\n\n# Clarity :\nI have found the paper to be mainly well-written and easy to follow.\nThat being said, I would invite the authors to address the following, mainly to increase accessibility and completeness of the paper:\n\n--> line 102 & 140 : Clarity : I feel I need more details on the nature of $\\Psi$, why defining it as a **class** of symmetries rather than a set of symmetries, for instance? It may be obvious but I am missing the point, I am sorry.\n\nCould this choice be influenced by the fact that $L_g$ and $K_g$ /$K_g^{-1}$are denoted as\"permutation matrices\" rather than functions? (line 140)\nTheir usage in Eq.2 would invite for a functional definition, if I understand correctly. as they operate on $x,\\psi(x)\\in (\\mathbb{R}^d)^2$ ...\n\n--> line 157: Clarity : I would invite the authors to provide an easier reading experience by detailing that proof:\n\n$$\n\\begin{align} \n\\pi ^ 1(a | \\tau ) &= \\pi^1( \\phi\\circ\\phi^{-1}(a) | \\tau ) & \\text{from } id=\\phi\\circ\\phi^{-1} \\\\\n&= \\pi^1(K_{\\phi}\\circ\\phi^{-1}(a) |\\tau) & \\text{by identification}\\\\\n&= \\pi^1( \\phi^{-1}(a) | L_{\\phi}(\\tau) ) & \\text{from Eq.2}\\\\\n&= \\pi^1(\\phi^{-1}(a) | \\phi(\\tau) ) & \\text{by identification}\\\\\n&= \\pi^2(a|\\tau) & \\text{from }\\pi^1=\\phi(\\pi^2)\n\\end{align}\n$$\n\n--> line 200 : Clarity : the denomination \"first approach\" is ambiguisly used, maybe? Isn't G-OP the first approach, while the second approach is to use the *symmetrizer* $S$ on SP-trained $\\psi$ and deploy $S(\\psi)$?\n\n--> line 220-221 : Clarity/Soundness : I am not sure I can agree with the last sentence: I do not think it is a fair description to say that the present method does not require G to be fully known, as the symmetrizer relies on it to be fully known, no?\n\nI agree that, in practice, the paper shows numerical results that a subset of all the environment symmetries ***may*** be sufficient, but the theoretical evidence requires full knowledge of them, if I understand correctly.\nTherefore I would invite the authors to rephrase here.\n\n--> Table 1 : missing result in Table 1, for S5-equivariant agents, maybe? Line 257 seems to refer to it but those results are not in the table.\n\n\n# Significance :\nI think the paper is rightfully placed in the literature and very significant to the field of machine learning as zero-shot/human coordination is a very important problem for the field going forward. \nWhile the theoretical evidence are, in my viewpoint, very significant, I must confess that I am left disappointed with the numerical results not providing, at first glance, a very significant improvement to the state-of-the-art.\nThus, I am hoping that the statistical analysis that I am inviting the authors to perform will provide a compelling statistical significance argument to the numerical results, hopefully.\nIf not, then at least readers will know that there is still legitimately some work to do here.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept