Boosting Sample Efficiency and Generalization in Multi-agent Reinforcement Learning via Equivariance
Authors: Joshua McClellan
Naveed Haghani
John Winder
Furong Huang
Pratap Tokekar
Venue: NeurIPS 2024 poster (2024)
Keywords: Equivariant Graph Neural Networks
Reinforcement Learning
Multi-agent Reinforcement Learning
Symmetry
generalization
sample efficiency
MARL
Primary area: reinforcement_learning
TL;DR: We demonstrate improved sample efficiency and generalization in Multi-Agent Reinforcement Learning (MARL) via using Exploration-enhanced Equivariant Neural Networks instead of traditional function approximators such as MLPs.

Abstract: Multi-Agent Reinforcement Learning (MARL) struggles with sample inefficiency and poor generalization [1]. These challenges are partially due to a lack of structure or inductive bias in the neural networks typically used in learning the policy. One such form of structure that is commonly observed in multi-agent scenarios is symmetry. The field of Geometric Deep Learning has developed Equivariant Graph Neural Networks (EGNN) that are equivariant (or symmetric) to rotations, translations, and reflections of nodes. Incorporating equivariance has been shown to improve learning efficiency and decrease error [ 2 ]. In this paper, we demonstrate that EGNNs improve the sample efficiency and generalization in MARL. However, we also show that a naive application of EGNNs to MARL results in poor early exploration due to a bias in the EGNN structure. To mitigate this bias, we present Exploration-enhanced Equivariant Graph Neural Networks or E2GN2. We compare E2GN2 to other common function approximators using common MARL benchmarks MPE and SMACv2. E2GN2 demonstrates a significant improvement in sample efficiency, greater final reward convergence, and a 2x-5x gain in over standard GNNs in our generalization tests. These results pave the way for more reliable and effective solutions in complex multi-agent systems.

Boosting Sample Efficiency and Generalization in Multi-agent Reinforcement Learning via Equivariance
Joshua McClellan
Naveed Haghani
John Winder
JHU APL
JHU APL
JHU APL
University of Maryland joshmccl @umddddmmmm..mmmmmmmmmm edu
naveed. aaggani@jhuapp edu
john. winder0jhuapl. edu
Furong Huang University of Maryland furongh@umd, edu
Pratap Tokekar
University of Maryland
tokekar@umd edu
Abstract
Multi-Agent Reinforcement Learning (MARL) struggles with sample inefficiency and poor generalization [1. These challenges partially due lack of struc- ture or inductive bias in the neural networks typically used in learning the policy. One such form of structure that commonly observed in multi-agent scenarios is symmetry. The field of Geometric Deep Learning has developed Equivariant Graph Neural Networks (EGNN) that equivariant (or symmetric) to rotations, translations, and reflections of nodes. Incorporating equivariance has been shown improve learning efficiency and decrease error (2). In this paper, we demonstrate that EGNNs improve the sample efficiency and generalization in MARL. How- ever, we also show that naive application of EGNNs to MARL results poor early exploration due to a bias in the EGNN structure. To mitigate this bias, we present Exxplorationeenhanee compare F2GN2 to other common function approximators using common MARL benchmarks MPE and SMACy2. E2GN2 demonstrates a significant improvement in sample efficiency, greater final reward convergence, and a 2x-5x gain in over standard GNNs our generalization tests. These results pave the way for more reliable and effective solutions in complex multi-agent systems.
Equivariant Gruph Neural Networks or E2GN2. We
Introduction
Multi-Agent Reinforcement Learning (MARL) has found success in various applications such
as robotics 3 4 5), complex strategy games T
and power grid management 9 10 How- ever, MARL algorithms can be slow to train, difficult to tune, and have poor generalization guarantees [ [I]. This is partially because typical implementations use neural networks such as Multi-Layer Per- ceptrons (MLP) that do not take the underlying structure into account. The models learn simple input/output relationships with no constraints priors on the policies learned. These generic architectures lack strong inductive bias making them inefficient in terms of the training samples required.
All learnable
learnalla equiverient
functions
All leamable functions constralned
functiorss
want learn
MARL techniques
Figure 1: An example of how using an equivariant func- tion approximator shrinks the total search space.
'The Johns Hopkins University Applied Physics Lab
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

design a GNN equivariant to Symmetries are commonplace in the world. As humans, we exploit symmetries in our daily lives to improve reasoning and learning. is basic concept learned by children. Humans do not need to relearn how to eat an apple simply because it has been moved from the right to the left. In soccer, if you have learned pass to someone on the right, it is easier to learn to pass to someone on the left. Our objective is to develop agents that are guaranteed to adhere to these basic principles, without needing to learn every single scenario from scratch.
Symmetries are particularly common in MARL. These occur in the form of equivariance and invariance. Given a transformation matrix T, if f(Tx) fxx that function is said to be invari- ant. Similarly. f(.) is equivariant if fTxx) = f [2]. Rotational and reflection symme- tries (the ((n) symmetry group) are particularly common in Reinforcement Learning (RL) sce- narios. For example, consider the Multi-agent Particle Environment (MPE) [13 benchmark for MARL which has agents with simple dynamics and tasks. We note that this scenario adheres to rotational equivariance. As shown in Figure rotating the agent's 's positions results in the op- timal actions also being rotated. A policy that is rotationally equivariant effectively shrinks the state-action space for the problem, potentially making the problem easier to learn (Figure D.
Rotate 90"
An example of rotational equivari- ance/symmetry in MPE simple spread environment. Note as the agent (in red) positions are rotated, the opti- mal actions (arrows) are also rotated.
Figure 2:
One way to guarantee equivariance to rotations and reflections in MARL is to use an Equivariant Graph Neural Network [2]. Due to the more complex and equivariant structure of the EGNN, directly adapting EGNNs to MARL is not straightforward. Specifically, consider action spaces typically represented as discrete choices (e.g., up, down, left, shoot). Typically the policy over such an action space is represented using logits (to specify the probability of choosing each action). EGNN outputs a continuous equivariant output. Mapping this output to logits is non- con-trivial. Therefore, we ask the question What is the correct representation of complex action space when using Equivariant GNNs?
Another specific issue we observed in EGNNs an exploration bias in the early stages of learning due to the specific nature of the equivariant computation. This bias can lead to poor initial exploration, decreasing the probability of the agent finding the optimal policy, and potentially resulting in sub- optimal convergence. Therefore we ask the question How rotations, but without early exploration bias?
The main contribution of this paper is Exploration-enhanced Equivariant Graph Neural Networks (E2GN2). This addresses the two research questions. Specifically, we show how to apply Equivariant GNNs to complex MARL tasks. to mixed discrete/continuous action spaces, and to mitigate the early exploration bias. Our contributions can be summarized as:
(1) Our approach the first to successfully demonstrate Equivariant GNNs for MARL on standard MARL benchmarks with complex action spaces.
(2) We propose E2GN2 which has no bias early exploration for MARL and is equivariant to rotations and reflections.
(3) We evaluate E2GN2 common MARL benchmarks: MPE SS and Starcraft Multi-agent Challenge v2 [14) using PPO. It learns quicker, outperforming standard GNNs and MLPs by up to 2x-5x in sample efficiency on terran and protoss challenges. It is worth noting that E2GN2 an improvement on the function approximation for MARL and thus is compatible with most MARL actor-critic methods.
(4) We showcase E2GN2's ' ability to generalize to scenarios it wasn't 'tttt trained on, due to the equivari- ance guarantees. This results in 5x performance over GNNs
For terran, E2GN2 reaches 0.4 win rate after about 1.25 x 10t timesteps, GNN reaches -0.4 after 5 x 10 timesteps and MLP maxes out at -0.3 win rate after 10 x 10 timesteps. For Protoss, E2GN2 reaches 0.4 win rate after 2.5 x 10 timesteps, and GNN and MLP do not reach 0.4 within 10 x 10 timesteps. See figure

---

2 Related Works
A key theoretical foundation for our paper is [15], which formulated the structure for equivariant MDPs. One important takeaway is that if a reward equivariant to a transformationkyymmery we want the policy and dynamics be equivariant that symmetry. However, this work was limited to very simple dynamics with discrete actions such as cart-pole. [16] followed up with equivariance in multi-agent, but was again limited to simple problems. Their method is specifically formulated and tested on small discrete grid world problems with simple dynamics and discrete up, down,left actions. For example, the trafic control problem has input of 7x7 grid. Extending their work to continuous environments with large state spaces, large mixed discrete/continuous action spaces is not straightforward without significant modifications.
Contemporary to our research, T7 demonstrated B(3) equivariant networks on simple cooperative navigation problems. However, their results more complex tasks, such as Starcraft, did not excel over the baseline. Additionally, they used SEGNN [8] which can result in very slow training times, making tuning difficult and cumbersome. Others took the approach [19 of attempting learn symmetries via an added loss term. However, since this approach needed to learn the symmetries in parallel with learning did not have the same guarantees Equivariant GNNs, and did not result in significant performance gains. Another work [20] demonstrated rotation equivariance for complex robotic manipulation tasks. This work, while promising, was for single-agent RL and used image observations and many problems don't 't have access to image observations.
AI research in chemistry has taken a particular interest in adding symmetry constraints GNNs. Works such as EGNN, [2) SEGNN, [88 E3NN, [21 and Equivariant transformers have demonstrated various approaches to encoding symmetries in GNNs. In this paper, we chose to focus EGNN due to its simplicity, high performance, and quick inference time. Other works took a different approach, such as [22], which proposes Equivariant MLPs, solving a constrained optimization problem to encode a variety of symmetries. Unfortunately, in our experience, the inference time was slow, and we preferred network with graph structure such as EGNN.
3 Background
3.1 MARL
Multi-agent reinforcement learning (MARL) considers problems where multiple learning agents interact in a shared environment. The goal is for the agents to learn policies that maximize long-term reward through these interactions. Typically, MARL problems are formalized as Markov games [23] A Markov game for N agents is defined by a set of states S, a set of actions AI,...,AA for each agent, transition probabilities P S x Ai x... x AN x S + [0, 1J. and reward functions RN. mapping each state and joint action to a scalar reward.
The goal of each agent i is to learn a policy (a (s) that maximizes its expected return: ((T)) =
ET .... TN Σ 72 (st, .... aN
Where T is the time horizon, 7 (0, 1] is a discount factor, and of J. (-1st) The presence of multiple learning agents makes this more complex than single-agent RL due to issues such as non-saationarity and multi-agent credit assignment.
3.2 Equivariance
Crucial to equivariance is group theory. A group is an abstract algebraic object describing a symmetry. For example, O(3) describes the set of continuous rotation symmetries. A group action is an element of that particular group. To describe how groups o perate on data we use representations of group actions. A representation can be described as mapping from group element to a matrix, p: G + GL(m) where P(g) E m.mxm Or instead we can more simply use: L, : X + X where g E G where Lg is the matrix representation of the group element g G 24
function is equivariant to a particular group or symmetry if transforming the input is equivalent to transforming the function output. More formally, (ar( = [(Lgi)) for g E G. L.g : X + X and T, : Y -> Y. Related to equivariance is the key concept of invariance, that is a function does not change with transformation to the input: f(x) = f fLgzz). [24]
3

---

Previous work [16] has shown that if Markov game has symmetries in the dynamics and the reward function then the resulting optimal policy will be equivariant and the value function will be invariant. That is, (Tgg)) (s) and πt (I.g8) K (s) where Tig and Kg. with E G are transformations to the state and action respectively.
3.3 Equivariant Graph Neural Network
Equivariant Graph Neural Networks 2 (EGNN) are an extension of a standard Graph Neural Network. EGNNs are equivariant to the E(n) group, that is, rotations, translations, and reflections in Euclidean space. EGNNs have two vectors of embeddings for each graph node i: the feature embeddings denoted by h.. and coordinate embeddings denoted by u/, where denotes the neural network layer. The equations describing the forward pass for single layer are below:
maj &e (h's 1 (ul, w/)) 113)
u 1+11 ut + C Σ (ut u!)) du (mis)
(2)
j#
m: Σ mijs
h !+1 on (h': m.)
(3)
57
Here o represents multi-layee perceptron, where de R R.. 6u R R. and
Rm+p + RR. The key difference between EGNN and GNN is the addition of coordinate embeddings u, and equation 2 which serves to update the coordinate embeddings in manner that is equivariant to transformations from E(n). Note that u, will be equivariant and h, will be invariant to these transformations (2)
As noted Section application of EGNN MARL not straightforward. the following section, we discuss these issues in more depth and present our solution towards addressing them.
4 Methods
In this section, we address both theoretically and empirically how the output of EGNN is biased by the input, leading to suboptimal exploration in RL. To mitigate this issue, we introduce Exploration- enhanced Equivariant Graph Neural Networks (E2GN2) as a method that ameliorates this bias, leading to improved exploration.
4.1 Biased Exploration in EGNN Policies
An important component reinforcement learning is exploration. Practitioners often use a policy parameterized by Gaussian distribution, where the mean is determined by the policy network output, and the standard deviation is a separately learned parameter. Best practices are that the actions initially have zero mean distribution to get good sampling of potential aate-nction trajectories, i.c., (a, ())s) ~ N (0, or. Below we show that an EGNN will initially have a non-zero mean distribution, which can cause problems in early training.
Theorem Given layer of EGNN with randomly initialized weights, with the equivari- ant component input vector ut E R", equivariant output vector ui+ E R" and the multi- layer perceptron du I. + R. where the equivariant component is updated as: u, 1+1 ui CE (u) -u!) Qu (maz). Then the expected value of the output vector is approximately the expected value of the input vector:
E
Furthermore, given a full EGNN with L layers then the expected value of the network output is approximately the expected value of the network input
E
4 E [
(See appendix A for proof.)

---

Corollary 1.1 Given a por agent represented by an EGNN and parameterized with a Gaussian distribution, and the equivariant component of the state se the policy will have the following distribution:
Te(a:ls) N(
(See Appendix A for proof.)
In many cases 8 is the agent's position. Corollary 1.1 indicates that an agent's action distribution will be skewed towards its own position. If the magnitude of the state representation 8 is significantly larger than o, the agent may output actions approximately equal to its position. We refer to this phenomenon, where agent actions are biased towards replicating the input state as the action, as the early exploration bias. Such bias not conducive to effective exploration, potentially limiting the agent's s ability to discover and converge to optimal solutions.
Examples of early exploration bias Suppose there is an agent placed at the position u, [2, 0). and its objective is to reach the origin (i.e. 0). The agent needs an action of less than 0 to move to the left. Since its action using the EGNN is close Gaussian distributed with a mean of then P(X < 0) = 0.02. If the agent moves 0.25 with each action, then the probability of the agent reaching the center objective in eight steps is 2.9e-8. On the other hand, the agent has a high likelihood of moving toward infinity as each move away increases the likelihood of moving towards infinity. This example shows that the bias leads to a low probability of finding the solution, which is especially harmful in sparse reward environments.
To confirm our hypothesis on EGNN early ex- ploration bias, we conducted a simple experi- ment. We trained a Proximal Policy Optimiza- tion (PPO) [3] agent on the MPE simple-spread problem. This problem consists of three agents that seek to navigate to three goals cooperatively; the agents are rewarded based on the sum of the minimum distance from each target to an agent. We give further details of this set up and training procedure in the experiments section. Figure shows the results. The EGNN policy biased the agents to move in the direction of their cur- rent position, causing them to move away from the origin and receive low rewards. Due to the dense rewards and simplicity of the problem, the EGNN agent was able to overcome this initial bias and still solve the problem. However, in more complex problems this early bias could cause more problems. Even if an agent finds an optimal trajectory, it will be relatively small proportion of the sampled trajectories and may result in sub-optimal convergence.
Nevart haiing Crees
Bebavior in carly lsaming
-923
-22
1911
-35
2030
80400
Figure 3: An example of biased learning in MPE simple spread environment. Left: We observed the behavior of the EGNN agents in this early training phase. Fach agent moved away from the origin due the EGNN bias. Right: Note the very low reward in early training steps due to the biased policies moving away from the goals.
4.2 Exploration-Ennaane EGNNS
As discussed previously, EGNN's severe early exploration bias can decrease learning performance. In this section, we propose our solution to this problem in the form of Exploration-enhancee Equivariant Graph Neural Networks (E2GN2). To create E2GN2 we make the following modification to Equation 2 of the equivariant component of EGNN:
u
[+| (m)
C Σ (ut >.
u²)) du (m:))
371
where du, (m:) : RT + R is an MLP parameterized by U2 The remaining layer update equations of the EGNN remain the same.
Theorem Given a L. layer E2GN2 with randomly initialized weights, where the equivariant component is updated as: u, = (m)))(m;)) + C Exzs (u': ut) du (m.))). Then the expected value of the output vector is approximately the expected value of the imput vector:
E [[ttt]]
5

---

Corollary 2.1 Given a policy for agent represented by an E2GN2 and parameterized with Gaussian distribution the policy will have the following distribution:
Ti (a(a)))) (0,00)
Thus we see that E2GN2 should have unbiased early exploration for the equivariant component of the actions. The primary difference between EGNN and E2GN2 the addition of dug- which serves offset the bias from the previous layer (or input) u.. To validate that this did indeed solve the exploration bias we tested E2GN2 on the same MPE simple spread environment. We observe in Figure 3 that E2GN2 did indeed have unbiased behavior when it came to early exploration, the agents had smooth random actions, and the reward did not drastically decrease.
Analysis of E2GN2 Equivariance Here we verify that E2GN2 still retains EGNN's guarantee of equivariance to rotations and reflections.
Theorem E2GN2 is equivariant to transformations from the O(n) group. In other words, is equivariant rotations and reflections.
(See appendix A for proof.)
Retaining our symmetries to rotations and reflections is important, since it should increase our sample efficiency and generalization. Note that E2GN2 is no longer equivariant to tramslations. However, in MARL translation equivariance not necessarily a benefit. For example, consider the MPE problem with translational equivariant policy. If we shift an agent to be 10 units right, this will add 10 to the action as well, causing it to move to the right! Essentially, this adding an undesirable bias to the policy output: ((s + b) = x (s) + b However, we can expect ((n) policy equivariance to improve our sample efficiency in MARL, and is key that E2GN2 retains this guarantee.
4.3 Adapting Architectures for Complex Action Spaces
Applying EGNN/E2GN2 to sophisticated MARL problems requires careful consideration. Many MARL environments have discrete action spaces or mixed comiinooss-dicreet action spaces. Some components of these action spaces may be invariant and others may be equivariant. Further com- plicating the problem, MARL requires the neural networks to output parameters for probability distribution (which is sampled for use in exploration). For continuous actions, agents typically use gaussian and for discrete actions agents generally use logits. Unfortunately, it is not straightforward to map the distinct invariant and equivariant coordinate embeddings onto single distribution in manner that preserves equivariance.
core benefit of the GNN structure in MARL is the scalability afforded by permutation equivariance: the ability handle variable number of agents without retraining. For example, (25) demonstrates using GNN to train N agents, then to control N + agents without retraining. However, this example did not operate with discrete mixed action spaces or equivariant structures as we do here. Improperly mapping GNN outputs to logits risks losing this scalability.
To address these issues, we propose leveraging the GNN's 's graph structure output different compo- nents of the action space from different nodes in an equivariant manner:
Discrete Actions & Invariant Components: the invariant feature embeddings h; of each agent's 's node are used to output discrete logits/actions and other invariant components. For action spaces with both equivariant and invariant components, this can be used as the 'action type selector' to select which type of action to apply that type step (ie move action targeting action).
the equivariant coordinate embeddings u, of each agent's node are used for continuous spatial actions such movement.
Continuous Spatial Actions:
Targeting Specific Entities: for multi-agent environments with entities beyond the learning agents (e.g., enemies), logits for discrete actions pertaining to each entity (e.g.. which enemy to target) are output from that entity's 's corresponding node. This enables the discrete action space to scale with the number of agents.

---

Equivarinnt Action
Invoriant
Composent
Equivariant Graph Neural Network
Idnuctured with Invariant and equiaariant componenti)
Pediyy Hend
Lapen
Value (ivveriant)
LB
O
Figure 4: An example using an Equivariant Graph Neural Network in MARL. Note the state must be structured as graph with each node having an equivariant M, and invariant h; component. As discussed in 4.3 the output of the policy uses for equivariant (typically spatial) actions, and the h; for invariant components of the actions
Each of these components is parameterized as a distribution (ie logits or gaussian), resulting in one to three seperate distributions (depending on the environment and which components are used). After the exploration phase samples from each of these distributions, the final action can be constructed from each of these components via concatenation or the action selection component.
This formulation enables us to retain the permutation equivariance and scalability of a GNN. An alternative to approach is to add MLP the end of the GNN to convert the GNN outputs to the mixed action spaces. This will lose the scalabilityloocaiity of the neural network. For example, if you add two more agents how do you modify the final MLP expand the action space? In formulation, whenever an new entity i is added to the environment, we can simply add the node N 1 to the graph. The action space the GNN will now be supplemented with uN+1 and hN+1- allowing us to expand the action space without retraining.
By structuring the GNN output in this manner, we handle discrete mixed discrete-continuoe action spaces while retaining the equivariance of EGNN/E2GN2 and the flexibility of GNNs to a variable number of agents and entities. This approach allows MARL agents based equivariant GNNs to be applied to challenging environments with minimal restrictions on the action space complexity. Indeed this approach key enabling the success of EGNN/E2GN2 on SMACv2.
Experiments
We seek to answer the following questions with our experiments: (1) Do rotationally equivariant policy networks and rotationally invariant value networks improve training sample efficiency? (2) Does E2GN2 improve learning and performance over EGNN? (3) Does Equivariance indeed demonstrate improved generalization performance?
To address these questions, we use common MARL benchmarks: the multi-agent particle-world environment (MPE) [13] and Starcraft Multi-agent Challenge version 2 (SMACv2) [14] Our experiments show that equivariance does indeed lead to improved sample efficiency, with E2GN2 in particular performing especially well. We also demonstrate that generalization is guaranteed across rotational transformations.
We want to focus our experiments on the neural networks' effects on MARL performance. To isolate the impact of the network architecture, we avoid using specialized tips and tricks sometimes employed in MARL (26). This allows us to demonstrate that proposed networks can improve performance
7

---

without relying on these additional techniques. Thus we use a common standardized open source MARL training library RLlib [27] We use the default multi-agent PPO algorithm, which does not use centralized critic. We followed the basic hyperparameter tuning guidelines set forth in [26]. That is, we use large training batch and mini-batch size, low numbers of SGD iterations, and a small clip rate. Further hyperparameter details are found in appendix B
We compare our results with common neural networks used for MARL benchmarks: multi-layer perceptrons (MLP), and GNNs (we use GNN structure similar to equations 3 see appendix B
for details) ). We also compare with the approach from 17 which we refer to as E3AC in our plots. Recall that E3AC also uses neural networks that guarantee equivariance: SEGNN. We integrated E3AC into RLLIB to ensure the RL training procedure remained consistent across our comparisons. Note that the majority of MARL papers on SMAC/SMACv2 use MLPs as the base network. We also compare with GNNs to show the improvement is not primarily due to the graph structure. For the main paper results, we assume full observability since we have not explicitly tackled the partial observability problem yet. In future work, we will seek to remedy this. However, to be thorough we performed experiments with partial observability, resulting in surprising success using E2GN2 (see appendix
5.1 Training Environments
From the MPE environment ( (13), ). two environments to benchmark our performance: co- operative navigation also known as spread, and predator-prey, also known as tag. The MPE tag environment consists of three RL-controlled agents sceking to catch third evader agent. For easier comparison, we use a simple heuristic algorithm to control the evader agent. The evader simply moves in the opposite direction of the pursuers. The other MPE environment is the cooperative navigation or simple spread. In this environment, each agent seeks to minimize the distance between all obstacles. To better test equivariance, for the MPE environments we use continuous actions instead of the default discrete actions.
Next we test on SMACv2, a much more difficult environment than MPE. In SMACv2, the units are heterogenous with different capabilities (with different attack ranges, total health, and sometimes action spaces). The unit types are randomized at the beginning of the scenario. The actions include more components than simply movement (such as in MPE), agents can move and attack. The goals more complex well. Instead of simply navigating cooperatively as in MPE, the agents must learn attack formations and strategies. Sometimes it may be optimal to sacrifice health or allies in the purpose of the greater strategic objective. SMACv2 has three basic scenarios defined by the unit types: terran, protoss, and zerg. We use agents for each team, and the initial position configuration termed "surrounded" (see fig]]
The SMACv2 action space poses an interesting problem for GNN struc- tures. A key advantage of GNNs is the permutation equivariance, which leads to scalability without retraining. The default SMACv2 agents will out- put simply a discrete action determin- ing movement target for shooting. For our purposes, we modify the ac- tion space be a mixed action space. This consists of a continuous vector for movement actions, a discrete ac- tion determining the attack target, and a boolean determining if the agent should shoot, move or no-op (to be complete, we include plots with a dis- crete action space in appendix C As discussed in section 4.3 this will both prove a greater test for our algorithms and allow for the GNN to scale to larger numbers of agents.
-00
Hount
DOLS
Nauna lit uitte Dund
-20
Ped unit
2.0
env erice
-75
190 punn W4-0
124
TTme Doars
13
Figure 5: Comparing PPO learning performance on MPE with various Neural Networks. (TOP) reward as a function of environ- ment steps. We show the standard errors computed across 10 seeds. (BOTTOM) reward as function of wall clock time
8

---

terran
protoss
zerg
0.6
Win Rate
PPO w/ E2GN2 (Ours)
0.4
PPO EGNN (Ours)
PPO w/ GNN
0.2
PPO E3AC
PPO MLP
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00 le7
of steps
1e7
of steps
le7
of env steps
Figure 6: Comparing performance of PPO on SMACv2 with various Neural Networks representing the policy and value function. Each chart represents different race from the SMACv2 environment. We show the standard errors computed across seeds.
5.2 Training Results
We present the results from our training in this section. The results for MPE in Figure are averaged across 10 seeds. As discussed previously, EGNN has poor early performance due to the early exploration bias. Despite this poor exploration, EGNN outperforms GNNs and MLPs, demonstrating the power of equivariance in MARL. Similarly, we note that in MPE tag. E2GN2 results in strong final agent, while EGNN suffers in the initial training phases. We see that E2GN2 is able to greatly outperform EGNN's final reward, partly due superior early exploration.
In figure we also compare the wall clock time required train each algorithm. Note that these were all trained on the same hardware and using the same training algorithm (RLLIBs PPO). E3AC remains competitive sample efficiency on MPE spread, but it requires nearly four hours to gather one million time steps, compared to less than one hour for E2GN2. This is likely due to the SEGNN [18] underlying structure used by E3AC, which is slower due to the be slower for inference time. We also note that E2GN2 does manage to outperform E3AC MPE Tag.
Next, we review the results from SMACV2 in Figure 6 these results are averaged across 10 seeds. The equivariant networks have a clear improvmement in sample efficiency over the MLP and GNN. On the terran environment, EGNN once again learns slower the initial phases, but due to equivariance, it outperforms MLP/GNN. For protoss, we note that EGNN performs well but has high variance for its performance. E3AC struggles to perform well on SMACv2, likely since it was unable to resolve the hurdle we addressed in section 4.3 Instead for SMACv2 E3AC used simple MLP for the policy
and SEGNN for the value function 17 The training speed and performance for E2GN2 is especially impressive since this environment is much more complex than the MPF.
5.3
Generalization
Sur Gardudded
Surrounded Left
Sur Surrounded Hight
Thus far we have demonstrated that equivariance does indeed lead to improved sample efficiency. The next question to answer is whether equiv- ariance leads to improved generalization. We test this by using the 'surrounded' initial config- uration from SMACv2 shown in figure / By default, the agents are randomly generated in all directions (along specific axes). To test general- ization, we only initialize agents on the left side of the map (i.c., surrounded left). We then test to see if the agents are able to generalize when they tested with initial positions starting the right (called surrounded right), and with the default surrounded configuration (termed Surrounded All or Surrounded). Theoretically, the equivariant network should see equivalent performance between the training and testing initialization, due to the guarantee of rotational equivariance for movement, and invariance for shooting actions.
Figure 7: SMACv2 initialization schemes used for test- ing generalization
The results of our tests are shown in Table I As expected E2GN2 has stellar generalization performance. The win rate remains the same from the training configuration (Surrounded Left) to Surrounded Right. We see an increase in win rate when testing in Surrounded All. This is likely because this Surrounded All an easier scenarion; the enemy is divided into more groups, so the agents can defeat the smaller groups one time. As mentioned in the abstract, this table

---

Table I: Generalization Win Rate in SMACv2. Note that E2GN2 retains high performance, while GNN and
lose performance when generalizing
Training Initialization
Testing Initialization
Environment
Network
Surrounded Left
Surrounded Right
Surrounded All
E2GN2
0.57 +.01
0.55 #.01
0.63 +.01
Terran
GNN
0.51 +.02
0.11 +.02
0.32 :.02
MLP
0.33 +.02
0.12 +.02
0.24 =
E2GN2
0.59 4.01
0.56 +.02
0.57 +.02
Protoss
GNN
0.5 +.02
0.14 +.01
0.32 +.01
MLP
0.42 #.02
0.17 +.02
0.27 =.02
E2GN2
0.35 1.02
0.32 1.02
0.29 1.02
Zerg
GNN
0.4 1.02
0.07 1.01
0.21 1.01
MLP
0.21 +.02
0.05 +.01
0.12 #.01
indicates that for the Surrounded Right generalization test E2GN2 has -5x the performance GNN and MLP.
Next, in Table we test E2GN2's s ability to scale to various numbers of agents without retraining. We use the agents trained in SMACv2 from Figure for for these tests. We do not include the MLP agent since cannot scale up without retraining. Note that the method described section 4.3 was essential to enable scaling between agents. For example, if we passed the GNN output to a final MLP to map to the discrete action space this wouldn't be scalable more agents.
The results for scalability confirm that both F2GN2 and GNN are able to maintain performance with larger numbers of agents. As the number of agents gradually increases the win rate does slowly decrease. E2GN2 does seem to have a slightly steeper loss in performance in the Zerg domain, which was also the most difficult domain for the initial training phase. We believe this is due the higher complexity of the Zerg scenario, with certain unit types (banelings) that can explode to damage many other agents.
Conclusion and Future Work
In this paper, have demonstrated that E2GN2 merits strong consideration for many MARL applications. We addressed several important problems including the early exploration bias and how to apply it to complex action spaces. The sample-efficincy has dramatically improved, and there are now guarantees of generalization built into the network. There still further work to do, such handling the partial or incomplete symmetries. Furthermore this approach would need to be improved further to be applied to environments with angular momentum and accelcrations, which are not included in these benchmarks. Currently, we expect that the improvement gained from E2GN2 in MARL will depend on the amount of rotational symmetry applicable in the observations. We believe that building this work be helpful deploying MARL agents with greater degree of trust (due to the guarantees). This could be helpful in many fields such as robotics, medicine, and power systems, In summation, the results of this paper provide solid foundation upon which build.
Table Generalization Win Rate: Testing RL agents ability scale different numbers of agents (originally trained with agents)
Training Setup
Testing Setup
Environment
Network
5 Agents
4 Agents
6 Agents
7 Agents
Agents
B2GN2
0.69 +.02
0.65 +.02
0.63 +.02
0.62 +.02
0.54 #.04
Terran
GNN
0.45 +.01
0.44 +.01
0.42 +.02
0.39 +.01
0.32 +.02
E2GN2
0.62 +.03
0.61 +.02
0.59
+.03
0.47 +.04
0.37 4.03
Protoss
GNN
0.39 +.02
0.37 +.01
0.36
4.03
0.30 +.02
0.21 #.02
E2GN2
0.36 +.03
0.32 +.03
0.31 #.01
0.23 #.01
0.18 #.03
Zerg
GNN
0.28 #.04
0.28 #.02
0.25 +.03
0.2 #.02
0.15 #.02
10

---

References
[1] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-Agent Reinforcement Learning: A Se- lective Overview of Theories and Algorithms, pages 321-384, Springer International Publishing. Cham, 2021.
[2] Victor Garcia Satorras, Emiel Hoogcboom, and Max Welling, E(n) equivariant graph neural networks, 2021.
[3] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/ 0ass/1007.0630 00347,, 2017.
[4] Shai Shaaar-Shwarz,, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. ArXiv, abs/1610.03295,, 2016.
[5] Pinxin Long, Tingxiang Fan, Xinyi Liao, Wenxi Liu, Hao Zhang, and Jia Pan. Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA). pages 6252-6259, 2018.
[6] O. Vinyals, I. Babuschkin, and W.M. et al Czarnecki. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575, 2019.
[7] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140-111 2018
[8] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, L. Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588:604 609, 2019.
[9] Esmat Samadi, Ali Badri, and Reza Ebrahimpour. Decentralized multi-agent based energy management of microgrid using reinforcement learning. International Journal of Electrical Power I& Energy Systems, 102:106,,,, 2020.
[10] Ziming Yan and Yan Xu. A multi-agent deep reinforcement learning method for cooperative load frequency control of multi-area power system. IEEE Transactions on Power Systems, 35(6):459944608 2020.
[11] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1282-1289. PMLR, 09-15 Jun 2019.
[12] Andrew M. Pitts. Nominal Sets: Names and Symmetry in Computer Science. Cambridge Tracts in Theoretical Computer Science. Cambridge University Press, 2013.
[13] Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[14] Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning, 2023.
[15] Elise van der Pol, Daniel E. Worrall, Herke van Hoof, Frans A. Oliehock, and Max Welling. Mdp homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Processing Systems, 2020-Decemher, 6 2020.
[16] Elise Van Der Pol. Herke Van Hoof, Uva-Bosch Deltalab, Frans A Oliehoek, and Max Welling. Multi-agent mdp homomorphic networks, 10 2021.

---

[17] Dingyang Chen and Qi Zhang. e(3) -qquiiarrri actor-critic methods for cooperative multi-agent reinforcement learning, 2024.
[18] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J. Bekkers, and Max Welling. Geometric and physical quantities improve E(3) equivariant message passing. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,
2022. OpenReview.net, 2022.
[19] Xin Yu, Rongye Shi, Pu Feng, Yongkai Tian, Simin Li, Shuhao Liao, and Wenjun Wu. Lever- aging partial symmetry for multi-agent reinforcement learning. Proceedings of the AAAI Conference Artificial Intelligence, ((16)):775331757))) Mar. 2024.
[20] Dian Wang, Robin Walters, and Robert Platt. (500 reinforcement learning, 2022.
[21] Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks, 7 2022.
[22] Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing cquivariant multilayer perceptrons for arbitrary matrix groups, 2021.
[23] Michael L. Littman. Markov games as framework for multi-agent reinforcement learning.
Morgan Kaufmann, San Francisco (CA), 1994.
[24] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021.
[25] Akshat Agarwal, Sumit Kumar, and Katia P. Sycara. Learning transferable cooperative behavior multi-agent teams. CoRR, abs/1906 01 202, 2019.
[26]
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Wu. The surprising effectiveness of ppo in cooperative, multi-agent games, 2022.
[27] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In International Conference Machine Learning (ICML), 2018.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026-1034, 2015.
[29] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge, 2019.
12

---

Appendix and Supplemental Material
A Proofs
A.1 EGNN Bias Proof
Here prove assertion that the EGNN layers have biased outputs, and thus poor exploration. We do this by computing the expected value of a single
We start by showing that B (mu,)))) neural network layer.
)] =s
An individual neural network layer is defined as y Wæ + b where b is N-by-1 vector, W is N-by-M matrix and æ is M-by-1 vector. Note that similar to (28) we initialize the elements in W to be mutually independent and share the same distribution. æ is initialized similarly, and and W are independent. Thus for each element (subscript ) of the equation, we have:
[xr]
Σ E [wrar]] [br] ny [tran] + EE [bi] n2E [i]] [21]
E [br]
We can expect wy and by to be zero mean, since many commonly used initializers use either uniform, normal distribution, with zero mean 28 This results in:
74 E [2y] + 0
Thus, we can assume that for an individual layer the output is 0, since the final layer of any mlp is O, can see that E (m:,))
Next we show that E [n]]] Rs E [ua]] Recall that uf is the Lth layer of ith node of the equivariant component of the network. We take the expectation over a sampling of the inputs u, and hú, treating each as random variable. Note that u; and uj will have identical distributions, since each node will be sampled in the same manner. We start our proof by taking the expected value of the output:
++1 E
u
+
Σ (u) -u/) or (mis) jt
=E [u]]
Σ j
[(u u/)) de (m.;))
=
[E +
pu (m.;))
[u (m.;)]
jitt
E [4 + C
[22]
TE (. (m.;)) - [2]]] E so (m:j))
E [ + C Σ 400 -u, = ut
~
E
[ut+1] ~ E [u]]
Thus,
E [uf]
E
[
Rs EE [u]]
Where used the assumption that E. (u (m:y))
E
[n]]] E (dh (m:;))
The corollary: Ti (a.(s) N(,"" .) is a simple result from the fact that the equivariant action for an agent is the output of the coordinate embedding of the EGNN: uf. Here we assume the standard deviation a separately trained parameter and not function of the EGNN. this is the case, then the policy mean E' (a, s)] = E [uf]] ~ E [
E [s]"]] (by definition)
13

---

A.2 Proof E2GN2 is Unbiased
Here we show that E2GN2 leads to unbiased early exploration. We begin with the equation for the E2GN2 layer:
ntt1 (n/ow2(mm)) + C
Σ
(u: - -uy) on (mij))
Next we take the expected value of the output:
E [1] E. [u, ( (ne(m))))
+ C Σ B [uto (mis)) - E [uu (mi;)
jfi
E [uutt] (mw2)
+ Σ
E
[u]]]
IE D (m:;)] E [(u]]] fo (m:))
j&1
E
E. [
E
[ut]]] 0= -0
jzi
Thus we see that for each layer E [u]tt] R O and therefore [u.'] =s 0. Similar to before, we used the assumption that Where we used the assumption that E [2] ou (m:y)]
= E (e)) E
[ (m.y)) and
E [u) (m) E
[u]]] E (m.;))
Similar to the first corollary the result: T=(a,|s) N(.. o)) follows mostly from the definition and the above proof. The corollary aa 8) N (0, )) is simple result from the fact that the equivariant action for an agent is the output of the coordinate embedding of the EGNN: uf Here we assume the standard deviation a separately trained parameter and not a function of the EGNN. If this is the ~ O (by definition) case, then the policy mean E' (T s)] E [
A.3 Equivariance and Invariance of E2GN2
We follow the structure of 2 for how show equivariance. Note that (2 showed that di (m.y) will be invariant to F(n) transformations. This should still hold in our case as we make no modifications to maj. similarly Φu2 (men) will be invariant to translations. To show equivariance rotations and reflections we show that applying a transformation T to the input, results in tranformation to the output. That is: f(Tu)) Tu, where f.) is the update equation of E2GN2:
Tufdwe(m)) C.)
Σ (Tu! Tu²) du (m.j)
= =Tu(oha(mm)) TC Σ (ul. - -n²) sw (miz)
jti
-Tulon2(m)) +TC Σ
(ut- -u²) ou. (mis)
jfi
= (l) + +C Σ (ut - u/) ou (mis))
= Tuu
Since there are no modifications to E2GN2 update equation for h;, then E2GN2 should retain invariance transformations from F(n). More precisely (using the fact that the initial mi is invariant to translations from ( if we translate the input u. by
(u)) + b) (m,) C. >e (lu) + b) -uu, + b)) Φu (miy)
j#i
bova (m:)) u/oua(m.) C (u)
E -() &n (miy) bb))(m;) + utt + b u
This shows that E2GN2 is not translation equivariant.
14

---

separate networks for the B Additional Training Details
Hyperparameters
value
train batch size
8000
mini-batch size
2000
PPO clip
0.1
learning rate
25e-5
num SGD iterations
15
gamma
0.99
lambda
0.95
Table PPO Hyperparameters for SMACv2
Hyperparameters
value
train batch size
2000
mini-batch size
1000
PPO clip
0.2
learning rate
30c-5
num SGD iterations
10
gamma
0.99
lambda
0.95
Table 4: hyperparameters for MPE
MLP uses two hidden layers of 64 width each for MPE and 128 for SMACv2. All MLPs in the GNNs use 2 layers with width of 32. For both MPE and GNN structures policy and value functions.
Graph Structure The graph structure for MPE environments is set as a complete graph. The graph structure for SMACv2 is complete among the agent controlled units (or all of the friendly nodes), and each friendly node alse has an edge to each of the enemy controlled units. This was an attempt to model what we imagine a real world scenario to look like (agents see the enemies, communicate with allies, but do not see the enemies communication signals)
Graph Inputs For MPE environments the input invariant feature for each node hę is the id (pursuer, evader, or landmark). For SMACv2, h. is made of the features: health, shield unit type, and team. For each u as the position of node i. For MPE there is also velocity feature, which we incoporate following the procedure described in (2)
Graph Outputs for Value function Similar to the visual in section 4.3, the value function output comes from the invariant component of the agent's 's node of final layer of the EGNN/E2GN2. In other words the value function is: h/ where is L the final layer, and is the agent in making the decision.
Graph Outputs for Policy For MPE environments the actions are forces particular direction, thus the action output for agent i is uf where I. is the final layer.
Section 4.3 discussed applying EGNN to more sophisticated action spaces. Specifically for SMACv2 the discrete action space has an equivariant component (movements) and an invariant component (shoot). The outputs for the equivariant component of EGNN/E2GN2 are continuous values, mapping these values to logits is not straightforward. Furthermore, they would need to be mapped onto the same distribution logits being represented by the shoot commands. We solve this problem by having three distributions output by the GNN structure: the continuous/equivariii gaussian distribution, the discrete/invariant distribution, and third distribution that determines whether we should move shoot. After the RI. sampling from each distribution is performed for the actions, then the three components the action can be converted to the final action for either mixed discrete-continuogg action space discrete action space.
movement
15

---

Graph Neural Network details We used the same GNN as the EGNN paper: Specifically, we use equations and 3, but update equation be my = @hh, R,. 22, For convenience we rewrite the GNN equations describing a single layer here:
mij de (k/ h1)) m. Σ
mij:
h 2x (h) m.)
jt
Recall that 6% and Oe are two layer MLPs. Note that the GNN only has one variable h; per node (compared the two variables of h, and u, of EGNN/E2GN2), The features inputs are similar to components concatenated together. The output for GNN will all come from the variable h as there is no need to distinguish between equivariant and invariant outputs (the action may be composed across several nodes, see above and figure
an EGNN but with the two invariant/equivar
Hardware For training hardware, we trained the graph-structured networks using various GPUs. Many were trained on an A 1000, but that is certainly not necessary, they didn't need that much space. MLPs were trained on CPUs. We typically used 4 rollout workers across 4 CPU threads, so each training run used 5 CPU threads.
Further Notes/details SMACv2 SMACv2 ((14)) is modification of the original SMAC ([29). Since SMAC had deterministic initialization and static scenarios it turms out policies could memorize the solution. In each scenario, there is pre-specified number of agents each team (we use agents for each team). In each scenario, the agents are randomly initialized in a specific formation. For this work, is important to note that SMACv2 will randomly sample unit types from 3 different units (for each scenario). We the same policy network for each agent but have the unit type an input in the observation space, so each policy must learn to condition its behavior on the unit type.
the initial position configuration termed "surrounded" (see fig 77. Finally, it
C Supplementary Results
Below we include supplementary results we believe interesting to understanding further E2GN2's performance.
terran
protoss
zerg
0.6
Win Rate
PPO w/ E2GN2 (Ours)
0.4
PPO w/ EGNN (Ours)
PPO w/ GNN
0.2
PPO w/ E3AC
PPO MLP
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
# of env steps
1e7
# of env steps
1e7
# of env steps
1e7
Figure 8: These results are using SMACv2 with the 'surrounded" initialization and 5 agents. Specifically, here we map the actions from the mixed discrete-comtimuuus actions back to the default SMACv2 discrete action space. This is done by simply rounding the continuous actions to the nearest axis. These results are using the standard SMACv2 action input.
16

---

terran
protoss
zerg
0.6
Win Rate
0.4
MLP
GNN
EGNN
0.2
E2GN2
0.0
0.00
0.25
1.00 1e7
0.50
0.75
1.00 1e7
0.25
0.50
0.75
1.00
Timesteps
Timesteps
Timesteps
1e7
Figure 9: These results are using SMACv2 with the 'surrounded' initialization and 5 agents. Specifically, the observations here partially observable. Note that E2GN2 still performs well with partial observability.
terran
protoss
zerg
0.6
Win Rate
0.4
MLP
GNN
0.2
EGNN
E2GN2 (Ours)
0.0
0.00
0.75
1.00
0.00
0.25
0.50
0.00
0.75
1.00
Timesteps 1e7
Timesteps 1e7
Timesteps 1e7
Figure 10: These results are using SMACv2 with the 'surrounded' initialization and 5 agents. Specifically, the observations here partially observable. In all other experiments MARL). Here we did perform the recentering. Note how important this step for EGNN. Without the centering observations the EGNN is assuming rotational equivariance around the wrong center. Interestingly, this seems to have little impact on E2GN2's performance.
the states 0 (standard in
terran
protoss
zerg
www. w Made
0.6
Win Rate
0.4
E2GN2
MLP cont act
0.2
MLP discrete act
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Timesteps
1e7
Timesteps
1e7
Timesteps
1e7
Figure 11: These were trained on SMACv2 using the 'surrounded' initialization with 5 agents. We were curious how MLP would perform using the mixed continuous/discrete action space vs all discrete, Note that our E2GN2 is still able outperform both.
17

---

terran
protoss
zerg
Reward
MLP
10
GNN
EGNN
5
E2GN2
0.00
0.25
0.50 Timesteps
0.75
0.25
0.50
0.75
1.00
0.00
0.25
0.50 Timesteps
0.75
1.00
1e7
Timesteps
1e7
1e7
terran
protoss
zerg
0.4
Win Rate
0.3
MLP
0.2
GNN
EGNN
0.1
E2GN2
0.0
0.00
0.25
1.00
0.50 Timesteps
0.75
1.00 1e7
0.25
0.50
0.75
1.00
Timesteps
1e7
Timesteps
1e7
Figure 12: SMACv2 has various initial position configurations. Here we show the results using the 'Surrounded and Reflect" initial configuration.
terran
protoss
zerg
0.8
Win Rate
0.6
0.4
0.2
0.0
GNN
terran
protoss
zerg
EGNN
E2GN2
0.8
Win Rate
0.6
0.4
0.2
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Timesteps
1e7
Timesteps
1e7
Timesteps
1e7
Figure 13: SMACv2. Surrounded and Reflect, agents. Here we dive deeper into the performance when trained individual unit types (instead of a mix of unit types). Top Row: Only Marine, stalker Zergling units Bottom Row: Only Marauder, Zealot or hydralisk unit
18

---

Table Generalization Win Rate: Training 5 agents in Surrounded-Lelf configuration, Testing agents in
all configurations
Training Initialization (but with 7 agents)
Testing Initialization
Environment
Network
Surrounded Left
Surrounded Right Surrounded All
E2GN2
0.362
0.354
0,432
Terran
GNN
0.383
0.095
0.246
E2GN2
0.431
0.418
0.424
Protoss
GNN
0..15
0,206
0.083
E2GN2
0.207
0,156
0.145
Zerg
GNN
0,320
0.047
0.153
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe that our charts and tables reflect the claims made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, we discuss these briefly in the conclusion and results.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
Yes we provide our proofs and rationale in the appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We specifically used an open-source MARL library (RLlib) for our RL training. The code for EGNN is available online. We described any modifications we made to the environments, as well as hyperparameters and training details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Much of the necessary code is available online. Part of this work was developed for private company, and permission for the full code release was not given.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Yes, we mention that we selected hyperparameters following guidelines set by (26). and included the hyperparameters used, the unspecified ones were simply the RLlib default parameters.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The charts all had confidence intervals (95 percent), computed using the standard seaborn package.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe this the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: Yes, believe this work adheres to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in the conclusion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We did not release data or code.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited the code library used. RLlib uses Apache license 2.0
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There were assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There were no human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "9IJ3FsizFj", "reviewer": "Reviewer_N4Ry", "summary": "The paper demonstrates for the first time the successful application of Equivariant Graph Neural Networks (EGNNs) to standard MARL benchmarks with complex action spaces. \nTo address the Early Exploration Bias, the paper provides proof of the biases in standard EGNNs and the unbiased nature of E2GN2. In practice, it is achieved by adding an additional multi-layer perceptron (MLP), which helps to offset the bias from the previous layer's output, leading to actions that are not biased towards the agent's current position.\nE2GN2 provides a method to handle complex and heterogeneous action spaces, which are common in MARL environments. It leverages the GNN's graph structure to output different components of the action space from different nodes in an equivariant manner, allowing for a more flexible and scalable approach to handling mixed discrete/continuous action spaces.\nThe paper shows that E2GN2 outperforms standard GNNs and MLPs by more than 10 times in sample efficiency on common MARL benchmarks like MPE and SMACv2. It also demonstrates greater final reward convergence and up to 5 times gain in generalization over standard GNNs.", "questions": "The paper introduces Exploration-enhanced Equivariant Graph Neural Networks (E2GN2), a novel architecture that significantly boosts sample efficiency and generalization in Multi-Agent Reinforcement Learning (MARL) by leveraging equivariance to symmetries and addressing early exploration biases. However, I still have some concerns:\n1. For the complex action space issue, the detail of the method is not enough. How do you map the embedding to action space, such as an additional MLP layer? For the value head, does it share the same embedding with the policy head? Section 4.3 and Fig. 4 provide the high-level idea but ignore the details.\n2. For the experiment part, this paper seems to use independent ppo as the baseline, which is not widely used in most MARL papers. Although the improvement is significant compared with ppo, it would be better to apply the E2GN2 to other MARL methods. \n3. Another issue is the input of the network. To use the EGNN, it seems that the states of all agents are required. However, we assume that only partial observation can be obtained in most MARL settings. For example, we cannot get the observation of opponents in SAMC. Please provide more details of the implementation.", "limitations": "see questions", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a novel method with interesting insight and sound theoretical analysis.", "weaknesses": "see questions part", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments. \n\n1. There is no MLP on the output of the GNN, EGNN, or E2GN2. Adding an MLP to the output would cause us to lose the guarantee of equivariance as well as the ability to add more agents without retraining. \n\nThe output is similar to what is shown in the diagram. It is composed of outputs from various nodes. The outputs all come from the $L$th layer of the network. For agent $i$, the movement component of the action will output from the own node's equivariant component $u_i^L$. For SMAC, the invariant component of the action (ie the logits for determining which target to attack) will be output from the invariant component of that target's node: $h_j^L$ where $j$ is the potential target/enemy. (see lines 232-239 and 290-293).  I hope that helps clarify things. \n\n(we will add this to Appendix B), we use a separate networks for the policy and value function. The value function output is comes from the invariant component of the agent's node of final layer of the EGNN/E2GN2. In other words the value function is: $h_i^L$ Where $L$ is the final layer, $i$ and is the agent in making the decision. \n\n2. That is a good point to bring up. We believe E2GN2 would improve MAPPO, and we hope to eventually apply this to MAPPO. We use independent PPO because  1) MAPPO can be more difficult for one to one comparisons [1], as it depends on how the centralized critic is shaped. 2) for many cases IPPO is comparable to MAPPO [1] 3) we wanted to use common public libraries for more reliable open source comparisons. we selected RLLIB since it seemed to be a popular benchmark, RLLIB does not have MAPPO built in, it must be added on your own, and implementations can vary. \n\n3. This is an input variable to SMACv2. A user can change the variable 'partially\\_observable' to False to make it fully observable. Note that we did also do a preliminary comparison with partially observable observations (per SMACv2 standard) and saw surprisingly good performance (see appendix)\n\nCitation\n1) The surprising effectiveness of multi-agent PPO", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bE5hDGtD76", "reviewer": "Reviewer_7FjK", "summary": "This paper studies the setting of multi-agent reinforcement learning (MARL). The work tries to tackle the challenges of generalization and sample efficiency using inductive biases. In this case, the work proposes to use equivariant graph neural networks to model the policy and value function of a multi-agent actor critic. The work points out a bias in the output of such networks that harms exploration and proposes an architectural fix. It also proposes an architectural modification that will allow the usage of discrete actions in addition to continuous ones. The approach is evaluated on two different benchmarks and in some cases shows improved sample efficiency and final return compared to an MLP and graph neural network baseline.", "questions": "Q1: Can you elaborate what makes your environments more complex than the ones used by Van Der Pol et al. [19]?  \nQ2: Can you elaborate on why previous work such as that by Van Der Pol et al. [19] does not observe the exploration bias?  \nQ3: How does EGNN in the experiments do action prediction with discrete actions? Does it adapt the suggested solution?", "limitations": "The limitations of the work have not been addressed properly. I believe that this would be quite beneficial since the chosen experiments clearly have a structure that benefits from the imposed inductive bias. This does not mean that any claims that are being made can be true **in general**. It is unclear to me whether these methods still function in environments that are not hand-picked for the inductive bias at hand.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "I would like to preface this review by stating that I am not very familiar with the multi-agent RL literature. My estimation of the novelty of this approach has high uncertainty. However, I am very familiar with the standard RL literature.\n\nProblem statement\n* Understanding what types of inductive biases can lead to improved sample efficiency in reinforcement learning is in general and interesting and important problem given the high cost of sampling in the real world.\n\nClarity\n* The text is well written and easy to follow. \n* The results are presented in a easy to parse manner\n* Multiple little example plots are spread throughout the paper to increase readability.\n\nNovelty\n* From a brief lit review and given that the work builds on a relatively recent neural network architecture and that other works are concurrently trying to integrate this architecture into MARL agents, I'm willing to believe that fixing the exploration bias and integrating multi-modal action prediction are novel contributions.\n\nMethod\n* The proposed method uses a well-reasoned fix for a problem in an existing architecture to solve the exploration bias problem. This makes for an interesting investigation.\n\nRelated work\n* The paper highlights several other works that use equivariance in MARL.", "weaknesses": "Clarity\n* In the abstract the main claims are 10x improvements in sample efficiency and final reward. This is just unnecessarily exaggerated and possibly misleading. I think the paper would benefit from a clear depiction of what is actually shown.\n* The mathematical depiction of the model is unclear and needs to be improved\n  * Notation is in places not defined or ill defined. E.g.\n    * inputs to equations 1, 2, 3 are unclear to me\n    * Some symbols are not defined, e.g. X and Y in section 3.2 The function $\\phi_e$ is defined wrt one input in L137 but takes multiple inputs in Eq 1. \n  * A viasualization of equations 1, 2, 3 would improve readability. This could have been integrated with Figure 4 to not take up more space.\n* Shaded regions in experiments are not explained. I'm assuming they are standard errors.\n* In the examples for early exploration, a description of the MDP is required. It was unclear to me what states, actions and rewards are.\n* Theorem 2 is missing a pointer to a proof.\n\nRelated work\n* As mentioned before I am not an expert in the MARL field. The following might not be the most meaningful metric but I figured I'd point it out for completeness. In general, the number of cited papers in the manuscript is low compared to many other works. I'm not going to base my recommendation on this point because a small number of precise citations can be sufficient but I'm pointing this out in case reviewers more familiar with the literature have similar concerns. That is, I don't necessarily expect more citations to be added to raise my score.\n\nNovelty\n* The text argues that this work is the first work to successfully demonstrate multi-agent equivariant networks on complex tasks but point to some work whose tasks are not complex. It is unclear to me what the measure of complexity is and when a testbed would be sufficiently complex to make this claim about novelty correct. See Q1.\n\nExperiments and Claims\n* Several claims are overstated and need to be adjusted or clarified\n  * The work claims that the method outperforms standard GNNs and MLPs by more than 10x sample efficiency. First, this is not validated in the experiments. I do not know what I should look at to get this 10x number. Further, it seems that this is not generally true since in some cases the benefits are marginal (see Figure 6 right) and needs to be rephrased. \n  *  In section 5.2, the text states that \"equivariant networks have a clear improvement in sample efficiency over the MLP and GNN\". This is not fully supported and needs to be concretized. In Figure 6 (right) the statement is incorrect.\n* The experiments in Figure 6 are somewhat inconclusive. The final performance of E2GN2 is within variance of EGNN for all three experiments while also being conducted over only 5 seeds. For protoss and zerg, the whole training seems to be within variance, challenging the sample efficiency claim. Increasing statistical significance of these results would make the paper stronger.\n* The experimental results in Table 1 and 2 lack a measure of variation. Given that other results seem to be within variance, these measures should be added.\n* There is a lack of baselines that compares to prior work. It is, for instance, unclear to me why theoretical guarantees are useful if other approaches might be better. A comparison against the architectures from [19, 21] may have strengthened arguments for being to first to handle complex action-spaces.\n* The paper states that the base models were chosen and special tricks from the literature were avoided L255 but that the proposed method \"is compatible with most MARL actor-critic methods\" L78. First, the paper does not demonstrate that the latter claim is true. Second, I think the paper would have been stronger if it had been shown that the method work *with* these tricks and stronger methods. In general, we have seen (at least in the single agent literature) that combining many advancements is quite beneficial [1 mine, 2 mine, 3 mine]. This would also solve the problem of the baselines being weak.\n\n[1] Combining Improvements in Deep Reinforcement Learning. Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver. The Thirty-Second AAAI Conference\non Artificial Intelligence, 2017.  \n[2] Bigger, Better, Faster: Human-level Atari with human-level efficiency. Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G Bellemare, Rishabh Agarwal, Pablo Samuel Castro. Proceedings of the 40th International Conference on Machine Learning, PMLR 202:30365-30380, 2023.   \n[3] Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control. Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, Marek Cygan. arxiv eprint 2405.16158.  \n\nI am happy to slightly raise my score if the language of the claims is adjusted, limitations are addressed and the experiments are made statistically significant. I am happy to raise my score more if additional baselines are included to strengthen claims and the generalizability to stronger approaches is demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. We appreciate the time you took to dive into the details of our work and provide specific advice for improving these results. \n\n\nWe tried to take your feedback into account in our updated results. We increased the number of seeds up to 10, and we added a new baseline (see the global rebuttal for those charts) from [20]. We will remove the language regarding \"up to 10x improvement in sample efficiency\" and replace it with simply \"in many cases we see a significant improvement in sample efficiency over GNN and MLP networks\". Perhaps that is worded better?\n\nClarity\nThank you for this feedback. We will make the recommended changes to the mathematical depiction, improve the definitions for equations 1,2,3 and add the theorem 2 pointer to a proof. \n\nExperiments:\n* On further baselines: We addressed the concerns regarding the baselines [19, 20,21] in the global rebuttal. We added the baseline [20] to our comparisons, and discussed further [19,21]. Let us know if you have any other thoughts on these. \nBefore the original submission we did attempt to use [21] as a baseline, but when we tried to implement their approach we did not see any improvement over the standard PPO approach for MPE. (we tried various hyperparameters, and learning rates as well). The experiments in [19] are similar to a grid-world type environment. The input to the traffic control is a 8x8 grid and the cars move one grid per timestep. Their wildlife monitoring environment is a 7x7 grid where the agents move around the grid. Extending their specific approach from a gridworld (discrete number of states) to SMACv2 would likely require further innovations.\n\n\n* On the shaded regions in the experiments: in the submitted paper these regions are the 95% confidence error calculations computed by bootstrapping using python's seaborn. Per your comment on standard errors, the updated plots (see global rebuttal) use standard error instead.\n\n* On the number of seeds, we now have 10 seeds for each run. We note that due to the higher run time of MARL experiments, other MARL papers use around 10 seeds [1a,2a]\n\n\nQ1: On why SMACv2 is more complex: the units are heterogenous with different capabilities (with different attack ranges, total health, and sometimes action spaces). The unit types are randomized at the beginning of the scenario. The actions include more components than simply movement (such as in MPE), agents can move and attack. The goals are more complex as well. Instead of simply navigating cooperatively as in MPE, the agents must learn attack formations and strategies. Sometimes it may be optimal to sacrifice health or allies in the purpose of the greater strategic objective. (see lines 278-286)\n\nQ2: Van der Pol's approach is to find the basis for a specific group (ie rotations). Once the find the basis they make the neural network weights a linear combination of the basis vectors. I don't believe their network would have a bias, but it has the limitation of being applied only to discrete grid-world like environments (see global rebuttal for a further discussion on this). \n\nQ3: In the experiments on SMACv2 (using mixed discrete-continuous actions) EGNN does indeed adapt the suggested solution in 4.3, as does the GNN network. In fact, using the method described in section 4.3 is why GNN was able to increase the number of agents without retraining (Table 2)\n\nLimitations\nWe plan to add a further discussion on limitaitons. It is true that the more symmetry in the environment the greater the improvement we would see from EGNN/E2GN2. I will note these environments weren't necessarily hand-picked. MPE and SMAC are the standard/go to environments for MARL. For example, [1a] uses SMAC and MPE and it has 1000 citations. The SMACv2 paper identified problems with the original SMAC environment, which is why we use SMACv2. That said, E2GN2/EGNN could be overly restrictive in environments where symmetries are inexact or unclear. If EGNN is assuming an exact symmetry, but that symmetry is inexact, this could cause a loss in performance as the inductive bias is keeping the network from learning the optimal solution.\n\nOther citations\n\n1a https://arxiv.org/abs/2103.01955\n2a https://arxiv.org/abs/1706.05296", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IKf1rWJSo4", "reviewer": "Reviewer_2qF5", "summary": "This paper studies an intricate issue, called “early exploration bias”, when applying EGNN, an GNN preserving Euclidean invariance/equivariance, to cooperative MARL that exhibits Euclidean symmetries. The paper reveals that, with randomly initialized weights, the output of the EGNN layers is not centered around zero, which can cause difficulties for cooperative MARL. The paper then proposes a convenient solution to this issue, with a cost of removing translation invariance/equivariance.", "questions": "1. In Figure 1, why is “All learnable equivariant functions constrained by data” not a subset of  “All learnable equivariant functions”?\n\n2. The original EGNN paper [2] uses it for applications other than RL. Is EGNN’s property presented in Theorem 1 also problematic for the applications there? \n\n3. On the theoretical results:\n    - 3a.  In Theorems 1 and 2, what is exactly approximated/hidden in the approximate equalities? \n    - 3b.  In Corollary 1.1 and 2.1, can you clarify the use of subscripts $i$ vs $k$?\n    - 3c.  In Corollary 1.1 and 2.1, what’s the definition of $s_k^{eq}$?\n    - 3d.  In Corollary 1.1 and 2.1, there seems to be a hidden assumption:  $a_i$ has the same dimensionality with $s_k^{eq}$, according to the displayed equation?\n\n4. How exactly is a policy represented by EGNN?\n    - 4a. How is the input graph defined/constructed? \n    - 4b. What is the dimension of $u^l_i$ for all l?\n    - 4c. According to Section 3.3., the dimension of $h^l_i$ is $p$, how to map this $p$-dimensional vector $h_i$ to invariant action components (e.g., logits for the attack targets in SMACv2)? How to ensure such a mapping is equivariant/invariant?\n\n5. The paper does not talk much about value function approximation. How exactly is the value function represented by EGNN?\n\n6. On the experiments:\n    - 6a. What specific GNNs are used in the experiments?\n    - 6b. The experiments use a decentralized critic (IPPO?). Why not use a centralized critic (MAPPO), which is shown in previous work to perform better?\n    - 6c. In Line 287-293, why is “permutation equivariance” relevant? What aspect(s) of the proposed method “allow for” the GNN to scale to larger numbers of agents, in a way that a vanilla GNN cannot?", "limitations": "The paper mentions several future work directions, including addressing partial observability, exploring other domains like robotics.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is overall easy to follow in terms of its motivation.\n- The issue of early exploration bias is well-described using an example (Figure 3), discovering an intricate issue when applying EGNN to MARL.", "weaknesses": "- Euclidean equivariance in deep MARL has been explored in several recent works, namely [20] and [Esp: Exploiting symmetry prior for multi-agent reinforcement learning]. This paper’s novelty against these recent works is limited. The most significant contribution seems to be limited to the pathology of EGNN, fixing it to overcome the “early exploration bias” issue. It is unclear whether this pathology and treatment is even relevant in other types of equivariant NNs (e.g., [20] uses the alternative of SEGNN). Moreover, the paper suggests its method can deal with discrete movement actions at the beginning of Section 4.3, which could be a significant novelty from prior work. However, it turns out that the paper bypasses this issue by changing the movement actions in the original SMACv2 from discrete to continuous.\n\n- The method is motivated by the “early exploration bias”. Its cause and the proposed remedy which seems very specific to the proposed usage of EGNN and particularly specific to (MA)RL (e.g., similar issues could arise in ML for chemistry). The paper might benefit from an investigation and discussion of EGNN used in other domains/applications.\n\n- The presentation of the theoretical results (the Theorems and Corollaries) and the proposed methods lacks clarity. Please refer to the Questions on this.\n\n- This paper’s argument on translation equivariance is not that convincing. The proposed method (E2GN2) removes translation invariance/equivariance from EGNN and the paper argues (e.g., Lines 213-218) that translation invariance/equivariance is not a benefit for MARL. Lines 213-218 uses an example to argue against translation equivariance, while previous work (e.g. [20]) suggests translation invariance is indeed benefiting MARL, supported by both theoretical and empirical results. So, the paper could benefit from a revised approach that preserves  translation invariance, not equivariance.\n\n- There are minor issues regarding the quality of writing, e.g.,\n    - There is an unwanted line break between lines 110 and 111.\n    - Line 122-123: $T_g$ should be $Y \\to Y$ and $L_g$ be $X \\to X$.\n    - Theorem 1: the output vector should be $u_i^{l+1}$.\n    - The format of the references is sloppy, e.g. [2, 19, 20, 22]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough, in depth response! Our replies are below:\n\nWeaknesses\n1.  Note that we added a new comparison with [20] demonstrating ours outperformed the results from [20]. Regarding SEGNN, we initially decided to use EGNN specifically because SEGNN and related networks (ie those using spherical harmonics, wigner-D matrices, etc)  tend to be slow. Looking at the original SEGNN paper it is 10x slower than EGNN. MARL experiments are already too slow, so we did not want to exacerbate this problem.  However, we did add a global response, addressing adding SEGNN from [20] to our comparisons. \n\nRegarding the action space, please see the global rebuttal response. We showed this method is still relevant to SMAC when using discrete actions. Note that E2GN2 and EGNN outperform the baselines in with the discrete action space as well. Our method in 4.3 was crucial for achieving this result. \n\nNote that [20] was not able to get competitive results on SMAC. Additionally, this solution in section 4.3 is not to only change the movement actions to be continuous, it is to decompose the discrete action outputs such that each node for each enemy is outputting one component of that action. This is crucial for retaining the GNN locality structure and allowing us to scale agents without retraining. \n\n\n\n2. I agree it would be interesting to explore the use case in other domains.. However, our focus in this paper is improving MARL, so perhaps we can explore this in the future.\n\n3. Okay. We will update our paper accordingly\n\n4. The results from [20] demonstrate that translation \\textit{invariance} is important. Our observation is wrt translation \\textit{equivaraince} We certainly observed that in practice translation equivariance was harmful. As a simple example. if you have an agent at the position (0,-1) Perhaps the optimal action for agent 1 is to move up: (0,1). Now shift this down by 100, so now this agent's position is at (0,-100). A translation equivariant network will shift the action by -100, leading to the action of (0,-99). This is not desirable for many MARL environments. When testing in MPE and SMAC, this may not cause a large problem, as the inputs are often normalized to be near (-1,1).\n\nQuestions\n1. Thank you for catching this. The orange circle should say \"All learnable functions constrained by data\".\n\n2. This is an interesting question to explore in the future.\n\n3.  On the theoretical results\n   3b I see how that is confusing. We will update to use just $i$ instead of $k$\n   3c The definition of $s_k^{eq}$ is the component of the action space that is equivariant, we will formalize this definition.\n   3d Yes, that is an assumption that originates with EGNN. \n\n\n4. How a policy is represented:\n       \n    4a) The input graph is described in appendix B (we will change the word choice from \"connections\" to \"edges\"). For MPE we use a complete graph. For SMACv2 we use a graph such that it is complete between all friendly nodes. Each friendly node has an edge to each of the enemy nodes. There are no edges between enemy nodes. This was an attempt to model what we imagine a real world type world to look like (you see the enemies, communicate with your allies, but don't see the enemies communications).\n\n   4b) This is partially in appendix B (we reference the width of the MLPs as 32), we can improve the wording by referencing the specific notation, and add details for each environment. For the intermediate GNN layers $u^i_l$ has a dimensionality of 32. For MPE the input $u^0_i$ is the id (pursuer, evader, or landmark). For SMACv2 $u^0_i$ is made up of the features: health, shield unit type, and team. \n\n   4c) This is described in section 4.3 and is key to our approach. For smacv2, the discrete logits for attack targets are the outputs of each node \\textit{corresponding to the attack target}. The logit for attacking agent j is the output for $h_j^L$. Since the output of each individual node $h_j^L$ is invariant, these will then remain invariant. These logits are concatenated from each of the nodes to compose the final action. This allows us to scale to larger numbers of agents without retraining. \n\n5) We did not spend much discussion on the value function approximation, as it has been discussed in prior works [19,21]. The value function is fairly straightforward, we simply used an EGNN/E2GN2 for the value function. The ouput of the own node's invariant component was the value function output ie $u_i^L$ for agent $i$. We will add this to appendix B\n\n6) On the experiments\n\n     6a) We will add this to the appendix. We used the same GNN as the EGNN paper: Specifically, we use equations 1 and 3, but update equation 1 to be $m_{ij} = \\phi(h_i, h_j, x_i,x_j)$.\n\n     6b) A standard GNN layer is permutation equivariant [24]. It also has a local structure that enables adding/removing nodes without needing to retrain the layers. For example, [1a] demonstrates using a GNN to train  N agents, then to control $N+n$ agents without retraining. However, this example did not operate with discrete or mixed action spaces as we do with SMAC. \n     \n     Our formulation in 4.3 enables us to retain the permutation equivariance of a GNN.  An alternative to our approach is to add an MLP at the end of the GNN to convert the GNN outputs to the mixed action spaces. This will lose the scalability/locality of the neural network. For example, if you add two more enemies how do you modify the final MLP to expand the action space? In our formulation, whenever an new enemy $i$ is added to the environment, we can simply add the node $N+1$ to the graph. The action space of the GNN will now be supplemented with $u_{N+1}^L$ allowing us to expand the action space without retraining.\n\n        \n \nNew Citations:\n[1a] Learning Transferable Cooperative Behavior in Multi-Agent Teamhttps://arxiv.org/abs/1906.01202 (GNN)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "57dqfcYSze", "reviewer": "Reviewer_wBzN", "summary": "The paper applies Equivariant Graph Neural Networks (EGNN) to address the issue of low sample efficiency in multi-agent reinforcement learning (MARL). The authors demonstrate that in certain scenarios, EGNN outperforms Multi-Layer Perceptrons (MLP) and Graph Neural Networks (GNN). Key contributions include showcasing EGNN's ability to handle complex action spaces and proposing methods to enhance the exploration capabilities of agents within this framework. The paper presents a limited number of experimental results, attempting to illustrate the superiority of EGNN over some methods in improving exploration efficiency in MARL tasks.", "questions": "1. **Difference from Trajectory Prediction**: How does your application of EGNN in multi-agent reinforcement learning (MARL) fundamentally differ from trajectory prediction in multi-agent systems?\n2. **Rationale for Zero-Mean Gaussian Policies**: Why is using zero-mean Gaussian policies considered best practice?\n3. **Partial Observability Implementation**: How is partial observability handled in your experiment? Could you provide a detailed explanation of your method?\n4. **Enhancing Exploration with Entropy-Based Methods**: Is it possible to incorporate entropy-based methods to enhance exploration in your EGNN framework?", "limitations": "The authors need to explore the scope of symmetry utilization to clearly articulate the limitations of their method.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors propose using Equivariant Graph Neural Networks (EGNN) to address the lack of sample efficiency in multi-agent reinforcement learning (MARL), providing insights to promote the application of EGNN in MARL further.\n2. They identify the challenge of handling complex action spaces with EGNN and propose a solution to this problem.\n3. Highlighting the lack of exploration capability when directly applying EGNN to MARL introduces a new intuition.", "weaknesses": "1. The paper lacks baselines that utilize symmetry [1,2,3].\n2. There is a lack of entropy-based baselines to compare with for improving exploration capabilities [4].\n3. Not all tasks can leverage symmetry, so the paper needs to specify the applicable scope of the proposed method.\n4. The innovation is unclear, as extensive work on multi-agent systems is already based on EGNN. Applying EGNN to MARL appears more like trajectory prediction [3]. If the system is controlled solely by displacements in the x and y directions, it essentially remains trajectory prediction, similar to EqMotion [3].\n5. The experimental descriptions are insufficient, lacking clarity on how features are designed, such as what constitutes equivariant and invariant features. Additionally, the use of symmetry is closely related to system state transitions, but the dynamics of these transitions are not introduced.\n6. The approach to handling partial observability in multi-agent systems is unclear.\n7. The reasoning for using zero-mean Gaussian policies is insufficiently explained. Moreover, if the goal is to enhance exploration, it is unclear why entropy-based methods are not used for comparison.\n8. Due to the unclear method descriptions and the lack of open-source code, the reproducibility of the paper is poor.\n9. Although EGNN can be applied in multi-dimensional spaces, the experiments are all conducted in two dimensions.\n\n[1] Muglich D, Schroeder de Witt C, van der Pol E, et al. Equivariant networks for zero-shot coordination[J]. Advances in Neural Information Processing Systems, 2022, 35: 6410-6423.\n\n[2] Yu X, Shi R, Feng P, et al. Leveraging Partial Symmetry for Multi-Agent Reinforcement Learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(16): 17583-17590.\n\n[3] Xu C, Tan R T, Tan Y, et al. Eqmotion: Equivariant multi-agent motion prediction with invariant interaction reasoning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 1410-1420.\n\n[4] Liu I J, Jain U, Yeh R A, et al. Cooperative exploration for multi-agent deep reinforcement learning[C]//International conference on machine learning. PMLR, 2021: 6826-6836.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your response. We have updated our charts, and discuss the novelty further below. We hope you will consider increasing the score given the updates and content.\n\n1. See the common rebuttal for an additional baseline. [3] is not a MARL algorithm. we attempted to replicate [2], but failed to see any improvement over standard PPO. [1] is only relevant to discrete action spaces and discrete observation spaces. However, we will add citations to [1,3] (we already cite [2]) in the related works section: with the following statement)\n\n2. Entropy baselines, while potentially helpful are not the standard for MARL (see questions below), and our goal was to compare with standard MARL approaches. Additionally, our contribution was not limited to addressing the exploration bias of EGNN, it was also our contribution to apply EGNN to complex MARL environments leading to improved sample efficiency and generalization.\n\n3. We believe many tasks will have some form of symmetry, but that symmetry may be inexact, or only partially symmetric. Our approach here cannot currently address those types of problems. However, we are excited to attempt to pursue this in future works.\n\n4. This seems to be the first work applying EGNN to MARL, and the first work to successfully apply any equivariant neural networks to the common MARL SMACv2 benchmark. See questions section for discussion of trajectory prediction\n\n5. These are common benchmarks in MARL. We did not design any new features, we used the standard outputs of these environments. \n\n6. see lines 265-266\n\n7. See below in questions. for discussion about the gaussian. The goal is not necessarily to only enhance exploration. The goal was to apply Equivariant Neural networks to MARL problems. In the process of using EGNN we noticed that it’s bias was causing a problem in exploration. Thus we had to modify EGNN to improve it’s performance in MARL.\n\n8. We believe this approach to be fairly reproducible. We knew we may not get permission to release our code. To that end, we specifically used open source libraries for everything we did. The RL code is the open source library RLLIB. We used the code from the EGNN paper for the neural networks. The environments are also open source, as are the features of the environments. \n\n9. We applied EGNN in 2 dimensions, because our problems (standard benchmarks in MARL literature) were in 2 dimensions. It could certainly be applied to more depending on the problem.\n\n\nQuestions\n\n1. MARL and trajectory prediction are related but fairly different. MARL is attempting to control a set of agents to achieve some objective. This entails learning a value function to approximate the value of each state in achieving that overall objective and a policy to control the agents to achieve that objective. Trajectory prediction is focused on predicting the next step of dynamics. Trajectory prediction is relevant to model-based RL algorithms. but in this case we are focused on model free algorithms\n\n2. We used the gaussian output for the policy, because that is common practice in the field. Common RL code libraries such as stable baselines https://stable-baselines.readthedocs.io/en/master/, RLLIB all use gaussian parameterized policies. See also: ”...outputting the mean of a Gaussian distribution, with variable standard deviations, following [Sch+15b;\nDua+16]” from original PPO paper ”The two most common kinds of stochastic policies in deep RL are categorical policies and diagonal Gaussian policies.” https://spinningup.openai.com/en/latest/spinningup/rl intr\nWe wanted our approach to be as broadly applicable as possible so wef ocused on common practices.\n\n3. see lines 265-266\n\n4. I think that is an interesting idea, and one worth exploring in future efforts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uXqEZ14o6L", "reviewer": "Reviewer_TKRS", "summary": "this paper proposes an exploration-enhanced equivariant graph neural networks. The effective utilization of samples has been achieved in the experimental results. and It  is an improvement on the function approximation for MARL and thus is compatible with most MARL actor-critic methods.", "questions": "No", "limitations": "No", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The algorithm proposed by the author has been extensively experimented in MPE and SMACv2 environments, and its clear structure is a good paper. And this structure can be applied to other MARL algorithms", "weaknesses": "No", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your response and praise in strengths! We added some additional plots in the global response. Let us know if you have any questions regarding those additional results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all of their reviewers for their thoughts, comments and advice for improving this research work. We know how busy you all are, and we are grateful you have taken time out of your schedule to provide a thorough and fair review. We are encouraged they found  appreciated our insight into describing and adressing the bias of EGNN in MARL, to improve sample efficiency in MARL.

Based on the below feedback, we added an additional benchmark to our comparisons [20].If accepted we will update the plots with these included here. We will refer to [20] as E3AC (referring to their title).* As before, we ran [20] using RLLIB's PPO to ensure we were using a common benchmark, we trained using the hyperparameters listed in appendix B. Although E3AC is competitive with EGNN on MPE simple spread, it fails to perform as well as E2GN2 in the simple tag. Additionally, although their approach is sample efficient, it is very slow and cumbersome to run. This makes running, tuning and devloping E3AC fairly difficult This is one of the reasons why we chose to use EGNN over other equivarant grpah neural networks: EGNN is much quicker and simpler to run.

Further, as we mentioned in our original paper, E3AC fails to bring the benefits of increased sample efficiency to the more complex problem posed by SMACv2. The results of E3AC are comparable to an MLP. E3AC seems to perform slightly worse on the Zerg environment. We believe this may be due to the fact that some of the zerg agents have more complex action spaces and dynamics (ie some of them explode on contact).

Other benchmarks suggested for comparison include [19, 21]. For [19], their method is specifically formulated and tested on small discrete grid world problems with simple dynamics and discrete up, down,left actions. For example, the trafic control problem has an input of a 7x7 grid. Extending their work to continuous environments with large state spaces, large mixed discrete/continuous action spaces was not straighforward without modifications. For [21], in the past we attempted to replicate their methods , but we failed to see any improvment over PPO on the MPE environments, let alone the more difficult SMAC environments.


Several reviewers were curious about how our results would fare on SMAC with the default discrete action space instead of the mixed continuous discrete space. We present those results here. To map the E2GN2 output to discrete, the continuous movements are mapped back to discrete actions by simply rounding to the nearest discrete action \textit{after} performing exploration (ie sampling from the gaussian distribution). The remaining actions are already discrete. We present those results here as well. We believe this emphasizes the importance of our method described in 4.3, in enabling EGNN structures to mixed discrete-continuous and fully discrete action spaces.

Meta-review: This paper is focused on incorporating equivariance priors into MARL algorithms. This is done by evaluating the already-established EGNN architecture in MARL algorithms. The paper then shows that such an approach has limitations, mainly regarding the exploration strategy it 
induces and the discussion around adapting this idea to more complex action spaces. The paper empirically validates its claims on two sets of environments, MPE and SMACv2, showing that EGNN has a slow start (which the paper attributes to exploration issues without necessarily providing data to back that up) and that E2GN2 (the proposed method fixing the exploration issue) has a much better start. The paper also validates its claims around the generalization benefits of using such an inductive bias. Some of the criticisms around the number of independent runs used, and the lack of a specific baseline were addressed to some extent during the rebuttal phase.

The paper has issues, though. One issue that was brought up by pretty much every reviewer is clarity. Many reviewers think the paper is unclear on some of the design choices, and the descriptions provided are insufficient. Although open-sourcing the code is not a requirement for getting a paper accepted at NeurIPS, the lack of clarity added to the impossibility of open-sourcing the code is a major concern. Another major concern is overclaiming. The original version of the paper has very strong statements that are hard to understand where they came from (e.g., 10x improvement). The authors promise to fix the issue, and they provide an example of how they would do so in a specific instance, but it is hard to guarantee that those issues will be fully addressed. There have been many other discussions about relevant related work that should maybe be discussed, or the specific neural network architecture being used, or the scope of the experiments (e.g., partial observability or not, centralized critic or not), which is something that I did not consider so much because I think the paper does a good job defining its scope.

I do feel the pros outweigh the cons here, and I recommend the paper's acceptance. However, I strongly encourage the authors to improve the presentation and adjust their claims.
Decision: Accept (poster)