Boosting Sample Efficiency and Generalization in Multi-agent Reinforcement Learning via Equivariance

Authors: joshua  mcclellan
jhu  apl
naveed  haghani
john  winder
furong  huang
pratap  tokekar
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:08:50

Abstract: Multi-Agent Reinforcement Learning (MARL) struggles with sample inefficiency and poor generalization [1]. These challenges are partially due to a lack of structure or inductive bias in the neural networks typically used in learning the policy. One such form of structure that is commonly observed in multi-agent scenarios is symmetry. The field of Geometric Deep Learning has developed Equivariant Graph Neural Networks (EGNN) that are equivariant (or symmetric) to rotations, translations, and reflections of nodes. Incorporating equivariance has been shown to improve learning efficiency and decrease error [ 2 ]. In this paper, we demonstrate that EGNNs improve the sample efficiency and generalization in MARL. However, we also show that a naive application of EGNNs to MARL results in poor early exploration due to a bias in the EGNN structure. To mitigate this bias, we present Exploration-enhanced Equivariant Graph Neural Networks or E2GN2. We compare E2GN2 to other common function approximators using common MARL benchmarks MPE and SMACv2. E2GN2 demonstrates a significant improvement in sample efficiency, greater final reward convergence, and a 2x-5x gain in over standard GNNs in our generalization tests. These results pave the way for more reliable and effective solutions in complex multi-agent systems.

Main paper: 
 introduction: 
 Multi-Agent Reinforcement Learning (MARL) has found success in various applications such as robotics [3, 4, 5] , complex strategy games [6, 7, 8] and power grid management [9, 10] . However, MARL algorithms can be slow to train, difficult to tune, and have poor generalization guarantees [1, 11] . This is partially because typical implementations of MARL techniques use neural networks such as Multi-Layer Perceptrons (MLP) that do not take the underlying structure into account. The models learn simple input/output relationships with no constraints or priors on the policies learned. These generic architectures lack a strong inductive bias making them inefficient in terms of the training samples required.Symmetries are commonplace in the world. As humans, we exploit symmetries in our daily lives to improve reasoning and learning. It is a basic concept learned by children. Humans do not need to relearn how to eat an apple simply because it has been moved from the right to the left. In soccer, if you have learned to pass to someone on the right, it is easier to learn to pass to someone on the left. Our objective is to develop agents that are guaranteed to adhere to these basic principles, without needing to learn every single scenario from scratch. Symmetries are particularly common in MARL. These occur in the form of equivariance and invariance. Given a transformation matrix T , if f (T x) = f (x) that function is said to be invariant. Similarly, f (.) is equivariant if f (T x) = T f (x) [12] . Rotational and reflection symmetries (the O(n) symmetry group) are particularly common in Reinforcement Learning (RL) scenarios. For example, consider the Multi-agent Particle Environment (MPE) [13] benchmark for MARL which has agents with simple dynamics and tasks. We note that this scenario adheres to rotational equivariance. As shown in Figure 2 rotating the agent's positions results in the optimal actions also being rotated. A policy that is rotationally equivariant effectively shrinks the state-action space for the problem, potentially making the problem easier to learn (Figure 1 ).One way to guarantee equivariance to rotations and reflections in MARL is to use an Equivariant Graph Neural Network [2] . Due to the more complex and equivariant structure of the EGNN, directly adapting EGNNs to MARL is not straightforward. Specifically, consider action spaces typically represented as discrete choices (e.g., up, down, left, shoot). Typically the policy over such an action space is represented using logits (to specify the probability of choosing each action). EGNN outputs a continuous equivariant output. Mapping this output to logits is non-trivial. Therefore, we ask the question What is the correct representation of a complex action space when using Equivariant GNNs?Another specific issue we observed in EGNNs is an exploration bias in the early stages of learning due to the specific nature of the equivariant computation. This bias can lead to poor initial exploration, decreasing the probability of the agent finding the optimal policy, and potentially resulting in suboptimal convergence. Therefore we ask the question How can we design a GNN equivariant to rotations, but without early exploration bias?The main contribution of this paper is Exploration-enhanced Equivariant Graph Neural Networks (E2GN2). This addresses the two research questions. Specifically, we show how to apply Equivariant GNNs to complex MARL tasks, to mixed discrete/continuous action spaces, and to mitigate the early exploration bias. Our contributions can be summarized as:(1) Our approach is the first to successfully demonstrate Equivariant GNNs for MARL on standard MARL benchmarks with complex action spaces.(2) We propose E2GN2 which has no bias in early exploration for MARL and is equivariant to rotations and reflections.(3) We evaluate E2GN2 on common MARL benchmarks: MPE [13] and Starcraft Multi-agent Challenge v2 [14] using PPO. It learns quicker, outperforming standard GNNs and MLPs by up to 2x-5x2 in sample efficiency on terran and protoss challenges. It is worth noting that E2GN2 is an improvement on the function approximation for MARL and thus is compatible with most MARL actor-critic methods.(4) We showcase E2GN2's ability to generalize to scenarios it wasn't trained on, due to the equivariance guarantees. This results in 5x performance over GNNs 
related works: 
 A key theoretical foundation for our paper is [15] , which formulated the structure for equivariant MDPs. One important takeaway is that if a reward is equivariant to a transformation/symmetry, we want the policy and dynamics to be equivariant to that symmetry. However, this work was limited to very simple dynamics with discrete actions such as cart-pole. [16] followed up with equivariance in multi-agent, but was again limited to simple problems. Their method is specifically formulated and tested on small discrete grid world problems with simple dynamics and discrete up, down,left actions. For example, the trafic control problem has an input of a 7x7 grid. Extending their work to continuous environments with large state spaces, large mixed discrete/continuous action spaces is not straightforward without significant modifications.Contemporary to our research, [17] demonstrated E(3) equivariant networks on simple cooperative navigation problems. However, their results on more complex tasks, such as Starcraft, did not excel over the baseline. Additionally, they used SEGNN [18] which can result in very slow training times, making tuning difficult and cumbersome. Others took the approach [19] of attempting to learn symmetries via an added loss term. However, since this approach needed to learn the symmetries in parallel with learning it did not have the same guarantees as Equivariant GNNs, and did not result in significant performance gains. Another work [20] demonstrated rotation equivariance for complex robotic manipulation tasks. This work, while promising, was for single-agent RL and used image observations and many problems don't have access to image observations. AI research in chemistry has taken a particular interest in adding symmetry constraints to GNNs. Works such as EGNN, [2] SEGNN, [18] E3NN, [21] and Equivariant transformers have demonstrated various approaches to encoding symmetries in GNNs. In this paper, we chose to focus on EGNN due to its simplicity, high performance, and quick inference time. Other works took a different approach, such as [22] , which proposes Equivariant MLPs, solving a constrained optimization problem to encode a variety of symmetries. Unfortunately, in our experience, the inference time was slow, and we preferred a network with a graph structure such as EGNN. 
marl: 
 Multi-agent reinforcement learning (MARL) considers problems where multiple learning agents interact in a shared environment. The goal is for the agents to learn policies that maximize long-term reward through these interactions. Typically, MARL problems are formalized as Markov games [23] . A Markov game for N agents is defined by a set of states S, a set of actions A 1 , ..., A N for each agent, transition probabilities P : S × A 1 × ... × A N × S → [0, 1], and reward functions R 1 , ...R N mapping each state and joint action to a scalar reward. The goal of each agent i is to learn a policy π i (a i |s) that maximizes its expected return:J(π i ) = Eπ 1 , ..., π N T t=0 γ t R i (s t , a 1 t , ..., a N t )Where T is the time horizon, γ ∈ (0, 1] is a discount factor, and a j t ∼ π j (•|s t ). The presence of multiple learning agents makes this more complex than single-agent RL due to issues such as non-stationarity and multi-agent credit assignment.Crucial to equivariance is group theory. A group is an abstract algebraic object describing a symmetry. For example, O(3) describes the set of continuous rotation symmetries. A group action is an element of that particular group. To describe how groups o perate on data we use representations of group actions. A representation can be described as a mapping from a group element to a matrix, ρ : G → GL(m) where ρ(g) ∈ R m×m Or instead we can more simply use: L g : X → X where g ∈ G where L g is the matrix representation of the group element g ∈ G [24] .A function is equivariant to a particular group or symmetry if transforming the input is equivalent to transforming the function output. More formally, T g f (x) = f (L g x) for g ∈ G, L g : X → X and T g : Y → Y . Related to equivariance is the key concept of invariance, that is a function does not change with a transformation to the input: f (x) = f (L g x). [24] Previous work [16] has shown that if a Markov game has symmetries in the dynamics and the reward function then the resulting optimal policy will be equivariant and the value function will be invariant. That is, V (L g s) = V (s) and π(L g s) = K g π(s) where L g and K g , with g ∈ G are transformations to the state and action respectively.Equivariant Graph Neural Networks [2] (EGNN) are an extension of a standard Graph Neural Network. EGNNs are equivariant to the E(n) group, that is, rotations, translations, and reflections in Euclidean space. EGNNs have two vectors of embeddings for each graph node i: the feature embeddings denoted by h l i , and coordinate embeddings denoted by u l i , where l denotes the neural network layer. The equations describing the forward pass for a single layer are below:EQUATIONEQUATIONEQUATIONHere ϕ represents a multi-layer perceptron, where ϕ e : R n → R m , ϕ u : R m → R, and ϕ h : R m+p → R p . The key difference between EGNN and GNN is the addition of coordinate embeddings u i and equation 2, which serves to update the coordinate embeddings in a manner that is equivariant to transformations from E(n). Note that u i will be equivariant and h i will be invariant to these transformations [2] .As noted in Section 1, application of EGNN to MARL is not straightforward. In the following section, we discuss these issues in more depth and present our solution towards addressing them. 
methods: 
 In this section, we address both theoretically and empirically how the output of EGNN is biased by the input, leading to suboptimal exploration in RL. To mitigate this issue, we introduce Explorationenhanced Equivariant Graph Neural Networks (E2GN2) as a method that ameliorates this bias, leading to improved exploration.An important component of reinforcement learning is exploration. Practitioners often use a policy parameterized by a Gaussian distribution, where the mean is determined by the policy network output, and the standard deviation is a separately learned parameter. Best practices are that the actions initially have a zero mean distribution to get a good sampling of potential state-action trajectories, i.e., π(a i |s) ∼ N (0, σ). Below we show that an EGNN will initially have a non-zero mean distribution, which can cause problems in early training.Theorem 1 Given a layer l of an EGNN with randomly initialized weights, with the equivariant component input vector u l i ∈ R n , equivariant output vector u l+1 i ∈ R n and the multilayer perceptron ϕ u : R m → R, where the equivariant component is updated as:u l+1 i = u l i + C j̸ =i u l i -u l j ϕ u (m ij ).Then the expected value of the output vector is approximately the expected value of the input vector:E u l+1 i ≈ E u l iFurthermore, given a full EGNN with L layers then the expected value of the network output is approximately the expected value of the network inputE u L i ≈ E u 0 i (See appendix A for proof.)Corollary 1.1 Given a policy for agent i represented by an EGNN and parameterized with a Gaussian distribution, and the equivariant component of the state s eq i the policy will have the following distribution:π i (a i |s) ∼ N (s eq i , σ) (See Appendix A for proof.) In many cases s eq i is the agent's position. Corollary 1.1 indicates that an agent's action distribution will be skewed towards its own position. If the magnitude of the state representation s eq i is significantly larger than σ, the agent may output actions approximately equal to its position. We refer to this phenomenon, where agent actions are biased towards replicating the input state as the action, as the early exploration bias. Such bias is not conducive to effective exploration, potentially limiting the agent's ability to discover and converge to optimal solutions. Examples of early exploration bias Suppose there is an agent placed at the position u i = [2, 0], and its objective is to reach the origin (i.e. 0). The agent needs an action of less than 0 to move to the left. Since its action using the EGNN is close to Gaussian distributed with a mean of 2, then P (X < 0) ≈ 0.02. If the agent moves 0.25 with each action, then the probability of the agent reaching the center objective in eight steps is 2.9e-8. On the other hand, the agent has a high likelihood of moving toward infinity as each move away increases the likelihood of moving towards infinity. This example shows that the bias leads to a low probability of finding the solution, which is especially harmful in sparse reward environments. To confirm our hypothesis on EGNN early exploration bias, we conducted a simple experiment. We trained a Proximal Policy Optimization (PPO) [3] agent on the MPE simple-spread problem. This problem consists of three agents that seek to navigate to three goals cooperatively; the agents are rewarded based on the sum of the minimum distance from each target to an agent. We give further details of this set up and training procedure in the experiments section. Figure 3 shows the results. The EGNN policy biased the agents to move in the direction of their current position, causing them to move away from the origin and receive low rewards. Due to the dense rewards and simplicity of the problem, the EGNN agent was able to overcome this initial bias and still solve the problem. However, in more complex problems this early bias could cause more problems. Even if an agent finds an optimal trajectory, it will be a relatively small proportion of the sampled trajectories and may result in sub-optimal convergence.As discussed previously, EGNN's severe early exploration bias can decrease learning performance. In this section, we propose our solution to this problem in the form of Exploration-enhanced Equivariant Graph Neural Networks (E2GN2). To create E2GN2 we make the following modification to Equation 2 of the equivariant component of EGNN:EQUATIONwhere ϕ u2 (m i ) : R m → R is an MLP parameterized by u 2 . The remaining layer update equations of the EGNN remain the same.Theorem 2 Given a L layer E2GN2 with randomly initialized weights, where the equivariant component is updated as:u l+1 i = u l i ϕ u2 (m i ) + C j̸ =i u l i -u l j ϕ u (m ij )).Then the expected value of the output vector is approximately the expected value of the input vector:E u l+1 i ≈ 0Corollary 2.1 Given a policy for agent i represented by an E2GN2 and parameterized with a Gaussian distribution the policy will have the following distribution:π i (a i |s) ∼ N (0, σ)Thus we see that E2GN2 should have unbiased early exploration for the equivariant component of the actions. The primary difference between EGNN and E2GN2 is the addition of ϕ u2 , which serves to offset the bias from the previous layer (or input) u l i To validate that this did indeed solve the exploration bias we tested E2GN2 on the same MPE simple spread environment. We observe in Figure 3 that E2GN2 did indeed have unbiased behavior when it came to early exploration, as the agents had smooth random actions, and the reward did not drastically decrease.Here we verify that E2GN2 still retains EGNN's guarantee of equivariance to rotations and reflections.Theorem 3 E2GN2 is equivariant to transformations from the O(n) group. In other words, it is equivariant to rotations and reflections.(See appendix A for proof.)Retaining our symmetries to rotations and reflections is important, since it should increase our sample efficiency and generalization. Note that E2GN2 is no longer equivariant to translations. However, in MARL translation equivariance is not necessarily a benefit. For example, consider the MPE problem with a translational equivariant policy. If we shift an agent to be 10 units right, this will add 10 to the action as well, causing it to move to the right! Essentially, this is adding an undesirable bias to the policy output: π(s + b) = π(s) + b However, we can expect O(n) policy equivariance to improve our sample efficiency in MARL, and it is key that E2GN2 retains this guarantee.Applying EGNN/E2GN2 to sophisticated MARL problems requires careful consideration. Many MARL environments have discrete action spaces or mixed continuous-discrete action spaces. Some components of these action spaces may be invariant and others may be equivariant. Further complicating the problem, MARL requires the neural networks to output parameters for a probability distribution (which is sampled for use in exploration). For continuous actions, agents typically use a gaussian and for discrete actions agents generally use logits. Unfortunately, it is not straightforward to map the distinct invariant and equivariant coordinate embeddings onto a single distribution in a manner that preserves equivariance.A core benefit of the GNN structure in MARL is the scalability afforded by permutation equivariance: the ability to handle a variable number of agents without retraining. For example, [25] demonstrates using a GNN to train N agents, then to control N + 1 agents without retraining. However, this example did not operate with discrete or mixed action spaces or equivariant structures as we do here. Improperly mapping GNN outputs to logits risks losing this scalability.To address these issues, we propose leveraging the GNN's graph structure to output different components of the action space from different nodes in an equivariant manner:• Discrete Actions & Invariant Components: the invariant feature embeddings h i of each agent's node are used to output discrete logits/actions and other invariant components. For action spaces with both equivariant and invariant components, this can be used as the 'action type selector' to select which type of action to apply at that type step (ie move action or targeting action). • Continuous Spatial Actions: the equivariant coordinate embeddings u i of each agent's node are used for continuous spatial actions such as movement. • Targeting Specific Entities: for multi-agent environments with entities beyond the learning agents (e.g., enemies), logits for discrete actions pertaining to each entity (e.g., which enemy to target) are output from that entity's corresponding node. This enables the discrete action space to scale with the number of agents.with invariant and equivariant components) Each of these components is parameterized as a distribution (ie logits or gaussian), resulting in one to three seperate distributions (depending on the environment and which components are used). After the exploration phase samples from each of these distributions, the final action can be constructed from each of these components via concatenation or the action selection component.Equivariant Graph Neural Network Value Output (Invariant) h 0 u 0 Agent 1 Agent 2 Agent 3 Agent 4 h 0 u 0 h 0 u 0 h 0 u 0 h L u L h L u L h L u L h L u L h L u L h L u L h L u L h L u L h u h u h u h u h u h u h u h u h u h u h u h u h u h u h u h u h u h u h u h u h u h u h u h uThis formulation enables us to retain the permutation equivariance and scalability of a GNN. An alternative to our approach is to add an MLP at the end of the GNN to convert the GNN outputs to the mixed action spaces. This will lose the scalability/locality of the neural network. For example, if you add two more agents how do you modify the final MLP to expand the action space? In our formulation, whenever an new entity i is added to the environment, we can simply add the node N + 1 to the graph. The action space of the GNN will now be supplemented with u N +1 and h N +1 , allowing us to expand the action space without retraining.By structuring the GNN output in this manner, we can handle discrete or mixed discrete-continuous action spaces while retaining the equivariance of EGNN/E2GN2 and the flexibility of GNNs to a variable number of agents and entities. This approach allows MARL agents based on equivariant GNNs to be applied to challenging environments with minimal restrictions on the action space complexity. Indeed this approach was key to enabling the success of EGNN/E2GN2 on SMACv2. 
experiments: 
 We seek to answer the following questions with our experiments: (1) Do rotationally equivariant policy networks and rotationally invariant value networks improve training sample efficiency?(2) Does E2GN2 improve learning and performance over EGNN? (3) Does Equivariance indeed demonstrate improved generalization performance?To address these questions, we use common MARL benchmarks: the multi-agent particle-world environment (MPE) [13] and Starcraft Multi-agent Challenge version 2 (SMACv2) [14] . Our experiments show that equivariance does indeed lead to improved sample efficiency, with E2GN2 in particular performing especially well. We also demonstrate that generalization is guaranteed across rotational transformations.We want to focus our experiments on the neural networks' effects on MARL performance. To isolate the impact of the network architecture, we avoid using specialized tips and tricks sometimes employed in MARL [26] . This allows us to demonstrate that our proposed networks can improve performance without relying on these additional techniques. Thus we use a common standardized open source MARL training library RLlib [27] . We use the default multi-agent PPO algorithm, which does not use a centralized critic. We followed the basic hyperparameter tuning guidelines set forth in [26] .That is, we use a large training batch and mini-batch size, low numbers of SGD iterations, and a small clip rate. Further hyperparameter details are found in appendix B.We compare our results with common neural networks used for MARL benchmarks: multi-layer perceptrons (MLP), and GNNs (we use a GNN structure similar to equations 1, 3, see appendix B for details ). We also compare with the approach from [17] which we refer to as E3AC in our plots.Recall that E3AC also uses neural networks that guarantee equivariance: SEGNN. We integrated E3AC into RLLIB to ensure the RL training procedure remained consistent across our comparisons. Note that the majority of MARL papers on SMAC/SMACv2 use MLPs as the base network. We also compare with GNNs to show the improvement is not primarily due to the graph structure. For the main paper results, we assume full observability since we have not explicitly tackled the partial observability problem yet. In future work, we will seek to remedy this. However, to be thorough we performed experiments with partial observability, resulting in surprising success using E2GN2 (see appendix C).From the MPE environment ( [13] ), we use two environments to benchmark our performance: cooperative navigation also known as spread, and predator-prey, also known as tag. The MPE tag environment consists of three RL-controlled agents seeking to catch a third evader agent. For easier comparison, we use a simple heuristic algorithm to control the evader agent. The evader simply moves in the opposite direction of the pursuers. The other MPE environment is the cooperative navigation or simple spread. In this environment, each agent seeks to minimize the distance between all obstacles. To better test equivariance, for the MPE environments we use continuous actions instead of the default discrete actions.Next we test on SMACv2, a much more difficult environment than MPE. In SMACv2, the units are heterogenous with different capabilities (with different attack ranges, total health, and sometimes action spaces). The unit types are randomized at the beginning of the scenario. The actions include more components than simply movement (such as in MPE), agents can move and attack. The goals are more complex as well. Instead of simply navigating cooperatively as in MPE, the agents must learn attack formations and strategies. Sometimes it may be optimal to sacrifice health or allies in the purpose of the greater strategic objective. SMACv2 has three basic scenarios defined by the unit types: terran, protoss, and zerg. We use 5 agents for each team, and the initial position configuration termed "surrounded" (see fig 7 ) The SMACv2 action space poses an interesting problem for GNN structures. A key advantage of GNNs is the permutation equivariance, which leads to scalability without retraining. The default SMACv2 agents will output simply a discrete action determining movement or target for shooting.For our purposes, we modify the action space to be a mixed action space. This consists of a continuous vector for movement actions, a discrete action determining the attack target, and a boolean determining if the agent should shoot, move or no-op (to be complete, we include plots with a discrete action space in appendix C). As discussed in section 4.3 this will both prove a greater test for our algorithms and allow for the GNN to scale to larger numbers of agents. We present the results from our training in this section. The results for MPE in Figure 5 are averaged across 10 seeds. As discussed previously, EGNN has poor early performance due to the early exploration bias. Despite this poor exploration, EGNN outperforms GNNs and MLPs, demonstrating the power of equivariance in MARL. Similarly, we note that in MPE tag, E2GN2 results in a strong final agent, while EGNN suffers in the initial training phases. We see that E2GN2 is able to greatly outperform EGNN's final reward, partly due to superior early exploration.In figure 5 we also compare the wall clock time required to train each algorithm. Note that these were all trained on the same hardware and using the same training algorithm (RLLIBs PPO). E3AC remains competitive in sample efficiency on MPE spread, but it requires nearly four hours to gather one million time steps, compared to less than one hour for E2GN2. This is likely due to the SEGNN [18] underlying structure used by E3AC, which is slower due to the be slower for inference time. We also note that E2GN2 does manage to outperform E3AC on MPE Tag.Next, we review the results from SMACv2 in Figure 6 ; these results are averaged across 10 seeds. The equivariant networks have a clear improvmement in sample efficiency over the MLP and GNN. On the terran environment, EGNN once again learns slower in the initial phases, but due to equivariance, it outperforms MLP/GNN. For protoss, we note that EGNN performs well but has a high variance for its performance. E3AC struggles to perform well on SMACv2, likely since it was unable to resolve the hurdle we addressed in section 4.3. Instead for SMACv2 E3AC used a simple MLP for the policy and SEGNN for the value function [17] . The training speed and performance for E2GN2 is especially impressive since this environment is much more complex than the MPE. Thus far we have demonstrated that equivariance does indeed lead to improved sample efficiency. The next question to answer is whether equivariance leads to improved generalization. We test this by using the 'surrounded' initial configuration from SMACv2 shown in figure 7 . By default, the agents are randomly generated in all directions (along specific axes). To test generalization, we only initialize agents on the left side of the map (i.e., surrounded left). We then test to see if the agents are able to generalize when they are tested with initial positions starting on the right (called surrounded right), and with the default surrounded configuration (termed Surrounded All or Surrounded). Theoretically, the equivariant network should see equivalent performance between the training and testing initialization, due to the guarantee of rotational equivariance for movement, and invariance for shooting actions.The results of our tests are shown in Table 1 . As expected E2GN2 has stellar generalization performance. The win rate remains the same from the training configuration (Surrounded Left) to Surrounded Right. We see an increase in win rate when testing in Surrounded All. This is likely because this Surrounded All is an easier scenarion; the enemy is divided into more groups, so the agents can defeat the smaller groups one at a time. As we mentioned in the abstract, this table indicates that for the Surrounded Right generalization test E2GN2 has 5x the performance of GNN and MLP.Next, in Table 2 we test E2GN2's ability to scale to various numbers of agents without retraining. We use the agents trained in SMACv2 from Figure 6 for these tests. We do not include the MLP agent since it cannot scale up without retraining. Note that the method described in section 4.3 was essential to enable scaling between agents. For example, if we passed the GNN output to a final MLP to map to the discrete action space this wouldn't be scalable to more agents.The results for scalability confirm that both E2GN2 and GNN are able to maintain performance with larger numbers of agents. As the number of agents gradually increases the win rate does slowly decrease. E2GN2 does seem to have a slightly steeper loss in performance in the Zerg domain, which was also the most difficult domain for the initial training phase. We believe this is due to the higher complexity of the Zerg scenario, with certain unit types (banelings) that can explode to damage many other agents. 
conclusion and future work: 
 In this paper, we have demonstrated that E2GN2 merits strong consideration for many MARL applications. We addressed several important problems including the early exploration bias and how to apply it to complex action spaces. The sample-efficiency has dramatically improved, and there are now guarantees of generalization built into the network. There is still further work to do, such as handling the partial or incomplete symmetries. Furthermore this approach would need to be improved further to be applied to environments with angular momentum and accelerations, which are not included in these benchmarks. Currently, we expect that the improvement gained from E2GN2 in MARL will depend on the amount of rotational symmetry applicable in the observations. We believe that building on this work can be helpful to deploying MARL agents with a greater degree of trust (due to the guarantees). This could be helpful in many fields such as robotics, medicine, and power systems. In summation, the results of this paper provide a solid foundation upon which to build. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Yes, we mention that we selected hyperparameters following guidelines set by [26] , and included the hyperparameters used, the unspecified ones were simply the RLlib default parameters.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The charts all had confidence intervals (95 percent), computed using the standard seaborn package.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Yes, we believe this work adheres to the code of ethics. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss this in the conclusion. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We did not release data or code. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cited the code library used. RLlib uses an Apache license 2.0 Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Here we prove our assertion that the EGNN layers have biased outputs, and thus poor exploration.We start by showing that E [ϕ u (m ij )] ≈ 0. We do this by computing the expected value of a single neural network layer.An individual neural network layer is defined as y = W x + b where b is a N-by-1 vector, W is a N-by-M matrix and x is a M-by-1 vector. Note that similar to [28] we initialize the elements in W to be mutually independent and share the same distribution. x is initialized similarly. and x and W are independent. Thus for each element (subscript l ) of the equation, we have:E [y l ] = E [w l x l ] + E [b l ] = n l E [w l x l ] + E [b l ] = n l E [w l ] E [x l ] + E [b l ]We can expect w l and b l to be zero mean, since many commonly used initializers use either uniform, or normal distribution, with zero mean [28] . This results in:= n l * 0 * E [x l ] + 0 = 0Thus, we can assume that for an individual layer the output is 0, since the final layer of any mlp is 0,we can see that E [ϕ u (m ij )] ≈ 0 Next we show that E u L i ≈ E u 0 i .Recall that u L i is the Lth layer of ith node of the equivariant component of the network. We take the expectation over a sampling of the inputs u i and h i , treating each as a random variable. Note that u i and u j will have identical distributions, since each node will be sampled in the same manner. We start our proof by taking the expected value of the output:E u l+1 i = E   u l i + C j̸ =i u l i -u l j ϕ x (m ij )   = E u l i + C j̸ =i E u l i -u l j ϕ x (m ij ) = E u l i + C j̸ =i E u l i ϕ u (m ij ) -E u l j ϕ u (m ij ) ≈ E u l i + C j̸ =i E u 0 i E [ϕ u (m ij )] -E u 0 j E [ϕ u (m ij )] ≈ E u l i + C j̸ =i u l i 0 -u l j 0 = u l i E u l+1 i ≈ E u l i Thus, E u L i ... ≈ E u 1 i ≈ E u 0 iWhere we used the assumption thatE u l j ϕ u (m ij ) ≈ E u l j E [ϕ u (m ij )] The corollary: π i (a i |s) ∼ N (s eq i , σ) is a simple result from the fact that the equivariant action for an agent is the output of the coordinate embedding of the EGNN: u L i . Here we assume the standard deviation is a separately trained parameter and not a function of the EGNN. If this is the case, then the policy meanE [π(a i |s)] = E u L i ≈ E u 0 i = E [s eq i ] (by definition) A.2 Proof E2GN2 is UnbiasedHere we show that E2GN2 leads to unbiased early exploration. We begin with the equation for the E2GN2 layer:u l+1 i = u l i ϕ u2 (m ij ) + C j̸ =i u l i -u l j ϕ u (m ij ))Next we take the expected value of the output:E u l+1 i = E u l+1 i ϕ u2 (m ij ) + C j̸ =i E u l i ϕ u (m ij ) -E u l j ϕ u (m ij ) ≈ E u l+1 i E [ϕ u2 (m ij )] + C j̸ =i E u l i E [ϕ u (m ij )] -E u l j E [ϕ u (m ij )] ≈ E u l+1 i 0 + C j̸ =i E u l i 0 -E u l j 0 = 0Thus we see that for each layer E u l+1 i ≈ 0 and therefore E u L i ≈ 0. Similar to before, we used the assumption that Where we used the assumption thatE u l j ϕ u (m ij ) ≈ E u l j E [ϕ u (m ij )] and E u l j ϕ u2 (m ij ) ≈ E u l j E [ϕ u2 (m ij )]Similar to the first corollary the result: π i (a i |s) ∼ N (0, σ) follows mostly from the definition and the above proof. The corollary π i (a i |s) ∼ N (0, σ) is a simple result from the fact that the equivariant action for an agent is the output of the coordinate embedding of the EGNN: u L i . Here we assume the standard deviation is a separately trained parameter and not a function of the EGNN. If this is the case, then the policy mean E [π(a i |s)] = E u L i ≈ 0 (by definition)We follow the structure of [2] for how to show equivariance. Note that [2] showed that ϕ u (m ij ) will be invariant to E(n) transformations. This should still hold in our case as we make no modifications to m ij , similarly ϕ u2 (m ij ) will be invariant to translations. To show equivariance to rotations and reflections we show that applying a transformation T to the input, results in a tranformation to the output. That is: f (T u l i ) = T u l i where f (.) is the update equation of E2GN2:T u l i ϕ u2 (m i ) + C j̸ =i T u l i -T u l j ϕ u (m ij ) = T u l i ϕ u2 (m i ) + T C j̸ =i u l i -u l j ϕ u (m ij ) = T u l i ϕ u2 (m i ) + T C j̸ =i u l i -u l j ϕ u (m ij ) = T   u l i ϕ u2 (m i ) + C j̸ =i u l i -u l j ϕ u (m ij )   = T u l+1 iSince there are no modifications to E2GN2 update equation for h i , then E2GN2 should retain invariance to transformations from E(n). More precisely (using the fact that the initial m ij is invariant to translations from [2] ), if we translate the input u i by b:(u l i + b)ϕ u2 (m i ) + C j̸ =i (u l i + b) -(u l j + b) ϕ u (m ij ) = bϕ u2 (m i ) + u l i ϕ u2 (m i ) + C j̸ =i u l i -u l j ϕ u (m ij ) = bϕ u2 (m i ) + u l+1 i ̸ = b + u l+1 iThis shows that E2GN2 is not translation equivariant. The graph structure for MPE environments is set as a complete graph. The graph structure for SMACv2 is complete among the agent controlled units (or all of the friendly nodes), and each friendly node alse has an edge to each of the enemy controlled units. This was an attempt to model what we imagine a real world scenario to look like (agents see the enemies, communicate with allies, but do not see the enemies communication signals)Graph Inputs For MPE environments the input invariant feature for each node h 0 i is the id (pursuer, evader, or landmark). For SMACv2, h 0 i is made up of the features: health, shield unit type, and team. For each u 0 i is set as the position of node i. For MPE there is also a velocity feature, which we incoporate following the procedure described in [2] Graph Outputs for Value function Similar to the visual in section 4.3, the value function output comes from the invariant component of the agent's node of final layer of the EGNN/E2GN2. In other words the value function is: h L i where is L the final layer, and i is the agent in making the decision. Graph Outputs for Policy For MPE environments the actions are forces in a particular direction, thus the action output for agent i is u L i where L is the final layer. Section 4.3 discussed applying EGNN to more sophisticated action spaces. Specifically for SMACv2 the discrete action space has an equivariant component (movements) and an invariant component (shoot). The outputs for the equivariant component of EGNN/E2GN2 are continuous values, mapping these values to logits is not straightforward. Furthermore, they would need to be mapped onto the same distribution of logits being represented by the shoot commands. We solve this problem by having three distributions output by the GNN structure: the continuous/equivariant movement gaussian distribution, the discrete/invariant distribution, and a third distribution that determines whether we should move or shoot. After the RL sampling from each distribution is performed for the actions, then the three components of the action can be converted to the final action for either a mixed discrete-continuous action space or a discrete action space.Graph Neural Network details We used the same GNN as the EGNN paper: Specifically, we use equations 1 and 3, but update equation 1 to be m ij = ϕ(h i , h j , x i , x j ). For convenience we rewrite the GNN equations describing a single layer here:m ij = ϕ e h l i , h l j , m i = j̸ =i m ij , h l+1 i = ϕ h h l i , m iRecall that ϕ h and ϕ e are two layer MLPs. Note that the GNN only has one variable h i per node (compared to the two variables of h i and u i of EGNN/E2GN2). The features inputs are similar to an EGNN but with the two invariant/equivariant components concatenated together. The output for GNN will all come from the variable h as there is no need to distinguish between equivariant and invariant outputs (the action may be composed across several nodes, see above and figure 4 ).Hardware For training hardware, we trained the graph-structured networks using various GPUs. Many were trained on an A100, but that is certainly not necessary, they didn't need that much space. MLPs were trained on CPUs. We typically used 4 rollout workers across 4 CPU threads, so each training run used 5 CPU threads.Further Notes/details on SMACv2 SMACv2 ( [14] ) is a modification of the original SMAC ( [29] ).Since SMAC had deterministic initialization and static scenarios it turns out policies could memorize the solution. In each scenario, there is a pre-specified number of agents on each team (we use 5 agents for each team). In each scenario, the agents are randomly initialized in a specific formation.For this work, we use the initial position configuration termed "surrounded" (see fig 7 ). Finally, it is important to note that SMACv2 will randomly sample unit types from 3 different units (for each scenario). We use the same policy network for each agent but have the unit type as an input in the observation space, so each policy must learn to condition its behavior on the unit type.Below we include supplementary results we believe interesting to understanding further E2GN2's performance.Figure 8 : These results are using SMACv2 with the 'surrounded' initialization and 5 agents. Specifically, here we map the actions from the mixed discrete-continuous actions back to the default SMACv2 discrete action space. This is done by simply rounding the continuous actions to the nearest axis. These results are using the standard SMACv2 action input.Figure 9 : These results are using SMACv2 with the 'surrounded' initialization and 5 agents. Specifically, the observations here are partially observable. Note that E2GN2 still performs well with partial observability.Figure 10 : These results are using SMACv2 with the 'surrounded' initialization and 5 agents. Specifically, the observations here are partially observable. In all other experiments we recenter the states at 0 (standard in MARL). Here we did perform the recentering. Note how important this step is for EGNN. Without the centering of observations the EGNN is assuming rotational equivariance around the wrong center. Interestingly, this seems to have little impact on E2GN2's performance.Figure 11 : These were trained on SMACv2 using the 'surrounded' initialization with 5 agents. We were curious how an MLP would perform using the mixed continuous/discrete action space vs all discrete. Note that our E2GN2 is still able to outperform both. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.For terran, E2GN2 reaches 0.4 win rate after about 1.25 × 10 6 timesteps, GNN reaches 0.4 after 5 × 10 6 timesteps and MLP maxes out at 0.3 win rate after 10 × 10 6 timesteps. For Protoss, E2GN2 reaches 0.4 win rate after 2.5 × 10 6 timesteps, and GNN and MLP do not reach 0.4 within 10 × 10 6 timesteps. See figure6. 


Decision: This paper is focused on incorporating equivariance priors into MARL algorithms. This is done by evaluating the already-established EGNN architecture in MARL algorithms. The paper then shows that such an approach has limitations, mainly regarding the exploration strategy it 
induces and the discussion around adapting this idea to more complex action spaces. The paper empirically validates its claims on two sets of environments, MPE and SMACv2, showing that EGNN has a slow start (which the paper attributes to exploration issues without necessarily providing data to back that up) and that E2GN2 (the proposed method fixing the exploration issue) has a much better start. The paper also validates its claims around the generalization benefits of using such an inductive bias. Some of the criticisms around the number of independent runs used, and the lack of a specific baseline were addressed to some extent during the rebuttal phase.

The paper has issues, though. One issue that was brought up by pretty much every reviewer is clarity. Many reviewers think the paper is unclear on some of the design choices, and the descriptions provided are insufficient. Although open-sourcing the code is not a requirement for getting a paper accepted at NeurIPS, the lack of clarity added to the impossibility of open-sourcing the code is a major concern. Another major concern is overclaiming. The original version of the paper has very strong statements that are hard to understand where they came from (e.g., 10x improvement). The authors promise to fix the issue, and they provide an example of how they would do so in a specific instance, but it is hard to guarantee that those issues will be fully addressed. There have been many other discussions about relevant related work that should maybe be discussed, or the specific neural network architecture being used, or the scope of the experiments (e.g., partial observability or not, centralized critic or not), which is something that I did not consider so much because I think the paper does a good job defining its scope.

I do feel the pros outweigh the cons here, and I recommend the paper's acceptance. However, I strongly encourage the authors to improve the presentation and adjust their claims.

Reviews: review: paper_summary: paper_summary: The paper demonstrates for the first time the successful application of Equivariant Graph Neural Networks (EGNNs) to standard MARL benchmarks with complex action spaces. 
To address the Early Exploration Bias, the paper provides proof of the biases in standard EGNNs and the unbiased nature of E2GN2. In practice, it is achieved by adding an additional multi-layer perceptron (MLP), which helps to offset the bias from the previous layer's output, leading to actions that are not biased towards the agent's current position.
E2GN2 provides a method to handle complex and heterogeneous action spaces, which are common in MARL environments. It leverages the GNN's graph structure to output different components of the action space from different nodes in an equivariant manner, allowing for a more flexible and scalable approach to handling mixed discrete/continuous action spaces.
The paper shows that E2GN2 outperforms standard GNNs and MLPs by more than 10 times in sample efficiency on common MARL benchmarks like MPE and SMACv2. It also demonstrates greater final reward convergence and up to 5 times gain in generalization over standard GNNs.
strength_weakness: strength_weakness: see questions part
questions: questions: The paper introduces Exploration-enhanced Equivariant Graph Neural Networks (E2GN2), a novel architecture that significantly boosts sample efficiency and generalization in Multi-Agent Reinforcement Learning (MARL) by leveraging equivariance to symmetries and addressing early exploration biases. However, I still have some concerns:
1. For the complex action space issue, the detail of the method is not enough. How do you map the embedding to action space, such as an additional MLP layer? For the value head, does it share the same embedding with the policy head? Section 4.3 and Fig. 4 provide the high-level idea but ignore the details.
2. For the experiment part, this paper seems to use independent ppo as the baseline, which is not widely used in most MARL papers. Although the improvement is significant compared with ppo, it would be better to apply the E2GN2 to other MARL methods. 
3. Another issue is the input of the network. To use the EGNN, it seems that the states of all agents are required. However, we assume that only partial observation can be obtained in most MARL settings. For example, we cannot get the observation of opponents in SAMC. Please provide more details of the implementation.
limitations: limitations: see questions
paper_summary: paper_summary: This paper studies the setting of multi-agent reinforcement learning (MARL). The work tries to tackle the challenges of generalization and sample efficiency using inductive biases. In this case, the work proposes to use equivariant graph neural networks to model the policy and value function of a multi-agent actor critic. The work points out a bias in the output of such networks that harms exploration and proposes an architectural fix. It also proposes an architectural modification that will allow the usage of discrete actions in addition to continuous ones. The approach is evaluated on two different benchmarks and in some cases shows improved sample efficiency and final return compared to an MLP and graph neural network baseline.
strength_weakness: strength_weakness: Clarity
* In the abstract the main claims are 10x improvements in sample efficiency and final reward. This is just unnecessarily exaggerated and possibly misleading. I think the paper would benefit from a clear depiction of what is actually shown.
* The mathematical depiction of the model is unclear and needs to be improved
  * Notation is in places not defined or ill defined. E.g.
    * inputs to equations 1, 2, 3 are unclear to me
    * Some symbols are not defined, e.g. X and Y in section 3.2 The function $\phi_e$ is defined wrt one input in L137 but takes multiple inputs in Eq 1. 
  * A viasualization of equations 1, 2, 3 would improve readability. This could have been integrated with Figure 4 to not take up more space.
* Shaded regions in experiments are not explained. I'm assuming they are standard errors.
* In the examples for early exploration, a description of the MDP is required. It was unclear to me what states, actions and rewards are.
* Theorem 2 is missing a pointer to a proof.

Related work
* As mentioned before I am not an expert in the MARL field. The following might not be the most meaningful metric but I figured I'd point it out for completeness. In general, the number of cited papers in the manuscript is low compared to many other works. I'm not going to base my recommendation on this point because a small number of precise citations can be sufficient but I'm pointing this out in case reviewers more familiar with the literature have similar concerns. That is, I don't necessarily expect more citations to be added to raise my score.

Novelty
* The text argues that this work is the first work to successfully demonstrate multi-agent equivariant networks on complex tasks but point to some work whose tasks are not complex. It is unclear to me what the measure of complexity is and when a testbed would be sufficiently complex to make this claim about novelty correct. See Q1.

Experiments and Claims
* Several claims are overstated and need to be adjusted or clarified
  * The work claims that the method outperforms standard GNNs and MLPs by more than 10x sample efficiency. First, this is not validated in the experiments. I do not know what I should look at to get this 10x number. Further, it seems that this is not generally true since in some cases the benefits are marginal (see Figure 6 right) and needs to be rephrased. 
  *  In section 5.2, the text states that "equivariant networks have a clear improvement in sample efficiency over the MLP and GNN". This is not fully supported and needs to be concretized. In Figure 6 (right) the statement is incorrect.
* The experiments in Figure 6 are somewhat inconclusive. The final performance of E2GN2 is within variance of EGNN for all three experiments while also being conducted over only 5 seeds. For protoss and zerg, the whole training seems to be within variance, challenging the sample efficiency claim. Increasing statistical significance of these results would make the paper stronger.
* The experimental results in Table 1 and 2 lack a measure of variation. Given that other results seem to be within variance, these measures should be added.
* There is a lack of baselines that compares to prior work. It is, for instance, unclear to me why theoretical guarantees are useful if other approaches might be better. A comparison against the architectures from [19, 21] may have strengthened arguments for being to first to handle complex action-spaces.
* The paper states that the base models were chosen and special tricks from the literature were avoided L255 but that the proposed method "is compatible with most MARL actor-critic methods" L78. First, the paper does not demonstrate that the latter claim is true. Second, I think the paper would have been stronger if it had been shown that the method work *with* these tricks and stronger methods. In general, we have seen (at least in the single agent literature) that combining many advancements is quite beneficial [1 mine, 2 mine, 3 mine]. This would also solve the problem of the baselines being weak.

[1] Combining Improvements in Deep Reinforcement Learning. Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver. The Thirty-Second AAAI Conference
on Artificial Intelligence, 2017.  
[2] Bigger, Better, Faster: Human-level Atari with human-level efficiency. Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G Bellemare, Rishabh Agarwal, Pablo Samuel Castro. Proceedings of the 40th International Conference on Machine Learning, PMLR 202:30365-30380, 2023.   
[3] Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control. Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, Marek Cygan. arxiv eprint 2405.16158.  

I am happy to slightly raise my score if the language of the claims is adjusted, limitations are addressed and the experiments are made statistically significant. I am happy to raise my score more if additional baselines are included to strengthen claims and the generalizability to stronger approaches is demonstrated.
questions: questions: Q1: Can you elaborate what makes your environments more complex than the ones used by Van Der Pol et al. [19]?  
Q2: Can you elaborate on why previous work such as that by Van Der Pol et al. [19] does not observe the exploration bias?  
Q3: How does EGNN in the experiments do action prediction with discrete actions? Does it adapt the suggested solution?
limitations: limitations: The limitations of the work have not been addressed properly. I believe that this would be quite beneficial since the chosen experiments clearly have a structure that benefits from the imposed inductive bias. This does not mean that any claims that are being made can be true **in general**. It is unclear to me whether these methods still function in environments that are not hand-picked for the inductive bias at hand.
paper_summary: paper_summary: This paper studies an intricate issue, called “early exploration bias”, when applying EGNN, an GNN preserving Euclidean invariance/equivariance, to cooperative MARL that exhibits Euclidean symmetries. The paper reveals that, with randomly initialized weights, the output of the EGNN layers is not centered around zero, which can cause difficulties for cooperative MARL. The paper then proposes a convenient solution to this issue, with a cost of removing translation invariance/equivariance.
strength_weakness: strength_weakness: - Euclidean equivariance in deep MARL has been explored in several recent works, namely [20] and [Esp: Exploiting symmetry prior for multi-agent reinforcement learning]. This paper’s novelty against these recent works is limited. The most significant contribution seems to be limited to the pathology of EGNN, fixing it to overcome the “early exploration bias” issue. It is unclear whether this pathology and treatment is even relevant in other types of equivariant NNs (e.g., [20] uses the alternative of SEGNN). Moreover, the paper suggests its method can deal with discrete movement actions at the beginning of Section 4.3, which could be a significant novelty from prior work. However, it turns out that the paper bypasses this issue by changing the movement actions in the original SMACv2 from discrete to continuous.

- The method is motivated by the “early exploration bias”. Its cause and the proposed remedy which seems very specific to the proposed usage of EGNN and particularly specific to (MA)RL (e.g., similar issues could arise in ML for chemistry). The paper might benefit from an investigation and discussion of EGNN used in other domains/applications.

- The presentation of the theoretical results (the Theorems and Corollaries) and the proposed methods lacks clarity. Please refer to the Questions on this.

- This paper’s argument on translation equivariance is not that convincing. The proposed method (E2GN2) removes translation invariance/equivariance from EGNN and the paper argues (e.g., Lines 213-218) that translation invariance/equivariance is not a benefit for MARL. Lines 213-218 uses an example to argue against translation equivariance, while previous work (e.g. [20]) suggests translation invariance is indeed benefiting MARL, supported by both theoretical and empirical results. So, the paper could benefit from a revised approach that preserves  translation invariance, not equivariance.

- There are minor issues regarding the quality of writing, e.g.,
    - There is an unwanted line break between lines 110 and 111.
    - Line 122-123: $T_g$ should be $Y \to Y$ and $L_g$ be $X \to X$.
    - Theorem 1: the output vector should be $u_i^{l+1}$.
    - The format of the references is sloppy, e.g. [2, 19, 20, 22]
questions: questions: 1. In Figure 1, why is “All learnable equivariant functions constrained by data” not a subset of  “All learnable equivariant functions”?

2. The original EGNN paper [2] uses it for applications other than RL. Is EGNN’s property presented in Theorem 1 also problematic for the applications there? 

3. On the theoretical results:
    - 3a.  In Theorems 1 and 2, what is exactly approximated/hidden in the approximate equalities? 
    - 3b.  In Corollary 1.1 and 2.1, can you clarify the use of subscripts $i$ vs $k$?
    - 3c.  In Corollary 1.1 and 2.1, what’s the definition of $s_k^{eq}$?
    - 3d.  In Corollary 1.1 and 2.1, there seems to be a hidden assumption:  $a_i$ has the same dimensionality with $s_k^{eq}$, according to the displayed equation?

4. How exactly is a policy represented by EGNN?
    - 4a. How is the input graph defined/constructed? 
    - 4b. What is the dimension of $u^l_i$ for all l?
    - 4c. According to Section 3.3., the dimension of $h^l_i$ is $p$, how to map this $p$-dimensional vector $h_i$ to invariant action components (e.g., logits for the attack targets in SMACv2)? How to ensure such a mapping is equivariant/invariant?

5. The paper does not talk much about value function approximation. How exactly is the value function represented by EGNN?

6. On the experiments:
    - 6a. What specific GNNs are used in the experiments?
    - 6b. The experiments use a decentralized critic (IPPO?). Why not use a centralized critic (MAPPO), which is shown in previous work to perform better?
    - 6c. In Line 287-293, why is “permutation equivariance” relevant? What aspect(s) of the proposed method “allow for” the GNN to scale to larger numbers of agents, in a way that a vanilla GNN cannot?
limitations: limitations: The paper mentions several future work directions, including addressing partial observability, exploring other domains like robotics.
paper_summary: paper_summary: The paper applies Equivariant Graph Neural Networks (EGNN) to address the issue of low sample efficiency in multi-agent reinforcement learning (MARL). The authors demonstrate that in certain scenarios, EGNN outperforms Multi-Layer Perceptrons (MLP) and Graph Neural Networks (GNN). Key contributions include showcasing EGNN's ability to handle complex action spaces and proposing methods to enhance the exploration capabilities of agents within this framework. The paper presents a limited number of experimental results, attempting to illustrate the superiority of EGNN over some methods in improving exploration efficiency in MARL tasks.
strength_weakness: strength_weakness: 1. The paper lacks baselines that utilize symmetry [1,2,3].
2. There is a lack of entropy-based baselines to compare with for improving exploration capabilities [4].
3. Not all tasks can leverage symmetry, so the paper needs to specify the applicable scope of the proposed method.
4. The innovation is unclear, as extensive work on multi-agent systems is already based on EGNN. Applying EGNN to MARL appears more like trajectory prediction [3]. If the system is controlled solely by displacements in the x and y directions, it essentially remains trajectory prediction, similar to EqMotion [3].
5. The experimental descriptions are insufficient, lacking clarity on how features are designed, such as what constitutes equivariant and invariant features. Additionally, the use of symmetry is closely related to system state transitions, but the dynamics of these transitions are not introduced.
6. The approach to handling partial observability in multi-agent systems is unclear.
7. The reasoning for using zero-mean Gaussian policies is insufficiently explained. Moreover, if the goal is to enhance exploration, it is unclear why entropy-based methods are not used for comparison.
8. Due to the unclear method descriptions and the lack of open-source code, the reproducibility of the paper is poor.
9. Although EGNN can be applied in multi-dimensional spaces, the experiments are all conducted in two dimensions.

[1] Muglich D, Schroeder de Witt C, van der Pol E, et al. Equivariant networks for zero-shot coordination[J]. Advances in Neural Information Processing Systems, 2022, 35: 6410-6423.

[2] Yu X, Shi R, Feng P, et al. Leveraging Partial Symmetry for Multi-Agent Reinforcement Learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(16): 17583-17590.

[3] Xu C, Tan R T, Tan Y, et al. Eqmotion: Equivariant multi-agent motion prediction with invariant interaction reasoning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 1410-1420.

[4] Liu I J, Jain U, Yeh R A, et al. Cooperative exploration for multi-agent deep reinforcement learning[C]//International conference on machine learning. PMLR, 2021: 6826-6836.
questions: questions: 1. **Difference from Trajectory Prediction**: How does your application of EGNN in multi-agent reinforcement learning (MARL) fundamentally differ from trajectory prediction in multi-agent systems?
2. **Rationale for Zero-Mean Gaussian Policies**: Why is using zero-mean Gaussian policies considered best practice?
3. **Partial Observability Implementation**: How is partial observability handled in your experiment? Could you provide a detailed explanation of your method?
4. **Enhancing Exploration with Entropy-Based Methods**: Is it possible to incorporate entropy-based methods to enhance exploration in your EGNN framework?
limitations: limitations: The authors need to explore the scope of symmetry utilization to clearly articulate the limitations of their method.
paper_summary: paper_summary: this paper proposes an exploration-enhanced equivariant graph neural networks. The effective utilization of samples has been achieved in the experimental results. and It  is an improvement on the function approximation for MARL and thus is compatible with most MARL actor-critic methods.
strength_weakness: strength_weakness: No
questions: questions: No
limitations: limitations: No

Comments: comment: Thank you again for your review and also for increasing the score. It was very helpful to have specific actionable items to pursue in improving our paper. We believe this review helped improve the paper and strengthen the results.

Yes, we will update all of the claims to make them more precise and specific. In particular, the abstract, introduction and results section. We will update the comment in the results from  "equivariant networks have a clear improvement in sample efficiency over the MLP and GNN" to "equivariant networks demonstrate improved sample efficiency over the MLP and GNN in the protoss and terran SMACv2 environments".

After looking at the SMACv2 description in the paper, we agree that the description here is more useful for describing the complexities of this environment. We will add that to the description of SMACv2 in the paper. We will also update references to 'complex action space' or 'complex environments' to be more specific ie: 'mixed discrete-continuous action spaces', and/or 'environments with multi-tiered objectives, that require learning strategies to coordinate heterogenous agents and capabilities'. (Here by multi-tiered objectives we mean that in SMACv2 there are secondary-objectives of killing individual agents and not losing your own agents, vs the overall objectives of winning the scenario)

Thanks again for your time and effort in contributing to the AI/ML community.
Regarding the performance of E2GN2 on other applications. We were also curious and tested E2GN2 on the QM9 problem (from EGNN paper). On several QM9 tests, we saw little to no change in performance. We believe this is because the QM9 test predicts invariant quantities. Our modification to EGNN will primarily affect the equivariant component. We did not see an improvement on the n-body trajectory prediction problem either. E2GN2 does not help for trajectory prediction type problems because on those problems, biasing the output by the position is a desirable trait (ie it approximates kinematics, numerical solvers, etc). These were just preliminary tests.

Furthermore, it is important to consider the translation equivariance property of E2GN2 (again we will add the above discussion on translation equivariance to the appendix). In trajectory prediction translation equivariance is helpful (perhaps why E2GN2 does not improve the n-body task). In MARL translation equivariance is undesirable and harmful. Indeed simply being translation equivariant causes some bias to the network: as an example, for a policy network $\pi(.)$, it is translation equivariant if $\pi(x+b) = \pi(x)+b$, so translation equivariance does introduce a bias to the controls. We believe that is partly why E2GN2 greatly outperforms SEGNN on the Simple Tag problem: an SEGNN policy will have some measure of bias to the output due to the translation equivariance.  Of course, each of these architectures may have more or less bias, depending on each specific formulation. So one of these other architectures may perform better than EGNN, but they will likely have some measure of bias, as they were all designed for a different problem space than MARL/controls.  Although there may be other options, we selected EGNN because it had competitive results on supervised learning benchmarks, and it was much more computationally efficient. Our results and comparisons seem to indicate this was a fruitful decision.
As the discussion period is coming to a close we would like to thank you again for taking your time in reviewing this paper. We wanted to aggregate our recent comments into a single comment and add a couple of points.

To summarize responses regarding the innovations in the paper (see previous comment for more detail)

For i: We selected EGNN over other architectures due to computation speed. It takes 40 minutes to train vs SEGNN taking 4 hours for E2GN2 to train, this is why we stick with EGNN and then need to fix the exploration bias. We also note that E2GN2 outperformed both EGNN and SEGNN on MPE Tag. 

For ii: We believe this novelty is important, even if it is not overly complex. It was critical for using Equivaraint GNNs in a challenging MARL benchmark, and led to enormous improvements in SMAC Terran/protoss. Many AI innovations are simple in concept, but large in impact.

SMAC is a much more complex MARL problem due to heterogenous agents with different capabilities, action spaces, etc. It requires complex coordination and cooperation between agents. (see previous comment for more detail).
https://www.youtube.com/watch?v=5mUqtGir4e0 https://www.youtube.com/watch?v=VOdiYB3Ut8I (example videos)

iii: We believe a third novelty of this paper was applying EGNN to MARL. This did not exist in the literature prior, and EGNN is more feasible than SEGNN (or related formulations) for usage in MARL due to the quicker training time. This may be the key to broader usage of equivariant networks by the MARL community.
One quick note (re-novelty) we had much of our results before [20] was published, and did not find it until much of our paper was written.
Dear authors, 

I appreciate the thorough response and am glad you found some of my feedback useful and integrated it. 

"We will remove the language regarding "up to 10x improvement in sample efficiency" and replace it with simply "in many cases we see a significant improvement in sample efficiency over GNN and MLP networks". Perhaps that is worded better?"  
Yes, this wording is more appropriate. One could also talk about specific points in time or similar if that feels appropriate. I encourage the authors to make such adjustments where needed so that the claims accurately reflect what is being shown. Thus, my concerns with respect to phrasing of the claims have been partially addressed. I'm saying partially because this really needs to be done for all the claims where required. I'm going to be optimistic and assume that the authors would in fact do this for the final version of the paper even though they have not presented all the rewording here.

"Per your comment on standard errors, the updated plots (see global rebuttal) use standard error instead."  
I was not making a comment about which measure to choose but rather about the fact that you did not explain which one you were using. Bootstrapped confidence is just fine but the text should state that.

"On the number of seeds, we now have 10 seeds for each run."
These results are more convincing since the gap between methods is more pronounced. 

For environment complexity, the problem is that I might say: "Previous approaches can not land a rocket on a moon and thus the work did not use sufficiently complex environments." I think being specific about the environment differences in the paper would be beneficial to support the claim of environment complexity.   
""Our approach is the first to successfully demonstrate Equivariant GNNs for MARL on standard MARL benchmarks with complex action spaces."  
The text that the authors point to here does not talk about action or observation spaces. 
Something like the response to Q1 might be useful to specify what exactly the claim is. I'm going to assume that the authors can include a similar paragraph in the next version of the paper.

The new baseline results also put the work into a better perspective with respect to other approaches.

Overall, I think the changes improve the presented manuscript significantly and I think at this stage it would be fine to accept this paper. I'm updating my score to 6.
Thank you for your time in reviewing this paper.

We believe we addressed these concerns with our rebuttal and responses. In particular, we demonstrated that our approach greatly outperformed [20], with up to 2x the final win rate (on protoss and terran), more than 5x sample efficiency (on protoss and terran), and 4x faster learning time in hours (on MPE). We discussed why translation equivariance is a hindrance to MARL. Finally, we discussed that the MARL literature largely uses gaussian policies and how entropy based MARL exploration is neither state of the art nor common (especially for PPO).

Please see our larger official comment to these concerns (posted yesterday) for more details. In your comment you mention you will consider raising your score if we address these concerns, please let us know if there are any other concerns you may have. Thanks!
As the discussion period is coming to a close we would like to thank you for your time in the original review. To recap, we have added further discussion on the related works, added more seeds to our experiments, and added a related baseline [20] which our approach continues to outperform. We are eager to hear if our additions have addressed your concerns, and if there is anything else to add to this discussion. 

Although we added more seeds to the training charts, we had not yet updated the tables with more seeds (and standard errors). Here we present these:

Table 1 (with 10 training seeds)

| Environment | Network | Surrounded Left  (Training)      | Surrounded Right  (Testing)     | Surrounded All     (Testing)     |
| ----------- | ------- | ------------- | ------------ | ------------ |
| terran      | E2GN2   | .57±.01       | .55±.01      | .63±.01      |
| terran      | GNN     | .43±.02       | .07±.01      | .27±.02      |
| terran      | MLP     | .33±.02       | .12±.02      | .24 ±.02     |
| protoss     | E2GN2   | .59±.01       | .56±.02      | .57±.02      |
| protoss     | GNN     | .44±.02       | .08±.01      | .23±.01      |
| protoss     | MLP     | .42±.02       | .17±.02      | .27±.02      |
| zerg        | E2GN2   | .34±.02       | .3±.02       | .31±.02      |
| zerg        | GNN     | .37±.02       | .06±.01      | .18±.01      |
| zerg        | MLP     | .24±.02       | .04±.01      | .12±.01      |


Table 1 (with 10 training seeds)
|         |       | 5 Agents (Train) | 4 Agents (Test) | 6 Agents (Test) | 7 Agents (Test) | 8 Agents (Test) |
| ------- | ----- | ---------------- | --------------- | --------------- | --------------- | --------------- |
| Terran  | E2GN2 | .69+.02          | .65+.02         | .63+.02         | .62+.02         | .54+.04         |
|         | GNN   | .48+.02          | .45+.02         | .45+.02         | .40+.02         | .39+.02         |
| Protoss | E2GN2 | .62+.03          | .61+.02         | .59+.03         | .47+.04         | .37+.03         |
|         | GNN   | .38+.03          | .24+.02         | .35+.03         | .28+.03         | .2+.02          |
| Zerg    | E2GN2 | .36+.03          | .32+.03         | .31+.04         | .23+.01         | .18+.03         |
|         | GNN   | .33+.04          | .29+.03         | .31+.03         | .29+.03         | .27+.03         |


Furthermore, one question raised was whether the tips and tricks from [1a] would improve the performance of our approach. First, we clarify that we largely used the majority of their tips and tricks, especially relevant to performance (3 of 5 of their tricks were using a large training batch size, few numbers SGD of iterations, and small clip size. We used all of these).  Due to space in the rebuttal, we couldn't include this explicitly, but if you look at figure 1 from the rebuttal pdf and compare that to figure 5 from the paper, you may notice a difference between the performance of E2GN2 and EGNN. In Figure 1 of the rebuttal, all of the approaches used value normalization from [1a], in the paper that is not necessarily true. In fact, value normalizaiton was critical for E3AC learning at all. Thus we do observe that adding value normalization does indeed improve the results for MPE tag. We did not see an improvement with any other environment (and [1a] didn't see an improvement for other environments either).

Finally, in our blurb on limitations below we discussed how equivariance may be impartial or incomplete in the real world. We also want to add that our approach would need improvements to be applied to environments with angular momentum and mechanics, which are not included in these benchmarks (though to reiterate, these are the standard MARL benchmarks). Note that the total improvement from these approaches will likely depend on the amount of rotational symmetry applicable in the observations. Additionally, this approach is not directly compatible with environments using vision-based observations. 


We note the reviewer initially wrote they would raise the score if we adjusted the language, addressed limitations, and added more seeds. The reviewer also mentioned raising the score more if we added a new baseline to strengthen the claims and the generalizability to stronger approaches is demonstrated. We hope we addressed your concerns sufficiently, let us know what other concerns you may have.


1a the surprising effectiveness of Multi-agent PPO
I wanted to add a little bit more on why SMAC is such a big leap from MPE. MPE has homogenous units with the same capabilities and is primarily focused on cooperative navigation.

In SMAC, the units are heterogenous with different capabilities (different attack ranges, total health, and sometimes action spaces). The unit types are randomized at the beginning of the scenario. The actions include more components than simply movement (such as in MPE), agents can move and attack. Some units can heal instead of attack, some units simply explode, others can target multiple enemies. 
The goals are more complex as well. Instead of simply navigating cooperatively as in MPE, the agents must learn attack formations and strategies. Sometimes it may be optimal to sacrifice health or allies in the purpose of the greater strategic objective.

https://www.youtube.com/watch?v=5mUqtGir4e0 
https://www.youtube.com/watch?v=VOdiYB3Ut8I
(example videos)
Thank you for the response and for upgrading the score! We appreciate you taking the time to carefully consider this work.

To clarify are the significant concerns the two listed in your comment? (i and ii?) 

For i: Our greatest rationale for using EGNN (over SEGNN or other architectures) is the speed in computation (see the charts in the main rebuttal pdf) it takes 40 minutes to train vs SEGNN taking 4 hours for E2GN2 to train. This makes a big difference for MARL practitioners. This is why we stick with EGNN and then need to fix the exploration bias. We also note that while SEGNN outperformed EGNN in the original SEGNN paper, here we see that for MPE simple tag EGNN seems to outperform SEGNN. 


For ii: while this novelty may be fairly straightforward, we have demonstrated it was critical for MARL in scalability and performance on the much more complex SMAC environment (and opening the door for other MARL envs with discrete or mixed action spaces and both equivariant and invariant components). There are many novelties (such as the resnet paper with their skip connections) that were simple in concept, but very important for improving performance, as we see in this paper

iii: We believe a third novelty of this paper was applying EGNN to MARL. This did not exist in the literature prior, and EGNN is more feasible than SEGNN (or related formulations) for usage in MARL due to the quicker training time (in hours). This may be the key to broader usage of equivariant networks by the MARL community. 

One quick note (re novelty) we had much of our results before [20] was published, and did not find it until much of our paper was written.
3 continued) Note also that the paper in question [1a] regarding entropy-based exploration does not outperform the benchmarks on dense-rewards, it only outperforms the benchmarks on sparse-reward environments. All of our environments (the standard benchmarks in MARL) have dense rewards, which seems to see little to no improvement from their approach. Furthermore, they do not show how to apply it to actor-critic algorithms such as PPO (what we use here and is one of the best/fastest commonly used algorithms for MARL [2a, rllib])

Entropy-based exploration seems rather niche and has not become a mainstay or mainstream, especially not in MARL. Most of the state of the art papers and libraries use gaussian parameterized policies [2a, rllib, stablebaselines, etc].


1a Cooperative Exploration for Multi-Agent Deep Reinforcement Learning
2a The surprising effectiveness of multi-agent PPO
I would raise my rating from 3 to 4, considering the authors have made faithful efforts (especially the additional experiments) to address my comments.

However, I am still inclined to recommend a reject, because most of my significant concerns are not yet addressed:

The paper made two contributions: (i) correcting EGNN’s idiosyncrasy that causes “early-exploration bias” (sections 4.1 and 4.2)  and (ii) dealing with “complex action spaces” (section 4.3).

- For (i), the paper does not justify why we should stick with EGNN.  Switching to a better equivariant architecture should solve the problem. Afterall, the choice of architecture is just a hyperparameter on top of prior work like [20].  The EGNN’s idiosyncrasy is totally independent of MARL, which is why I encouraged the authors to investigate EGNN in other applications/domains.

- For (ii), it is a separate contribution from (i), as the “three-distribution” technique can be applied to architectures other than EGNN as well. It seems a valid novelty (prior work like [20] does not address this well), but it seems too incremental to justify an accept.

Besides, the paper should benefit from a more clear and rigorous presentation of its theoretical results and method description.
Thanks for responding!


1. Note that our paper was written contemporary to [20]. We had much of our results before they published. However, we did add an empirical comparison with [20] and note that our approach E2GN2 (see the pdf in the common rebuttal) greatly outperforms E3AC [20] on all of the SMAC benchmarks. On terran and protoss in particular, we more than double their final win rate (~0.6 vs ~0.3), and have much faster learning: ie E2GN2 learns a win rate of 0.3 in near 1e6 time steps, while E3AC [20] takes 1e7 time steps to learn that win rate. On MPE [20] is much slower, taking 4 hours to train. We believe our improvements are significant (due to using EGNN, E2GN2 and section 4.3) and should be shared with the community. 

  Much of the analysis in [20] is only a small extension from prior analysis in Van der pol's work. Van der pol established the theory for adding equivariance to multi-agent MDPs (albiet very simple grid world environments), the work in 20 did not add too much to this in terms of theory. Since van der pol had established this theory we chose not to focus on it as it would not be novel. Instead we focused on more practical applications and improvements.

  Regarding SEGNN, it is not necessarily an improvement over EGNN as it is very slow.  We initially decided to use EGNN specifically because SEGNN and related networks (ie those using spherical harmonics, wigner-D matrices, etc) tend to be slow. Looking at the original SEGNN paper it is 10x slower than EGNN. MARL experiments are already too slow, so we did not want to exacerbate this problem. In our global response we did observe that this results in much slower training times for the results from [20]

  Note that [20] was not able to get competitive results on SMAC. We were able to greatly improve learning on SMAC due to our approach described in section 4.3. Additionally, this solution in section 4.3 is not to only change the movement actions to be continuous, it is to decompose the discrete action outputs such that each node for each enemy is outputting one component of that action. This is crucial for retaining the GNN locality structure and allowing us to scale agents without retraining. 



2. It is true that E2GN2 loses translation equivariance guarantees. However it retains translation invariance guarantes* see below. For MARL policies translation invariance is useful [see 20] but translation equivariance is harmful.  We certainly observed that in practice translation equivariance was harmful (for one our results demonstrate E2GN2 out performs EGNN). As a simple example. if you have an agent at the position (0,-1) Perhaps the optimal action for agent 1 is to move up: (0,1). Now shift this down by 100, so now this agent's position is at (0,-100). A translation equivariant network will shift the action by -100, leading to the action of (0,-99). This means the agent will always move down, which is not optimal! This is not desirable for many MARL environments. When testing in MPE and SMAC, this may not cause a large problem, as the inputs are often normalized to be near (-1 ,1), but it can still potentially affect performance.

* we can add a full proof to the appendix, on  E2GN2 being translation invariant (just not equivariant), but briefly, we did not modify the equation for computing $h_i$ and are still using the output of $h_i$ for the invariant components. 

If we translate the input by $b$, that will translate the output by $\phi_u(m_{ij}) b$ ie   $u_i^l  + \phi_u(m_{ij}) b = u_i^{l-1} + b$ then the output of $h_i^{l+1}$ is translation invariant if  $|| u_i  -u_j  || $ is translation invariant, and $|| u_i + \phi_u(m_{ij}) b  - (u_j^{l} + \phi_u(m_{ij}) b ) || $ = $|| u_i  -u_j  || $ so we retain translation invariance.

3. For the citation concerns, we have mentioned we will add the citations you mentioned here (as well as others, some reviewers mentioned). We will add the content from these reviews going more in-depth on the differences between our on [19,20,21] (similar to the global rebuttal).  We are not certain we understand the rest of this comment. We were not seeking to change the exploration structure of the MARL learning problem. We were seeking to ameliorate a problem we noted in the EGNN structure (that of bias). We presented an in depth analysis of this bias and compared the EGNN vs E2GN2 across a variety of benchmarks (figure 3,5,6) demonstrating that the exploration bias of EGNN caused worse performance. 


  There are indeed many methods to attempt to improve a RL algorithm's exploration performance (such as the many papers on curiosity driven exploration). However, our focus was on modifying EGNN to improve common practices in MARL environments/bechmarks (stable baselines, rllib, etc). To that end, we focused on using the common gaussian parameterization of the policy (see our previous comment, Q2 for those citations). EGNN's biased will likely harm any biased exploration.
After carefully reading and reconsidering the manuscript and the authors' responses to my initial comments, I maintain that the paper requires significant improvements in several key areas:

1.Comparison with Reference [20]: Both this manuscript and reference [20] utilize equivariant neural networks for multi-agent systems. However, reference [20] exhibits greater theoretical depth and technical maturity. Specifically, [20] more thoroughly addresses the symmetry issues in multi-agent systems and utilizes SEGNN, which is more advanced than the EGNN employed in this paper. Consequently, the contributions in terms of innovation and technical depth appear insufficient.

2.Limitations of EGNN: The paper highlights deficiencies in EGNN's exploration performance, a valuable observation. However, the modification proposed in the paper results in the loss of translational equivariance properties. The current treatment and analysis do not adequately resolve or mitigate this issue.

3.Lack of In-depth Analysis on Exploration Performance: Although the paper discusses exploration performance issues, it lacks a thorough comparison with existing entropy-based methods. Such comparisons are essential for a comprehensive evaluation of the method's efficacy presented in the paper. Additionally, the summary of related work is not sufficiently comprehensive, failing to highlight the distinctions and advantages of this research over existing studies.

Based on these considerations, I maintain my initial rating. Should the authors address these issues, I will consider raising my score.
For E3AC, we used the model and observation processor from the E3AC github repo, so no, E3AC was not using the three-distribution technique we introduced here (since that was part of the novelty of our work). They did not have an observation processor for SMAC in their git repo. We wrote the SMAC observation processor by modifying their MPE observation processor according to the details in their appendix. Note that since E3AC was unable to resolve the difficulty with the SMAC action space (using SEGNN), they opted to use an MLP for the policy and an SEGNN for the value function. 

The third distribution is invariant, the logits are output from $h_i^L$ for agent $i$ (ie the invariant part of the output node corresponding to that agent)

 https://github.com/dchen48/E3AC
One other note regarding point 2
Since we used parameter sharing [ ie 2a], all of our agents were using the same critic. Since we had fully observable environments, the critic had all observations as an input. So using IPPO with parameter sharing and full observability is a fairly close approximation.

2a https://arxiv.org/abs/2005.13625
Thanks for the reminder. Here are the responses to those points. 




3a. See lines 457 and 469 (I tried copying it here, but the latex kept breaking)

6b That is a good point to bring up. For clarity, the difference between MAPPO and PPO is that MAPPO uses a centralized critic that takes in the observations and actions of all agents. We believe E2GN2 would improve MAPPO, and we hope to eventually apply this to MAPPO. We use independent PPO because 1) MAPPO performance can vary depending design of the centralized critic [see figure 4 in 1a].  2) For many cases IPPO is comparable to MAPPO [1a]   Since we used parameter sharing [2a], all of our agents were using the same critic. Since we had fully observable environments, the critic had all observations as an input. So using IPPO with parameter sharing and full observability is a fairly close approximation. 3) We wanted to use existing commonly used public libraries to increase reproducibility and utility to the community. We selected RLLIB since it seemed to be a popular benchmark, RLLIB does not have MAPPO built-in, it must be added on your own and implementations can vary (reducing reproducibility) 

One more clarification on point 4 of the weaknesses section. E2GN2 is still translation invariant (just not equivariant). We can add a full proof to the appendix, but briefly, we did not modify the equation for computing $h_i$ and are still using the output of $h_i$ for the invariant components. 

If we translate the input by $b$, that will translate the output by $\phi_u(m_{ij}) b$ ie   $u_i^l  + \phi_u(m_{ij}) b = u_i^{l-1} + b$ then the output of $h_i^{l+1}$ is translation invariant if  $|| u_i  -u_j  || $ is translation invariant, and $|| u_i + \phi_u(m_{ij}) b  - (u_j^{l} + \phi_u(m_{ij}) b ) || $ = $|| u_i  -u_j  || $ so we retain translation invariance.

1a https://arxiv.org/abs/2103.01955
2a https://arxiv.org/abs/2005.13625
In your added experiments, did you also use this three-distribution technique for E3AC?
Also, is the third distribution (move or shoot) equivariant/invariant?
Thanks for your response.
It seems that Questions 3a and 6b was not included in your rebuttal, if you would still like to provide a response. Thanks.
(continuing discussion on point 1 under weaknesses)

To further clarify, the issue with the discrete action space was not necessarily maintaining exact continuous equivariance (we will clarify this in 4.3). The problem is that the  SMACv2 action space has an equivariant component (movements) and an invariant component (shoot). The outputs for the equivariant component of EGNN/E2GN2 are continuous values, mapping these values to logits is not straightforward. Furthermore, they would need to be mapped onto the same distribution of logits being represented by the shoot commands. We solve this problem by having three distributions output by the GNN structure: the continuous/equivariant movement gaussian distribution, the discrete/invariant distribution, and a third distribution that determines whether we should move or shoot (described 290-292). After the RL sampling from each distribution is performed for the actions, then the three components of the action can be converted to the final action for either a mixed discrete-continuous action space or a discrete action space.