GradPS: Resolving Futile Neurons in Parameter Sharing Network for Multi-Agent Reinforcement Learning
Authors: Haoyuan Qin
Zhengzhu Liu
Chenxing Lin
Chennan Ma
Songzhu Mei
Siqi Shen
Cheng Wang
Venue: ICML 2025 poster (2025)
Keywords: multi agent reinforcement learning; parameter sharing; gradient conflict
Primary area: reinforcement_learning->multiagent
TL;DR: We studied the futile neurons in multi-agent reinforcement learning and proposed a new parameter sharing method for resolving the futile neurons.

Abstract: Parameter-sharing (PS) techniques have been widely adopted in cooperative Multi-Agent Reinforcement Learning (MARL). In PS, all the agents share a policy network with identical parameters, which enjoys good sample efficiency. However, PS could lead to homogeneous policies that limit MARL performance. We tackle this problem from the angle of gradient conflict among agents. We find that the existence of futile neurons whose update is canceled out by gradient conflicts among agents leads to poor learning efficiency and diversity. To address this deficiency, we propose GradPS, a gradient-based PS method. It dynamically creates multiple clones for each futile neuron. For each clone, a group of agents with low gradient-conflict shares the neuron's parameters.
Our method can enjoy good sample efficiency by sharing the gradients among agents of the same clone neuron. Moreover, it can encourage diverse behaviors through independently updating an exclusive clone neuron. Through extensive experiments, we show that GradPS can learn diverse policies with promising performance. The source code for GradPS is available in \url{https://github.com/xmu-rl-3dv/GradPS}.

GradPS: Resolving Futile Neurons in Parameter Sharing Network for Multi-Agent Reinforcement Learning
Haoyuan Qin 12*
Zhengzhu Liu 12*
Chenxing Lin 12 Chennan Ma 12 Songzhu Mei Siqi Shen 12
Cheng Wang" 12
Abstract
2020). MARL is challenging due to issues such non- stationary (Qiu et al., 2021), partial observability (Oliehoek & Amato, 2016), scalability, credit assignment (Foerster et al., 2018; Rashid et al., 2018), low sample efficiency, etc. Researchers have proposed various methods to address these issues. Many of them adopted the Centralized Training with Decentralized Execution (CTDE) paradigm (Olichoek et al., 2008), where agents are trained jointly with central information, but the agents execute their policies decentrally with the agent's local observation. the CTDE paradigm, Parameter Sharing (PS) (Rashid et al., 2018) is a widely adopted technique in MARL alleviate the scalability and low sample efficiency issues.
Parameter-sharing (PS) techniques have been widely adopted cooperative Multi-Agent Rein- forcement Learning (MARI). In PS, all the agents share a policy network with identical parameters, which enjoys good sample efficiency. However, PS could lead to homogeneous policies that limit MARL performance. We tackle this problem from the angle of gradient conflict among agents. We find that the existence of futile neurons whose up- date is canceled out by gradient conflicts among agents leads poor learning efficiency and di- propose versity. To address this deficiency, GradPS, gradient-based PS method. It dynami- cally creates multiple clones for each futile neu- ron. For each clone, a group of agents with low gradient-conflict shares the neuron's parameters. Our method can enjoy good sample efficiency by sharing the gradients among agents of the same clone neuron. Moreover, it can encourage di- verse behaviors through independently updating exclusive clone neuron. Through extensive experiments, we show that GradPS can learn di- verse policies with promising performance. The source code for GradPS is available in https: //github.. com/ xmu rl- 3dv//rraaeee
In PS, all agents share a neural network policy determined by the same set of parameters. PS is effective for MARL tasks where the agent behaviors are identical or similar. However, PS can lead to homogeneous policies, limiting the diversity agent behaviors and the overall capabilities of joint policies.
Existing studies PS can be classified into four categories: (1) full PS approaches where all agents share the same pa- rameters, the identity of an agent is used input to the policy to increase policy diversity (Rashid et al., 2018; Qiu et al., 2021; Wang et al., 2020), (2) group-based PS where agents are group (either dynamically or statically) into mul- tiple groups based the behaviors (e.g., trajectories), each group shares the same set of parameters (Christianos et al., 2021; Li et al., 2024b).
1. Introduction
(3) partial PS where the neural network of the policy is divided into shared part and an individual part (Li et al., 2021), (4) mask-based PS where the agent policy is determined by the combination a shared policy network and an agent-specific mask that can be static (Kim & Sung, 2023) or dynamic (Li al., 2024d), Our work is neuron-based PS approach that addresses the problems of PS from the angle of neuron gradient conflict among agents.
Many real-world tasks can be modeled as coopera- tive Multi-Agent Reinforcement Learning (MARL) prob- lems (Hernandez-Leal al., 2019), where multiple agents must collaborate achieve a common goal (Rashid al.,
Equul contribution FFujian Key Laboratory Sensing and Computing for Smart Cities, School of Informatics, Xiamen Uni- versity (XMU), China *Key Laboratory of Multimedia Trusted Perception and Efficient Computing. XMU, China "School of Computer, National University of Defense Technology, China. Correspondence to: Siqi Shen <siqishen& xmu.edu.en>.
Neuron gradient refers to the gradient of a loss with re- spect to a neuron. The gradient of the neuron represents the change in the knowledge learned by the agent. a PS network, agents could lead different knowledge which lead to gradient conflict. The neuron gradient conflict indicates
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARI
gradient directions for neuron among some agents are different. Regarding learning efficiency, gradient conflict is harmful, as it cancels out the learning progress of agents. Regarding policy diversity, gradient conflict could be ben- eficial, suggesting that agents have learned different skills and knowledge.
(T(u)) T2 = (O, x U,)* represents agent's local action- observation history. the global action-observattin history is denoted E T :- X ... X TN. on which it con-
ditions the joint policy π =< T1....: TT >. The joint policy π has a joint action-value function: Q (8t, Up) = B 1100 [ St, un], where Rs = Ei= 7'Te4s is the discounted return.
In this work, we study the gradient conflict in PS networks. We find the existence of futile neurons whose updates are canceling out significantly by gradient conflict. We find that these futile neurons can affect the expressiveness of the MARL policy network.
2.2. Gradient Conflict
Gradient conflict is a common phenomenon in multi-task learning (MTL), where multiple different but related tasks are jointly trained by sharing a model (Caruana, 1997). One implementation of MTL is to jointly train a network for all tasks, with the goal of discovering and leveraging relation- ships between tasks.
We propose a method, Gradient- based Parameter Shar- ing (GradPS), to resolve futile neurons the PS network. GradPS records the gradient conflicts between agents for the last T interval during training. For each futile neuron, GradPS dynamically creates K clones for the neuron. For each clone, group of agents with low gradient conflict shares the parameters of the clone. Its parameters are up- dated according to the gradient of the agents in the group. A group can consist of multiple agents, and their gradi- ents (knowledge) with respect to the neuron can be shared among the group. A group can consist only of few agents, which helps the agents to develop their specialized skills. As agent policies evolve, cloned neurons assigned to groups may once again become futile, while previously conflicting agents may no longer exhibit conflicts. To adapt to these changes, we re-evaluate inter-agent conflicts and restore a subset of cloned neurons with minimal conflicts to their original state, enabling dynamic regrouping process.
Learning multiple tasks may result in worse overall perfor- mance and data efficiency. One reason for this is gradient conflict where the gradients of multiple tasks diverge, such that the update for one task negatively affects the other. Re- searchers (Yu et al., 2020) consider the gradient conflict for vector, which is defined as follows.
Definition 1 (Vector Conflicting Gradients (Yu et al., 2020)). For vector, the gradients gi and 9j (i & for tasks and j are said to be conflicting with each other if cos øij < O, where Øig is the angle between gi and g3.
Conflicting gradients pose a challenge to optimizing the multi-task objective, as different gradients of individual tasks may impede the learning of knowledge.
Through extensive experiments on multiple benchmarks, we show that GradPS performs better than state-of-the-art PS methods. GradPS can learn diverse policies through utilizing the gradient conflict information.
Here, vector gradient conflict refers the conflict between gradient vectors composed of neuron gradients across dif- ferent tasks in neural network. In contrast, the neuron gradient conflict proposed in this paper specifically pertains to conflicts in gradient values for a particular neuron among different agents within a PS network.
2. Background
2.1. Dec-POMDPss
3. Related Work
We consider Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) (Olichock & Am- ato, 2016) in modeling cooperative multi-agent re- inforcement learning (MARL) scenarios. A Dec- POMDP can be formally described by the tuple G = (S, ( Prr, {O.). (or) N),),, where N rep- resents the set of agents, S is a finite set of states, and 24 is the set of actions available to agent i. At time step each agent i chooses an action uf E UG forming a joint action u' E N = Ui x ... x UN. This leads to transition to a new state sh+1 ~ P(-|s*, (')) and a joint reward r'. In consideration of partial observability, each agent can only access an individual observation of E O,, which is drawn from of (-(ss"). 7 denotes the dis- counting factor. Each agent acts base on individual policy
3.1. Parameter Sharing (PS)
Multi-Agent PS Parameter sharing has been widely used in MARL algorithms due to its simplicity and high sample efficiency. However, it could lead to homogeneous agent behaviors. ROMA (Wang et al., 2020) adopts a full PS approach, where agent behaviors are encoded into roles rep- resented by a hidden vector. Agent policies are sampled from the hidden vector. SePS (Christianos et al., 2021) adopts a group- based PS approach by creates multiple sets of parameters each shared by group of agents. Agents are clustered into groups based on their behaviors (trajectories) in the beginning of training. SNP (Kim & Sung, 2023) em- ploys a partial PS approach. It creates a big neural network, and then each agent builds its policy based on part of the
2

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARI
network through network pruning. AdaPS (Li al., 2024b) combines SNP and SePS by proposing cluster-based par- tial PS approach. Kaleidoscope (Li et al., 2024d) adopts a basee PS approach, an agent policy is determined on both a full PS network and a mask, which is learned dynamically for each agent.
Agant Agent Agent
Aspint Agunt:
Gradinnt
Gradine
100k
200K
JO9K
400K
509K
100K
200K
590K
400K
590K
Multi-Task PS Parameter sharing is also widely used in multi-task learning (MTL), which learns a single model for multiple different tasks. By PS among different tasks, MTL methods can learn more efficiently with overall smaller model size compared to learning with separate mod- els. PCGrad (Yu et al., 2020) found that parameter sharing of different tasks in MTL produces gradient conflicts and hurt performance. To avoid gradient conflicts, CAGrad (Yu al., 2020) seeks an update vector that maximizes the worst local improvement of any target within the average gradi- ent neighborhood, optimizing the minimum reduction rate of any specific task loss. Gradient Vaccine (Wang et al., 2021c) uses task relevance to set gradient similarity goals and adaptively aligns task gradients to achieve these goals. GradNorm (Chen et al., 2018) balances training deep multi-task models by dynamically adjusting the gradient magnitude among tasks. Recon (Guangyuan al., 2023) selects the layers with high conflict scores and turns them into task- specific layers.
Environmental Stape
Environmental Steps
Figure I: Gradients of neuron for different agents: VDN (Left) and QMIX (Right) in the SMAC 3m environment.
Prodator
Pray
Shared knowlediee
Individual knoweedge
Figure 2: Two agents collaborate to capture prey a predator-prey environment.
the parameters of the agent policy are shared among all agents. The observation of each agent is concatenated with its agent identity to increase policy diversity. The neural network of the PS agent network consists of two fully-connected layers and GRU layer. We analyze the neurons in the fully connected layers of the agent network.
3.2. Neuron Learning Efficiency
Redo (Sokar et al., 2023) identifies the existence of dormant neurons, whose activation score is low during the train- ing procedure. The dormant neurons lead to poor learning efficiency. Reset (Igl et al., 2021; Nikishin et al., 2022) improves the plasticity of neural networks through peri- odic resets of the weights of neurons in the last layer of neural network. (Dohare et al., 2024) proposes continue backpropagation algorithm which periodically re-initiates some less-used neurons. Reborn (Qin et al., 2024) perturbs parameters of dormant neurons in MARL mixing network.
4.1. Neuron Gradient Conflict Exists in MARL
In this section, we study the gradient of the temporal differ- ence loss with respect to neurons for different agents with the VDN and QMIX methods.
Definition 2 (Neuron Gradient). During a target network update period, the accumulate sum of the gradients gener- ated by the observation data of agent i on neuron n after backpropagation through loss is the neuron gradient Φs-
Our work analyzes the learning efficiency of MARL neu- rons based on gradient among different agents. Moreover, we identify the existence of futile neurons, special type of neurons whose update are canceled mostly by gradient conflicts.
In Figure/ 1, the gradient of two neurons for the SMAC 3m environment with the VDN and the QMIX method are plotted the left and the right part of the graph, respectively. Different agents could have different gradients, which could negative or positive. The aggregated gradient for the neuron (depicted as the sum) could become zero due to gradient conflict. For Figure 1, the RMSProp optimizer is used, we show in Appendix Figure 1 that gradient conflicts exist for the SGD and Adam optimizer too.
4. The Futile Neuron Phenomenon in MARL
In this section, we study the gradient conflict the Parame- ter Sharing (PS) network for MARL. We find the existence of gradient conflicts in PS, identify futile neurons whose up- dates affected significantly by the gradient neurons, and find that futile neurons hurt MARL network expressiveness.
Gradient conflicts are common in multi-agent reinforcement learning. Even if two predators cooperate to hunt the same target, the learned knowledge may still conflict due to the
We study the most popular PS (Sunchag et al., 2018; Wang et al., 2021a: la; Rashid et al., 2018; Shen et al., 2022), where
3

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
5m Vs 6m
MMM2
1.0
8m
0.8
19m
0.6
%
bourge
3-
2000
200K
BOOK
SO0K
6.6M
1.0M
0.2
Figure 4: Neuron Gradient Efficiency homogencous
0.0
(Left) and heterogeneous (Right) SMAC environments.
Figure 3: Average Neuron Gradient Conflict of neurons for each agent: 6m_6m (Left) and MMM2 (Right).
rutte Neuron Pancentag
10
Ruste Bouren Papterttag
35
Schaka
to
different initial positions in the environment. As shown in
Figure 2, the agents need learn shared knowledge and
20
individual knowledge. Differences in individual knowledge may lead to conflicts in certain neurons. The conflicting neuron gradients cancel each other out, which slows down the learning process of neurons. We define Neuron Gradient Conflict as follows.
300K
200K
1.0M
Environmsntall
Ensfrormental
Figure Percentage of Futile Neurons in homogeneous (Left) and heterogeneous (Right) SMAC environments.
Definition (Neuron Gradient Conflict (NGC)). During a target network update interval T, for neuron n, the accumu- lated gradient of agent at the activation layer oo' and the gradient of agent j is of. The gradient conflict between agents i and j this neuron within the T interval is defined
of agents during MARL task training.
4.2. Futile Neurons in MARL
We study the learning efficiency neurons in terms the NGC among agents. It is defined as follows.
C = EO - T
107
1o
+DD
(1)
Definition 4 (Neuron Gradient Efficiency (NGE)). During the last T periods of the target network update interval, for neuron n, the accumulated gradient for agent of the update interval T is of The Neuron Efficiency for neuron n. is defined as follows:
We study the NGC C of all the neurons the first FC layer for interval 7' with QMIX. Figure 3 (left) shows the average value of the NGC for the 5m_vs_m vn__vs_6m environment. The value of cell (i of the table represents the average NGC among agent and j. The darker the color of the cell in the table, the greater the conflict between the agents. It can be seen that there are gradient conflicts in such an environment where agents belong to the same agent type. Moreover, we find that the aggregate NGC behaviors among different agents vary. The conflict among agent and is low as their behavior similar. The average NGC between agent ( and agent 4 is high the learned behaviors of them are different.
T
T
1
Σ
Σ
/
o
D
(2)
1=
NGE is a value among 0 and 1. To evaluate NGE in MARI,, we conduct experiments in homogeneous SMAC environ- ments with only one agent type. As depicted Figure (left), with the increase of training time, the neuron effi- ciency decrease. This finding suggests that with the increase of the training time, the gradient conflicts among different agents increase, which may lead to inefficient learning. In Figure 4 (left), different environments with an increasing number of agents are considered. There are and agents in the 3m and the 5m environments, respectively. We find that with increasing number of agents, the neuron efficiency decreases. This is due to the fact that more agents could lead to more gradient conflicts.
Figure 3 (right) shows the average NGC for the SMAC MMM2 environment, where agents have heterogeneous agent types. Agents and belong to marauder agents, agents
are marine agents, and agent 9 is medivac agents. The agent policies should be diverse to finish such task. It can be seen that the mean NGC among all the marine agents (agents 2 to 8) are small compared to the conflicts among other agent types. Agent 9 has large gradient conflict with most of the agents, as it is a healing agent rather than a combat agent (c.g.. marine). The NGC among agents can be used measure to distinguish the role/task
In Figure 4 (Right), three environments with the same num- ber (i.e., 9) of agents are shown. There are 1, 2, and 3 types of agents the 9z, 3s6z, and lc3s5z environments, respec- tively. We find that with the increase of agent type, the NGE

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARI.
the training process. When f1 = J2 (w = 1), there is basically no futile neuron, while in other scenarios, futile neurons gradually appear. The larger the discrepancy from 1, the earlier the appearance of futile neurons. More- over, we find that the time of the appearance futile neu- rons correlates with the time that the MSE of the Q values is close to convergence, as shown in Figure 6 (right), For example. at step around 0.65k, the futile neurons for the curve fi 0.5 suddenly increase. At the same time, the MSE for f2 fi x 0.5 converges, and the loss does not drop thereafter.
209
50
R Ft 3.0 = 72 0. F2
F3 1.4
Wenn
0.4K
0.8K
0.4K
0.8K
1.2K
14K
2,0K
Bevirsnmental Stops
Figure 6: (a) Percentage of futile neurons, (b) MSE of reconstructed Q value
From Figure 6, we find that the appearance of futile neurons prevents the further convergence of the MSE loss for the matrix game. The gradient conflicts generated by knowledge differences will lead the generation of futile neurons. is possible that the gradient conflicts for these futile neurons cancel out most of the learning process, which hurts the expressiveness of the MARL networks.
decrease. For some neurons, their NGE can be very low. Such neurons are refer to Futile Neurons defined follows. Definition 5 (Futile Neuron). A neuron is a futile neuron if it satisfies the following condition.
en < Oz
(3)
where 0 a 1 (i.e., a = 0.2). To ensure fairly display of futile neurons across different scenarios, we fixed to constant threshold in experiments. For practical imple- mentation, more flexible methods such median absolute deviation (MAD) can be employed to dynamically identify futile neurons.
5. Method
In this section, we propose the GradPS method address the futile neuron phenomenon in MARL. The overview of the GradPS algorithm is described in Algorithm 1, with schematic diagram shown in Figure 7. GradPS groups agents into K distinct groups for each futile neuron, and the parameters of group are updated independently to reduce gradient conflicts. Thus, the gradient conflict for futile neurons is alleviated. We describe the key operation of the algorithm in the following sections.
Figure 5 depicts the percentage of Futile Neurons for dif- ferent SMAC environments. It shows that the percentage increases with time. And such percentage is non-neglect- able. Figure 5 (left) shows that with an increasing agent count, the futile neuron percentage increases. Such per- centage increases with the number of agent types in Figure 5 (right).
5.1. Conflict-Based Grouping
The appearance of futile neurons indicates significant con- flicts in the agents during recent periods. Gradient conflicts Gradient conflicts correlate with diverse observations and actions of individual agents, which gradient descent meth- ods struggle to resolve effectively. The gradients of agents with larger conflicts are more likely to cancel each other out, so we group agents with smaller conflicts based on the conflict matrix.
4.3. Futile Neurons Could Hurt Network Expressiveness
The expressiveness of value factorization functions (Rashid et al., 2018; Son et al., 2019) is shown to significantly im- pact MARL performance. this section, we conduct exper- iments to study whether futile neurons hurt agent network expressiveness. We conduct experiments on multiple simple one-step matrix games. In these games, there are only two agents with only one time step, all agents shared a team reward without receiving separated rewards.
experimenn
For each target update interval, the neuron gradient effi- ciency of each neurons is recorded, and futile neurons identified. To reduce gradient conflicts among agents, we cluster agents into K groups based on their gradient conflict values accumulated over 7 periods, aiming to minimize the total gradient conflict within each group. The clustering is performed based on the average gradient conflict among agents.
We build the payoff matrix as a sum of two utility functions, and use VDN as the value factorization function, which can model the summation relationship. The payoff matrix is constructed as Q(u,, u2) fi fu11 + f2())). where uj and U2 the actions taken by agent and 2, respectively, and fi and f2 are two utility functions. fi (A) 0, (B) 1, (C) = 2, (D) = 3. f2 = w f1. where w E TR. Each matrix games differ by the value w. Please refer to all matrices in Appendix D.2.3.
Specifically, convert the conflict matrix into an affinity matrix and use the spectral clustering algorithm for grouping the agents. Please refer to Appendix C.2 and C.3 for details about the recording of gradients and the clustering.
Figure (left) shows the percentage of futile neurons during
5

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
Share Parameters
Flows
Gradient Flows
b.b
Qen (T, s)
Clone
Group
Restore
Mixing Network
Futile neurva
Clone newrons
Nornal neuron
V
Cluster
u!) ... On (r" )
Futile neuron
Agent 1
gent
Clone neuron
0²
Conflict Matrix
Figure 7: The schematic plot of GradPS. Each futile neurons is duplicated into K clones, each is shared by distinct group of agents. the difference among clones are small, they will restored into normal neuron.
Algorithm GradPS Algorithm
show in the experiment section that the cost is low.
1: Get the accumulated gradients for all the neurons.
2: Identify the Futile Neurons
5.3. Group Parameter Sharing Restoring
3: for Each futile neuron n do
As agent policies evolve, cloned neurons assigned to groups may once again become futile, while previously conflicting agents may no longer exhibit conflicts. To adapt to these changes, we re-evaluate inter-agent conflicts and restore a subset of cloned neurons with minimal conflicts to their original state, enabling dynamic regrouping process.
4: Grouping agents into K groups according to gradient conflicts (Sec. 5.1)
5: Makes K clones of the futile neuron n, and assign each clone to distinct group of agents. (Sec. 5.2)
6: end for
7: if Restore operation required then
8: Calculate the variance of each neuron
In each restoration period, with probability of times the percentage of futile neurons, we select the group of clones, whose parameter variance are smallest, for restoration. The restoration process changes the weights of each clone to the same values.
9: Restoring the clones to a normal neuron (Sec. 5.3)
end if
5.2. Group-Based Neuron Clone
It is possible simply the weights of the clones as the average weights of all the groups. However, such an operation could lead to forgetting learned knowledge among different groups. To this end, we add a regularization loss for parameter restoration and use separate SGD optimizer for gradual parameter restoration. The regularization loss defined follows.
After the grouping operation, for a futile neuron n in linear layer, its input weights W.n and bias bin are cloned K time. Each clone j is shared among the group j of agents, and the update of clone j is independent of other clones. Each agent belongs distinct group.
The output of clone j for the futile neuron is defined W + by,,- Each clone does not individually connect to the neurons in the next layer, their outputs are aggregated into F(CK W in x + 1/,,) as the input for the next layer, where the activation function.
K
Lreg
( - Win + 162 bin
(4)
1=1
j=
where Wi, and bin are average values among groups. After M updates by the SGD optimizer, when the group's 'su's param- eters converge, we merge the clones and restore them into normal neuron that is shared among all agents.
During training, the input weights W for the clone neuron in j are updated independently from other clones. As the grouping operation has made the gradient conflict in each group small, the update of W for the clone neuron enjoys a better learning efficiency than without sharing. Moreover, agents can learn diverse policies without gradient conflict within a clone neuron. Although the efficiency and diversity improvement comes at the cost of more neurons, we will
6. Empirical Evaluations
We show that GradPS performs better than the other param- eters sharing (PS) methods for the SMAC and the Predator- Prey benchmarks. GradPS can improve the performance
6

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARI
(a) 2c_ VS_vv__64440 __64g
(b) 27m 30m
(c) 3s52 15.52.s_333 vs_3s62
70
GradPS
60
GradPS
GradPS
60
FuPSHID
FueS
FuPSHiD
FuFS
30
PuPS
FUPS-1D
50
NoPS
SMP
40
NoPS
SNP
NoPS
SMP
Test Win
40
SePS
Kaleidoscope
Test Win
30
Kaleidoscope
SePs
20
SaPS
Kaleidoscope
30
Tost
20
10
10
10
0.4M
0.8M
1.2M
1.6M
2.0M
0.4M
0.8M
1.2M
1.6M
2.0M
0.4M
0.8M
1.2M
1.6M
2.0M
Environmental Stops
Environmental Stops
Environmental Stops
(d) 2c vs. 642g
(e) 27m vs. 30m
(f) 3s52 vs 3s6z
GradPS
GradPS
GradPS
Futile Neuron Percentage
FuPS
Fottle Neuron Percentage
70
FuPS
FuPS
FuPS+1D
FuPS-ID
40
PuPS-10
40
60
SePS
SePS
Kaleidoscope
50
Kaleidoscope
Kalsidoscipp
30
30
40
Futile Neuron
30-
20
20
10
2.0M
L.6M
2.0M
Environmental Steps
Environmental Steps
Environmental Steps
(g) 5m
MMM2
1c3s8z 1c3s9z
50
FUPS
80
FUPS
40
Rate
NoPS
NoPS
Kaleidascopa
Kaleldascope
30
Kaliidascope
Test
20
20
20
0.4M
0.8M
1.2M
1.6M
2.0M
0.4M
0.8
1,2M
1.6M
2.0M
0.4M
0.8
1,2M
1.6M
2.0M
Environmental Stops
Environmental Stops
Envirommenten Stops
Figure 8: GradPS performs better than other PS methods: (a-c) The test win rate the futile neuron percentage (d-f) of the 2c 2c_vs, ss__v__ee vs__4zg, 27m_vs. 77m_vs_30m vs_30m, 3s5z_ _________66__ 3s62, environment. (g-i) The test win rate of the 5m_vs_6m, MMM2, and 1c3s8z_v vis_vs____l__ 1c3s9z environments.
of MARL algorithms by reducing the percentage of futile neurons in PS networks. GradPS is able to learn diverse policies thanks to identifying hidden properties of agents based on gradient confficts, and GradPS works for multiple MARL agent networks. Please refer to Appendix D.3 for a detailed experimental setup and more experimental results.
experiment, some agents hold hidden property that they will receive the same reward for capturing hare as capturing stag. Such hidden property cannot be observed by any agents. We evaluate three environments of Predator-Prey small, medium, and large. Each of them consists of different numbers of agents with varying grid sizes.
Baselines and training. We compare GradPS with vari- parameter sharing methods including (1) Full Parame- ter Sharing (FuPS), (2) Full Parameter Sharing with index (FuPS+id), (3) Selective Parameter Sharing (SePS) (Chris- tianos et al., 2021). (4) Structured Network Pruning with parameter Sharing (SNP) (Kim Sung, 2023), (5) Kaleido- scope (Li et al., 2024d), (6) No Parameter Sharing (NoPS). We evaluate the performance of GradPS and all these meth- ods with the QMIX (Rashid et al., 2018) value factorization function. Detailed implementations and parameter configu- rations are available in Appendix D.1.
6.1. Environmental Setup
Environments. The StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) is a popular benchmark used extensively in MARL, where multiple ally units con- trolled by MARL algorithms aim to defeat enemy units controlled by the game's built-in AI. Predator-Pres simu- lates a grid world where multiple predators collaborate to capture preys, which consists of stag and hare. Capturing stag can lead to higher reward than capturing a hare, but it requires close collaboration between two predators. In this

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
(a) predator-prey small
(b) predator-prey medium
(c) predator-prey large
120
200
300
GradPs
Grudrs
GradPS
100
PuPS1 HD
PuPS
160
PuPS+ ID
PuPs
250
PUPS+ID
PUPS
SMP
SNP
SNP
80
NoPS
NoPS
200
NePS
Return
SepS
Return
120
Seps
Return
SePS
Kaindoscope
Kaleldorcape
Kaleidoscope
60
150
80
40
100
20
40
50
0.2M
0.4M
0.6M
0.8M
1.0M
0.2M
0.4M
0.6M
0.8M
1.0M
0.2M
0.4M
0.6M
0.8M
1.0M
Enviro Steps
Environmental Steps
Environmental Steps
Figure 9: GradPS performs better than other PS methods Predator- Prey: (a) Small, (b) Medium, (c) Large
(a) predator- pray large (pymar13)
(b) 1c3s82 vs 1c3592 (pymart3)
(c) 3s5z vs 3s6z (pymar13)
300
Grad
GradPS
PUPS
FuPS
250
PUPS ID
80
FuPS+ID
SNP
SNP
200
Tast Win Rata
60
Kateidoncope
Text Win Rate
Katoidescope
Raturn
60
150
100
GradPS
PuPS
20
50
SNP
RLPS-1D
Kaleidoscopo
0.4M
0.9M
1.2M
1.6M
2.0M
1.6M
3.2M
4.8M
6.4M
8.0M
1.2M
2.4M
3.6M
4.8M
6.0M
Environmental Steps
Environmental Steps
Environmental Steps
Figure 10: Performance of GradPS based on PyMARL3 with more environmental steps. (a) Predator-Prey Large, (b)
lc3s8z_vs vs vs_lcss_l,pp__ (c) 3s52_vs_Sat 3366
(a) predater-prey small
(b) predeter pray medium
(d) predetarggey large
ronments, GradPS performs the best among all the methods as well. Figure 9 depicts the results for the Predator- Prey en- vironments. As shown in the graph, GradPS performs better than other PS methods for the Predator Prey environments.
In addition, implemented GradPS based on PyMARL3 and extended the environmental steps, as shown in the Fig- ure 10. More results are presented in Appendix 7 and 8.
Figure 11: The average gradient conflicts among agents for the (a) small, (b) medium, and (c) large maps of the predator-prey benchmark. The darker the cell, the larger the gradient conflict. The agents with the hidden property in the small map are and 3, in the medium map are 1, 5, and 6, and in the large map are 0, 9, 10, 12, and 13. Agents can be divided into two groups, with smaller conflicts within each group. GradPS successfully distinguishes agents with the hidden properties through gradient conflicts.
6.3. GradPS Can Encourage Diverse Policies
In real-world scenarios, there may exist unobservable fea- tures that could significantly influence agent policies. For some agents (with hidden property) in predator-pis prey envi- ronments, capturing a hare can lead to the same reward as capturing a stag, After carefully inspecting the learned poli- cies, we find that agents with such hidden property learn to prefer capture hares.
Figure 9 shows the comparison with other PS methods. Al- though FuPS+ID can also learn slightly different behaviors, this becomes difficult as the number of agents increases.
6.2. GradPS Is Superior to Other PS Methods
The experimental results for various PS methods and v__2g,, the
GradPS are shown in Figure 8. For the _c_vs_ vs 64zg, vm_0m, the 3s57_vs, sa___s_3eee vs_3s6z environments, as is shown in Figure& 8 (a-c), GradPS achieves the best performance in terms of win rate thanks to its ability to reduce the futile neu- rons better than other PS methods, as is shown in Figure 8 (d-f). For the 5m vs. 6m, MMM2, 1c3s8z vs_
To analyze such a phenomenon, we depict the average gra- dient conflict among agents for the last update periods in Figure 11. We find that GradPS can distinguish agents with the hidden property through gradient conflicts and learn di- verse policies. Agents with the hidden property have a large
vs_lvs___l.. 1c3s9z envi-
8

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
(a) _m_vs vS_ 6m
(b) MMM2
(c) predator-prey medium
200
80
QPLEX
OPLEX-GraPP
80
QPLEX
QPLEK-GradP
OPLEX GradPS
QPLEX
DMIX
DMIX
160
DMIX
DHIX GradPS
DM GriD-GGradPS
DHIX-GradPP
RMIX
Test Win Rate
60
RMIX
RMIX
RMIX GradPs
BMIX-- GradPS
Return
RMIX Gradhs
40
20
20
0.4M
0.8M
1.2M
1.6M
2.0M
0.4M
0.8M
1.2M
1.6M
2.0M
0.2M
0.4M
0.6M
0.8M
1.0M
Environmental Steps
Environmental Stops
Environmental Steps
Figure 12: GradPS works for different agent networks (i.e., QPLEX, DMIX, RMIX): (a) 5m_ vm_6,, (b) MMM2, (c) Predator-Pre Prey Medium environment.
(4) 5m dm
(b) 5m
5m 6m
(d) 5m 6m
Douats (7-121
Goeaing (Cx2)
Boeaht (prd)
Goeaing (ext.1))
Tux Vin Ratt
60
80
Tust Win
0-
9.4M 0.8M
1,3M
1.001
2.0M
9.4H
0.8M4
1.2H
1.401
2.0M
0.4M
0.004
1.2M
1.0M
2.0M
0.4H
0.8M
1.2M
1.401
2.0M
Snvtrommentall Steps
Enviroamentall Strps
Environmmental Stops
Enviroamental Strpss
Figure 13: Ablation of different hyperparameters in GradPS. (a) the period of accumulating gradients T. (b) the number of groups K. (c) the restoration probability (d) the futile threshold a.
gradient conflict with agents without the property. GradPS learns different behaviors through the use of the grouping of cloned neurons. Further, we show that GradPS leads to less futile neurons in Appendix Figure (e-f).
We individually modify each hyperparameter, and the ex- perimental results indicate that appropriate hyperparameters can help better distinguish and utilize futile neurons, thereby improving learning efficiency.
The parameter a can be adaptively optimized rather than constrained by fixed thresholds. It is not recommended to set a large p because gradual parameter recovery requires sufficient time to prevent rapid performance degradation. In future work, We will explore whether the aggregate K of all the neurons can represent agent heterogeneity.
6.4. GradPS Works for Multiple PS Agent Networks
In this section, we investigate the applicability of GradPS by validating GradPS's ability enhance performance across various value factorization algorithms (QPLEX, DMIX, RMIX) in different experimental scenarios $mm_s_66m MMM2, Predator-Prey Medium).
7. Conclusion
The agent architectures of QPLEX, DMIX, and RMIX fea- ture a separate value head, distributional function, and risk- sensitive function, respectively. They are different from the architecture of agents used in QMIX. According to the experimental results presented in Figures 12, GradPS can improve the performance of multiple PS agent architectures and effectively reduce gradient conflicts between agents.
In this work, we study the gradient conflict among agents for the Parameter Sharing (PS) networks of Multi-Agent Re- inforcement Learning (MARL). We identify the existence of futile neurons whose update is canceled out by gradient conflicts. We show that such neurons hurt the policy ex- pressiveness. We propose GradPS, simple yet effective gradient-based PS method that resolves futile neuron issues. It dynamically creates multiple independent clones for each futile neuron. Each clone is shared among group of agents with low gradient conflicts. GradPS can learn diverse be- haviors through multiple clones to avoid gradient conflict, and enjoys good sample efficiency by sharing the gradients among agents of the same clone neuron. Through extensive experiments, we show that GradPS is promising.
6.5. Parameter Sensitivity Analysis
The ablation study in Figure 13 illustrates the impact of dif- ferent hyperparameter settings in QMIX-GradP focusing on the ablation of the futile threshold o, the probability of the restoration p. the number of group K, and the group interval T. The default configuration of QMIX-GradPS is 0.2,7 T 20, K 3, and p 0.05.
9

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARI
Acknowledgements
Igl, M., Farquhar, G., Luketina, J., Bochmer, W., and Whiteson, S. Transient non-stationarity and gener- alísation in deep reinforcement learning. In ICLR, 2021. URL https: /tpppp/peeeeiieeeeeeeeeeeeeeeeee net. hertforum? i d==unn Fv4q5by..
This work was partially supported by the Fundamen- tal Research Funds for the Central Universities (No. 20720230033), by Xiaomi Young Talents Program. We would like thank the anonymous reviewers for their valu- able suggestions.
Kim, W. and Sung. Y. Parameter sharing with network prun- ing for scalable multi-agent deep reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pp. 1942- 1950, 2023.
Impact Statement
This paper presents work whose goal to advance the field of Machine Learning. Our work has many potential cietal consequences, none of which must be specifically highlighted here.
Li, C., Wu, C., Wang. T., Yang. J., Zhao, Q., and Zhang, C. Celebrating diversity in shared multi-agent reinforcement learning. In NeurIPS, 2021.
References
Li, C., Deng, Z., Lin, C., Chen, W., Fu, Y., Liu, W., Wen, C., Wang, C., and Shen, S. Dof: diffusion factorization framework for offline multi-agent reinforcement learning. In ICLR, 2024a.
Caruana, R. Multitask learning. Machine learning, 28: 41-75, 1997.
Chen, Z., Badrinarayanan, V., Lee, C.-Y., and Rabinovich, A. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML, pp. 794-
Li, D., Lou, N., Zhang, B., Xu, Z., and Fan, G. Adaptive parameter sharing for multi multi-agent reinforcement learning. ICASSP, pp. 6035-6039. IEEF, 2024b.
803. PMLR, 2018.
Li, X., Liu, Z.., Chen, S., and Zhang, J. Individual contri- butions as intrinsic exploration scaffolds for multi-agent reinforcement learning, arXiv preprint arXiv:2405. 181 10, 2024c.
Christianos, F., Papoudakis, G., Rahman, M. A., and Al- brecht, S. V. Scaling multi-agent reinforcement learning with selective parameter sharing. In ICML, pp. 1989- 1998. PMLR, 2021.
Li, X., Pan, L., and Zhang, J. Kaleidoscope: Learnable masks for heterogeneous multi-agent reinforcement learn- ing. In NeurlPS, 2024d.
Das, A., Gervet, T., Romoff, J., Batra, D., Parikh, D., Rab- bat, M., and Pineau, J. Tarmac: Targeted multi-agent communication. In ICML, volume 97, pp. 1538-1546, 2019.
Lowe, R., Wu, Y.. Tamar, A., Harb, J., Abbeel, P., and Mor- datch, I. Multi-agent actor-critic for mixed cooperative- competitive environments. In NeurlPS, pp. 6379-6390 2017.
Dohare, S., Samandez-GGrriin J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632(8022):768-774) 2024.
Mahajan, A., Rashid, T., Samvelyan, M., and Whiteson, MAVEN: multi- variational exploration. In NeurlPS, pp. 7611-7622, 2019.
Foerster, J. N., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. In AAAI, pp. 2974-2982, 2018.
Ng, A., Jordan, M., and Weiss, Y. On spectral clustering: Analysis and an algorithm. NeurlPS, 14, 2001.
Guangyuan, S., Li, Q., Zhang, W., Chen, J., and Wu, X.MM. Recon: Reducing conflicting gradients from the root for multi-task learning. In ICLR, 2023.
Nikishin, E., Schwarzer, M.. D'Oro, P., Bacon, P. and Courville, A. C. The primacy bias in deep re- inforcement learning. In ICML, pp. 16828-16847, 2022. URL v162/ w162/nikishin22a html.
mor.smmr..pp mir.press/
Hernandez-Leal, P., Kartal, B., and Taylor, M. E. Is mul- tiagent deep reinforcement learning the answer or the question? A brief survey. In AAMAS, pp. 750-797, 2019. URL http: /tttt:://aaaae /ttp///axvvvv.e org/ abs/ 1810.11000 05587.
Oliehock, F, A. and Amato, C. Concise Introduction to Decentralized POMDPs. Springer Briefs in Intelligent Systems. Springer, 2016.
Hu, S., Shen, L., Zhang, Y., and Tao, D. Learning multi- agent communication from graph modeling perspective. arXiv preprint arXiv:2405.0883 08550, 2024.
Olichock, F. A., Spaan, M. T., and Vlassis, N. Optimal and approximate q- g-value functions for decentralized pomdps.
10

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
Journal of Artificial Intelligence Research, 32:289-353, 2008.
Von Luxburg, U. A tutorial on spectral clustering. Statistics and computing, 17:395 2007.
Qin, H., Ma. C.. Deng, M., Liu, Z., Mei, S., Liu, X., Wang, C., and Shen, S. The dormant neuron phenomenon in ageet reinforcement learning value factorization. In NewrlPS, 2024.
Wang. J., Ren, Z., Liu, T., Yu, Y., and Zhang, C. Qplex: Duplex dueling multi-agent q q-learning. In ICLR, 2021a.
Wang. T., Dong, H., Lesser, V. R., and Zhang. C. ROMA: multi-agent reinforcement learning with emergent roles. ICML, 2020.
Qiu, W., Wang, X., Yu, R., Wang. R., He, X., An, B., Obraztsova, S., and Rabinovich, Z. RMIX: learning risk- sensitive policies for cooperative reinforcement learning agents. In NeurlPS, pp. 23049-23062. 2021.
Wang, Y., Zhong, F,, Xu, J., and Wang, Y. Tom2c: Target- oriented multi-agent communication and cooperation with theory of mind. NeurIPS, 2021b.
Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J. N., and Whiteson, S. QMIX: monotonic value function factorisation for deep multi-aged aggent reinforcement learning, In ICML, pp. 4292-4301, 2018.
Wang, Z., Tsvetkov, Y., Firat, O., and Cao, Y. Gradient cine: Investigating and improving multi-task optimization in massively multilingual models. In ICLR, 2021c
Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of ppo coop- erative multi-agent games. NeurIPS, 35:24611-2466 2022.
Rashid, T. Farquhar, G., Peng. B., and Whiteson, S. Weighted QMIX: expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In NeurlPS, 2020.
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. NeurlPS, 33:5824- 58366, 2020.
Samvelyan, M., Rashid, T., de Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G. J., Hung, C., Torr, P. H. S., Foerster, J. N., and Whiteson, The starcraft multi-agent challenge. In AAMAS, pp. 2186-2186 2019.
Zhu, Z., Liu, M., Mao, L., Kang. B., Xu, M., Yu, Y., Ermon, S., and Zhang, W. Madiff: Offline multi-agent learning with diffusion models. NeurIPS, 2024.
Shen, S., Qiu, M., Liu. J., Liu, W. Fu, Y., Liu, X., and Wang, C. Resq: residual q function-based approach for multi-agent reinforcement learning value factorization. In NeurlPS, 2022.
Shen, S., Ma, C., Li, C., Liu, W., Fu, Y., Mei, S.. Liu, X., and Wang, C. Riskq: Risk-sensitive multi-agent re- inforcement learning value factorization. In NeurIPS, 2023. URL hattps://operrewee id=FskZeRvMJT.
n forum?
Sokar, G., Agarwal, R., Castro, P. S., and Evei, U. The dor- mant neuron phenomenon in deep reinforcement learning. In JCML, pp. 321688 2023.
Son, K., Kim, D., Kang, W. J., Hostallero, D., and Yi, Y. QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In JCML, pp. 5887-5889, 2019.
Sukhbaatar, S., Szlam, A., and Fergus, R. Learning multia- gent communication with backpropagation. In NeurlPS, pp. 2244 2252, 2016.
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V. F., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., and Graepel, T. Value- decomposition networks for cooperative multi-agent learning based on team reward. In AAMAS, pp. 2085- 2087, 2018.

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARI
A. Background
A.1. Spectral Clustering
Spectral clustering (Von Luxburg, 2007; Ng et al., 2001) is a graph-based clustering method that first treats data points as nodes in graph and computes an affinity matrix based on similarity measures such as Euclidean distance or Gaussian kernel functions. The affinity matrix is then transformed into the Laplacian matrix. The Laplacian matrix L is constructed using the graph's 's affinity matrix A and its degree matrix D, as represented by the formula I. D A. The Laplacian matrix captures the connectivity between nodes in the graph and reflects the structural information of the graph, playing key role in spectral clustering. By performing spectral decomposition of the graph Laplacian, a low-dimenss -wwdimensional embedding of the data is obtained. In this low-dimensional space, data points within the same cluster are more tightly grouped, while the distance between different clusters is larger, enabling the clustering algorithm to perform better on complex and nonlinear datasets.
Finally, a conventional clustering algorithm, such as k-means, is applied this low-dimensional representation. Compared to traditional k-means, spectral clustering often outperforms in handling nonlinear structures and complex data, as it can better capture implicit relationships between data points.
B. Related work
B.1. Multi-Agent Reinforcement Learning
Multi-Agent reinforcement learning (MARL) provides a framework for modeling complex interactions among agents, with methods typically classified into policy-based, value decomposition, offline, communication-base and exploration approaches.
Policy-based methods, such as COMA (Foerster et al., 2018), MADDPG (Lowe et al., 2017), and MAPPO (Yu et al., 2022), optimize agent policies using gradients. In contrast, value decomposition methods, include VDN (Sunchag et al., 2018), QMIX (Rashid et al., 2018), QPLEX (Wang et al., 2021a), QTRAN (Son et al., 2019), ResQ (Shen et al., 2022), RiskQ (Shen et al., 2023), and RMIX (Qiu et al., 2021), which are particularly effective in cooperative tasks.
Offline methods, such as MADIFF (Zhu et al., 2024) and DoF (Li et al., 2024a) using diffusion to generate trajectories and policies of agents. Communication--asse 2019), ToM2C (Wang et al., 2021b), and CommForner (Hu et al., 2024), focus on enhancing multi-agent cooperation through improved communication. Exploration- baased methods like MAVEN (Mahajan et al., 2019) and ICES (Li et al., 2024c) aim to strengthen agents' exploration capabilities to better adapt to dynamic and complex environments.
approaches, such as CommNet (Sukhbaatar et al., 2016), TarMAC (Das et al.,
Our method, GradPS, is orthogonal to these techniques, focusing on improving learning efficiency by mitigating gradient conflicts among agents.
C. Method
C.1. Gradient Conffict
Gradient conflicts commonly occur in multi-agent systems. Even if agents share a common goal, they may have different partial observations, be located in different positions of the game which may lead to different agent gradients (e.g., divergent actions). For example, when capturing one prey, it is possible that the best action for agent 1 is to move left, while the best action for agent 2 is to move right. This will cause gradient conflict when the optimal actions differ between agents.
We apply VDN and QMIX to run 3m scenario. As shown in Figure 1, gradient phenomenon still exists when using different optimizers. The gradient conflict in multi-agent reinforcement learning is shown in Figure 2.
C.2. Gradient Monitoring
The gradient of neurons refers to the gradient on the activation layer a neural network. To calculate the gradient for each agent, we designed gradient detection layer capable of computing the partial derivative of the loss with respect the observation of specific agent. Specifically, the input data to the detection layer has dimensions of (batch agents, dim), and the layer's weights are fixed all-ones matrix with dimensions (agents, dim). The input dimensions are reshaped to
12

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
VDN-Adam
Adax--Adam
1.5
Sum
Sum
Sum
Agentt
Agent
Agent 2
Agenn
Agart
1.0
Gradlare
-0.3-
-0.4
-1.0
2000
LOOK
Gooc
GOOK
1.0CC
200K
SO0K
200K
2008
500K
Invironmenttal steps
BIVV orterraal Stepe
Eratranmeertal Steps
Environmmnntal Strppe
Figure 1: Gradient conflicts in VDN and QMIX when using different optimizers.
Quor
Gradient Conflict
FgJ
Mixing Network
Vga
VΦb V'b +V + V
Q1
Q
Qn
Sum
Agent-shared Network
Optimizer (
C
Figure 2: Gradient conflicts in multi-agent reinforcement learning (MARL). Different colors represent the observations and gradients of different agents. The overall gradient, computed by summing the gradients of all agents, is used to optimize the parameters of the agent network. However, conflicts may arise when positive and negative gradients counteract each other.
(batch, agents, dim), then multiplied element-wise by the all-ones matrix. The result is subsequently reshaped back the original dimensions of the input data to produce the output.
After the backpropagation of loss, each detection layer obtains the calculated partial derivative, which is the gradient of each neuron. We obtain the gradient of loss in the detection layer and clear these gradients to prevent the optimizer from updating the parameters of the detection layer. Afterward, the optimizer will continue to update the network parameters using the gradients of each parameter as usual.
This allows the gradient information be retained in the gradient detection layer without altering the original data values. Batch refers to the batch size, agents represents the number of agents, and dim denotes the number of neurons.
C.3. Conffict-Based Grouping
The gradients for the most recent T target network update periods accumulated and recorded. Please refer to Appendix C.2 for a detailed procedure of gradient recording. At the end of each target network update period, we evaluate the neuron efficiency of neurons with T period data and identify futile neurons if their neuron efficiency (defined in (4)) is low.
The appearance of futile neurons indicates significant conflicts in the agents during recent periods. To reduce gradient conflicts, We divide the agents into groups based on their gradient conflict values accumulated over periods, aiming to minimize the total gradient conflict within each group.
Based on the gradient conflict values between different agents, we apply the spectral clustering algorithm to clustering the agents. The clustering information is implicitly represented by the gradient conflict values between the agents. Therefore, we utilize the Gaussian kernel function to calculate the distance between agents based on gradient conflicts. First, the conflict matrix is transformed into an affinity matrix A using the Gaussian kernel function. The affinity matrix represents the similarity between agents. Each cell Atj is defined as follows.
A, where Ciz is the Neuron Gradient Conflict between agent and j defined (3), and a is the scale parameter of the Gaussian
(C.1)
13

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
kernel, which controls the influence of distance (conflict value) on similarity. Typically, is chosen be close the mean median of the sample distances, ensuring that the affinity matrix is neither too sparse nor too dense.
The graph Laplacian matrix L is computed as L D A, where D is the degree matrix (the degree of each node is the sum of the weights of its connected edges) and A is the affinity matrix.
Di.j = {
0, if
i 1
(C.2)
Σ, A if ==
The Laplacian matrix is subjected spectral decomposition, and the top K eigenvectors corresponding the smallest eigenvalues are selected the embedding vectors. In the embedding space, K-means clustering is applied to these eigenvectors to partition the agents into K groups.
C.4. Algorithm
The detailed GradPS algorithm is described in Algorithm 2.
Algorithm 2 GradPS
Require: cumulative gradient interval T, number of groups K, group restore probability p
1: Initialize parameters of the network 6
2: Initialize parameters the target network d'
3: Initialize replay buffer D
4: Initialize Neural Network Optimizer
5: Initialize the SGD optimizer for restoring
6: for (1, ( m episodes] do
7: Start new episode;
8: while Episode is not end do
9: Get the Agent action a;
10:
Execute a,, obtain global reward r and the next state s
11:
Update replay buffer D
Sample batch D' from replay buffer D
13:
Loss(O, Φ) (Q(ss, a; f,, Φ) )s
14:
Accumulated the gradient of each agent
15:
Update ø by Loss
16:
end while
17:
if Target network update then
18:
Get the set neurons that have accumulated T period gradients in the first activation layer Select the futile neurons for Each futile neuron do Grouping agents into K groups according to gradient conflicts
21:
22:
Make K copies of the parameters
23:
Assign parameters of the corresponding group to each agent
24:
Clear the accumulated gradient of this neuron
25:
end for
if Restore operation required then
27:
Calculate the variance of each neuron Set the SGD optimizer to start resetting the weights of this neuron end if
28:
29:
30:
if Some neurons have already been reset then
31:
Restore these neurons to fully parameter sharing
32:
end Φ
34:
end if
35:
end for
14

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARI
D. Experimental Details
D.I. Experimental Setup
We compare GradPS with six parameter sharing methods, including Kaleidoscope (Li ct al., 2024d), Selective Parameter & Sharing (SePS) (Christianos al., 2021), Structured Network Pruning with parameter Sharing(SNP) (Kim Sung. 2023), Parameter Sharing (FuPS), Full Parameter Sharing with index (FuPS+id), No Parameter Sharing (NoPS). For the sake of robustness, each experiment was carried out using different random seeds at least times.
Table 1: Baseline parameter sharing algorithms
Algorithms
Description
FuPS 2
Agents share all the parameters. Agents share all the parameters with agent ID in input.
FuPS+id
SNP (Kim & Sung, 2023)
The network is randomly pruned subnetworks.
SePS² (Christianos et al., 2021)
Agents are clustered to share parameters within each cluster. Agents do not share any parameters.
NoPS
Kaleidoscope* (Li al., 2024d)
Agents share parameters based on learnable masks.
We implement these algorithms based their open-source repositories to carry out performance analyses with hyperpa- rameters consistent with those in PyMARL. Our methods implemented within the PyMARL framework, and each is evaluated using five random seeds with 95% confidence intervals. Specific hyperparameters of different environments are listed in Table 2. We conduct experiments on a cluster equipped with multiple NVIDIA GeForce RTX 3090 GPUs.
Table 2: Hyperparameter of different environments
Hyperparameter
SMAC
Predator-Pre Prey
Action Selector
epsilon greedy
epsilon greedy
Batch Size
32
32
Buffer Size
5000
5000
Learming Rate
0.0005
0.0005
Hypemet Embed Dimension
64
Target Update Interval
200
200
Futile Threshold
0.2
0.2
Accumulated Period
20
Group Number K
3
Restore Probability P
0.05
0.025
Reg Learning Rate
0.005
0.005
D.2. Environment
D.2.1. PREDATOR-PREY
The predator-pre scenario simulates a grid world where multiple agents cooperate to capture prey scattered across the map. The prey consists of two types: stags and hares. Capturing a hare requires only one agent while capturing a stag necessitates at least two agents working together. Successfully capturing a hare provides a team reward of +2, while capturing deer yields reward of +10.
Some predators possess special, unobservable trait that enables them to receive the same team reward for capturing hare as they would for capturing stag. The goal is to maximize the total team reward within limited timeframe. We have developed three different environmental configurations medium, and each with varying numbers of agents and prey, as well as different map sizes.
'https: :tttps://gt ://git/ubbttt con/ http://whhrrhhmhaeae Phtpps: httpuuu..ccommtjaaaaaaaattooooot hatpttta//ap///a/aampm Shttps: :/gi thub. coms/ thub. tttpt.ccm//.xxxx///aaaeeeacceeee
15

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
Game Rules
Agent Movement: Agents can move in four directions or stay in place. Movement is restricted by the presence of other agents or preys.
Observation and Decision Making: Each agent observes a 7x7 grid centered around itself, receiving information about nearby agents and preys. Decisions are based on this local observation.
To capture a stag, at least two agents must be adjacent to it and must choose the capture action at the same time. contrast, capturing hare requires only one agent. Successful capture relies on strategic positioning and synchronized actions among agents.
Capture Mechanism:
Rewards and Penalties: Agents receive positive reward for each prey captured through cooperative action.
Episode Termination: An episode terminates if all preys are captured or after 150 steps, providing a fixed time frame for agents to maximize their collective reward.
Configuration
Number of All Predators
Number of Special Predators
Number Stags
Number of Hares
Map Size
Reward for Stags
Reward for Hares
Small
2
6
20 20
+10
+2
Medium
10
3
10
10
25 25
+10
+2
Large
15
5
15
15
30 3:
+10
42
Table Comparison of Predator-Prey Configurations
D.2.2. STARCRAFT II MULTI-AGENT CHALLENGES (SMAC)
The StarCraft Multi-Agent Challenge (SMAC) (Samvelyan al., 2019) is popular benchmark used extensively in the domain of multi-agen aaget- reinforcement learning. Built on the StarCraft II game engine, SMAC specializes in micromanagement scenarios where each agent is controlled by independent agent who must make decisions based on local observations. MARL algorithms coordinate team of agents to engage in combat against an opposing team managed by the game's built-in AI.
The performance of these algorithms quantitatively evaluated by the test win rate or the test return of the gameplay. Table 4 depicts the overview of SMAC scenarios used in the experiment.
Name
Difficulty
Ally Units
Enemy Units
3m
Easy
3 Marines
3 Marines
5m
Easy
5 Marines
5 Marines
1 55
Easy
15 Marines
15 Marines
9z
Easy
Zealots
9 Zealots
3s6z
Easy
3 Stalkers & 6 Zealots
3 Stalkers & 6 Zealots
lc3s5z.
Easy
1 Colossi
3 Stalkers & 5 Zealots
Colossi Stalkers Zealots
5m_vs_ 6m
Hard
5 Marines
6 Marines
2c_vs_642g
Hard
Colossi
64 Zerglings
MMM2
Super Hard
Medivac, Marauders & Marines
Medivac, Marauders & 8 Marines
27m_vs_30m
Super Hard
27 Marines
30 Marines
lc3s82_v 1isii_vv__le__ v8_1c3s92
Super Hard
I Colossi & 3 Stalkers & 8 Zealots
Colossi & 3 Stalkers & 9 Zealots
3s5z, vs. 3_6sz
Super Hard
3 Stalkers & 5 Zealots
3 Stalkers & 6 Zealots
Table Overview of SMAC scenarios used in the experiment.
16

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
D.2.3. ONE STEP MATRIX GAME
The One-Step Matrix Game is a classic experimental environment used evaluate the performance of multi-agent reinforcement learning (MARL) algorithms. By designing simple reward structure in the form of matrix, it simulates cooperative competitive interactions among multiple agents, enabling the assessment of an algorithm's 's ability to handle nonlinear relationships and coordination challenges. In this study, the One-Step Matrix Game involves two agents, each selecting an action, where the combination of all agents' actions constitutes a joint action. The rows and columns of the matrix represent the action choices of the respective agents, and each element within the matrix denotes the reward value corresponding to a specific joint action.
C
A
A
B
D
M1
11
2
0
0.5
A
13
1.3
23
C
0
2
2
2
2.5
35
0
5
D
3
D
3
33
43
D
6
Table S: f2 -1x
Table f2 0
Table
0.5 x Ji
Table f2 =
Table 9: Simple one-step payoff matrix game for understanding the impact of futile neurons network expressiveness.
Sum Agent Agunt
Agent Agent
Agent 2
MM
Agent 2 Aguac
Ageat
0.5
Graaionn
Mhhu
-1.0 Greeteet
-0.5
-0.3
0.0
4
-1.3
22.0
Iovirommental
Bevirommentall Steps
Bnvircementaal
Envirunmental Stops
Figure Conflicting neuron gradients in different one step payoff matrix games.
Figure depicts the gradients for different agents in two payoff matrix game using the VDN methods. As is depicted in the graph, the gradients for different agents can be significantly different. The conflict of gradients may lead canceling out each other, which makes the sum of the gradient for one close zero. GradPS can further reduce bias by resolving gradient conflicts in neurons, as shown in Figure 4.
-1.0
fl 0.0
f1 0..5
40
40
SeadPt
35
1224 Stadtt
0.4K
0.8K
1.2K
1.CK
2.0K
0.4K
0.8K
1.2K
1.CK
2.4K
0.4K
0.8K
3.2K
1.0K
2.0K
Enstronmental Stops
Ensironmental Stopa
Environmental Stops
Figure 4: Applying GradPS to different one step payoff matrix games.
D.3. Experimental Results
Predator-Prey
Figure 5 presents the return and the percentage of futile neurons in GradPS within the predator-pun prey environment, in comparison to other parameter-sharing algorithms.
SMAC
Figure presents the win rate and the percentage of futile neurons in GradPS within the SMAC environment, in comparison to other parameter-sharing algorithms.
17

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
(a) predator-prey small
(b) predator-prey medium
(c) predator-prey large
120
200
300
GradPs
GradPS
GradPS
100
PuPS HD
PuPS
160
PuPs
PaPS+10 ID
250
PUPS-ID
PUPS
SNP
SNP
80
NoPS
NoPS
200
NePS
Return
Kaindoscope
Return
120
Kaleldorcape
Kaleidoscope
60
150
80
40
100
20
40
50
0.2M
0.6M
0.8M
1.0M
0.2M
0.6M
0.8M
1.0M
0.2M
0.4M
0.6M
0.8M
1.0M
Enviro Steps
Steps
Steps
(d) predator-prey small
(e) predator-prey medium
(f) predator-prey large
GradPs
70
GradPS
80
PuPS
40
Futile Neuron
40
Futile Neuron
40
30
20
20
Futile
20
20
Environmental Steps
Environmental Steps
Environmental Steps
Figure 5: GradPS performs better than other PS methods on Predator (a-c) the return for predator-pre small/mmediu//lage environment. (d-f) the percentage of futile neurons for predator-prey small/medium/large environment.
D.4. GradPS Performance Under PyMARL3 Settings
We extended the training time based the settings of PyMARL and PyMARL3. The results shown in Figure 7 and Figure 8.
D.5. Ablation Study and Discussion
D.5.1. COMPARISON OF FUTILE AND DORMANT NEURONS
Feature
Dormant Neurons
Futile Neurons
Definition
Normalized Average Activation Value falls below
Neuron Efficiency is below
Normalized activation score
Small
Any
Gradient conflict Gradient
Unclear
Large
Near Zero
Any
Parameter Update
Near Zero
Slowed down by gradient conflict
Table 10: Overview of SMAC scenarios used in the experiment.
Dormant neurons (Sokar et al., 2023) have low normalized activation scores, while futile neurons suffer from gradient conflicts unrelated to activation. A non-dormant neuron can be a futile neuron. We have listed the differences as Table 10.
Additionally, we report the percentage of dormant neurons and futile neurons agent networks with VDN during the training process in three simple payoff matrix games. Moreover, we evaluate the MSE of reconstructing the Payoff matrix when using ReDo (Sokar et al., 2023) and GradPS. ReDo and GradPS are methods developed to reduce the percentage of dormant neurons and futile neurons, respectively. The default parameters of ReDo and GradPS are used. The results are shown in Figure 9.
For the f2 = fi 0.0 and f2 = fi 0.5 games, the Neuron Percentage graphs show that dormant neurons gradually decrease and stabilize to fix values (around 10%), whereas futile neurons rise to 100% after a few thousand environment steps. For the f2 f1 -1.0 game, the percents the dormant neurons and the futile neurons fluctuate around 30%-40% and 80%-90%.
18

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
vs. 642g
GradPS
70
PUPS
80
FUPS
60
RUPS+1D
PUPS
NoPS
NoPS
50
NoPS
Test Win
40
Kaleidoscope
SePS
Test Win
Kaloidoscope
SePS
40
Kaleidoscape
SePS
40
30
Test
20
20
20
10-
0.4M
0.8MM
1.2M
1.6M
2.0M
0.4M
0.8M
1.2M
1.6M
2.0M
0.4M
0.8M
1.2M
1.6M
2.0M
Environmental Steps
Environmental Steps
Envirommental Steps
5m vs 6m
MMM2
(d) 2c vs 642g
50
GradPS
GradPS
GradPS
FuPS
FuPS
FuPS
FuPS+ID
SMP
40
FuPS+iD
SNP
40
SAP
FuPS-IB
Kaleidoscope
Sers
ScPs
40
Kaleidoscope
Kaleidoscape
30
Futtle Neuron
Futtle Neuron
Futlle Mouron
20
10-
10
10
0.4M
0.8M
1.2M
1.6M
2.0M
0.4M
0.8M
1.2M
1.6M
2.0M
0.4M
0.8M
1.2M
1.6M
2.0M
Environmental
Environmental
Environmental Stops
27m vs. m0
1c3s8z 1c3s9z
3s52 vs. 3s66
60
GradPS
GradPS
GradPS
PuPS
FuPS
FuPS
FuPS+ID
40
FuPS-ID
30
PUPS-10
Test Win Rate
NoPS
NoPS
HoPS
SePS
SePs
SePS
300
Win
Win
20
20
10
10
10
0.4M
1.6M
2.0M
Environmental Steps
Environmental Steps
Environmental Steps
27m 30m
1c3s8z 1c3s9z
3s5z 3s6z
70
Patlle Neuron Percentage
60
50
40
SuPs
50
Kateidoscope
Kalcidesiopp
Kalsidoscopo
Petlle Neuron
Putlle Nouron
30
20%
10
10
0.4M
0.8M
1.2M
1.6M
2.0M
0.4M
0.00
1,2M
1.6M
2.0M
0.4M
0.8
1.2M
1.6M
2.0M
Environmental Stops
Environmental Stops
Environmental Stops
Figure 6: Comparison with other Parameter Sharing methods: the test win rate and the futile neuron percentage in SMAC
19

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
120
predator-pre smalll
200
predstor-prey medium
300
predstor-prey large
100
160
250
200
120
150
80
Fork
Fure
Fara
Kaleideecape
1
2.0M
Environmental Steps
Enstronmental Steps
Environmental Stape
(5) 5m, vs 6m
MMM2
2c vss 64.zg
FuRE
Puse
SMP
.6-
60
60
00-
0
0.4M
3:2M
1.6M
2.4M
3.0M
0.6M
3.2M
1,6M
2.4M
3.0M
0.0M
3:2M
1.OM
2.45
3.0M
Environmental Steps
Envyr inverrmmentlllocmmm Steps
Environmental Steps
27m, vs. 30m
1c3s8z VS. 935
35 VS. 3:52
PuPS
Purt
Pura
SMF
SMP
DMP
80-
80
0.6M
3.3M
1.804
2.4M
3.0M
1.2M
2.4H
2.60
4.0M
6.0M
1.2M
2.4M
2,0M
6.0M
Figure 7: Comparison with other Parameter Sharing methods: the test win rate in SMAC and the return in Predator-Pres Prey
predator-prey small (pymari3)
predator-pee medium (pymari3)
predator-grey large (pymari3)
120
200
300
100
160
250
200
120
350
30
Fuls
fuls
Fults
1
2.0M
Environmental Steps
Enstronmental Stepe
Environmental Stape
5m, vs. 6m (pymaris)
MMM2 (pymar13)
2c vs 642g (pymari3)
3.00
GradPs
Purs
FUPSID
SMP
descape
60
60-
40
20-
01
0.6M6
3:2M
1.6M
2.4M
3.0M
0.6M
1,2M
1,0
2:4N
KUM
0.6M
3.2M
1.O
2.45
3.0M
Environmental Steps
Enstronmental Steps
Environmental Staps
27m vs. 30m (pymari3)
1c3s8z vs 1c3s92 (pymar13)
3552_vs, 3352 (pymari3)
Gradrs
50
Puni
Pult
Pars
SRF
SMP
60
3.0
20-
0.6M5
3.2M
1.804
2:4M
5.094
1.2M
2.4M
5,004
4.3M
6.0M
1.2M
2.4M
5,0
6.60
6.0M
Environmental Stops
Enviir Stops
Environmental Stops
Figure 8: Comparison with other Parameter Sharing methods based on PyMARL3: the test win rate in SMAC and the return in Predator-Prey
20

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
respectively. These figures show that the percentages for the two types of neurons are different across training processes.
For all three matrix games, the MSE of reconstructed payoft matrix is shown in MSE.pdf figures. For these graphs, the MSE for the original VDN, VDN with ReDo (Sokar et al., 2023), VDN with GradPS are shown.
When using ReDo with VDN, the MSE does not change significantly compared to the VDN method. However, when using GradPS with VDN, the MSE drops significantly. For these matrix games, in terms of reducing MSE, GradPS, as a futile-neuron-method, works better than ReDo, dormant-neuron baasd method. These findings further highlight the significant difference among the two types of neurons.
VOR
VON
VON
VON-GradPP
VON diadPS
VoN GradPS
VoN Rodd
VoN Rode
VDN Rodeo
Squurre
20
20
15
15
15
110
Environmental
Ensironmetaal
Environmental Stops
- f1 -1.0
f2
fl 0.0
f2
fl 0.5
1.00
100
Meurono Percentipe
seatte
www
Neurom Porcentage
Dermant Searce
Heurono Porcentage
Durment leuume
8
MUY YUNI
1.2K Environmenttal
Environmental
Environmental
Figure 9: Conflicting neuron gradients different one step payoff matrix games.
D.5.2. THE IMPACT OF NETWORK WIDTH
In this section, we study whether gradient conflicts are different in networks of different widths. As shown in Figure 10, the percentage of futile neurons remains approximately equal across networks of different widths.
MMM2
MMM2
MMM2
60
Average Efficiency Percentage
60
PuPS-32DIM
Futile Neuron Parcantage
PUPS-128DIM
50
Test Win Rate
60
.50
RuP5-256DIM
40
45
40
40
30
20
20
30
10
0.4M
2.0M
0.4M
2.0M
2.0M
Environmental Steps
Environmental Steps
Environmental Steps
Figure 10: Average Efficiency Percentage and Futile Neurons Percentage in networks with different widths.
D.5.3. SHARING AT DIFFERENT DEPTII OF NETWORK LAYERS
In this section, we investigate whether gradient conflicts differ at different depths in the network. We deepen the QMIX agent network to have four intaa-activation layers. We apply GradPS to these activation layers and detect gradient conflicts separately. The results are shown in Figure 11, the number of futile neurons in the first layer is greater than in the other layers.
21

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
(a) MMM2
(b) MMM2
(c) MMM2
First Layer
gMik-t-Layer
QMil-e-Layer
Second Layer
Gradd5-1-Lyyer
GradP5-1-Laym
60
Fourth Layer
Third Layer
GradP5-2-Laye
L--yer
50
GradP6 L-Lyer
GradPS Layer
GradP5-44 Layeer
Gradh5-4 Layyer
M
50
40
Pattle Neuron
Futlle Mauron
40
30
Test
20
20
20
20
10
0.4M
0.8M
1.2M
1.6M
2.0M
0.4M
0.8M
1.2M
1.6M
2.0M
0.4M
0.8M
1.2M
1.6M
2.0M
Environmental Stops
Environmental Steps
Environmental Stops
Figure 11: (a) The Futile Neurons Percentage at different depths of four layer QMIX network. (b) The performance of GradPS in different network layers. (c) The Futile Neurons Percentage after using GradPS on different layers.
D.6. Execution Time Table and Memory Usage Table.
We present the wall-clock training time (in hours) for each algorithm, averaged over random seeds. The computational overhead of GradPS is small. It takes less training time than SNP, SePS and NoPS, most of the time.
Table 11: Training time of different parameter sharing algorithms
Training Time(h)
FuPS
GradPS
FuPS+id
SNP
SePS
NoPS
Kaleidoscope
Predator-Peee Prey Small
2.45
2,74
2.55
2.72
5.23
5.45
4.37
Predator- Predator-Prey Medium
5.11
5.65
5.23
5.34
6.86
7.18
6.32
Predator- Prey Large
5.89
6.32
5.84
6.62
9.92
11.65
9.05
5m_vs_
9.21
9.74
9.33
10.64
12.56
14.21
11.86
MMM2
12.94
13.83
13.02
14.95
18.42
19.34
15.53
27m vs 30m
23.12
24.75
23.45
25.77
36.53
42.32
31.45
The following table depicts the number parameters for the whole network and each agent network compared to the full parameter-sharing approach. The additional memory overhead of GradPS is lower than that of SePS and NoPS.
Table 12: Usage of the whole network parameters
The whole network Parameter usage
FuPS
GradPS
FuPS+id
SNP
ScPS
NoPS
Kalciduscope
Predator-Pres small
100%
105%
100%
103%
112%
159%
114%
Predator- Fredator-Prey medium
100%
104%
100%
102%
108%
171%
109%
Predator- Prey large
100%
103%
100%
101%
105%
178%
107%
5m_vs_ 66
100%
116%
101%
106%
180%
261%
151%
MMM2 27m_vs v3m00m
100%
122%
101 %
108%
156%
354%
134%
100%
114%
101%
105%
128%
466%
117%
Table 13: Usage of agent network parameters
Agent network Parameter usage
FuPS
GradPS
FuPS+id
SNP
SePS
NoPS
Kaleidoscope
Predator-Prey Small
100%
145%
102%
123%
198%
624%
218%
Predator-Prey Medium
100%
146%
103%
124%
197%
1034%
217%
Tredator-Prey Large
100%
147%
104%
124%
196%
1540%
216%
5m_vs 6m_6m
100%
140%
102%
116%
297%
519%
226%
MMM2
100%
176%
102%
128%
296%
1036%
218%
27m vs. 30m
100%
195%
105%
136%
293%
2762%
215%
22

---

GradPS: Resolving Futile Neurons in Parameter Sharing Network for MARL
E. Discussion
E.1. Societal Impact
Our research primarily concentrates on the technical and theoretical aspects of multi-agent reinforcement learning, aiming to enhance the performance of these agents across variety of tasks. While not foresee any direct negative consequences arising from our research, feedback from the community ensure work's contributions are beneficial and ethically sound.
committed maintaining an open dialogue. We highly appreciate and value constructive
E.2. Limitations and Future Work
Although the gradient-based grouped parameter sharing method proposed has achieved promising results compared other approaches, there is still room for improvement. Our hyperparametens, such the number of groups and the threshold for defining futile neurons, require further tuning. Additionally, increasing the number of groups may lead to higher parameter overhead. Defining futile neurons more effectively and exploring adaptive grouping will be key directions for our future research.
23


Reviews:
[{"review_id": "LFTeEkzyxs", "reviewer": "Reviewer_EmSH", "summary": "This paper identifies \"futile neurons,\" neurons with conflicting gradient updates, in parameter-shared multi-agent reinforcement learning (MARL). It proposes GradPS, which dynamically clones these neurons, grouping agents with low gradient conflict to promote diversity and efficiency. Experiments on SMAC and Predator-Prey benchmarks show improved performance compared to existing parameter-sharing methods.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Claims:\n> 1. ...convergence plots are missing, ...More explicit reproducibility analysis...\n\nAll experiments were repeated with **5 different random seeds** to ensure reproducibility, with means and variances shown in all experimental result figures of this work.\n\nConvergence of GradPS are evaluated by running GradPS in three environments for 3M instead of 2M steps. The results are depicted as follows. After 2M steps, for MMM2, the performance of GradPS continues to increase; for Predator-Prey and 5m_vs_6m, their performance is stable.\n\n|Env.|0.5M|1M|1.5M|2M|2.5M|3M|\n|-|-|-|-|-|-|-|\n|Predator-Prey Medium(return)|138.5|174.4|187.5|191.2|189.7|190.4|191.5|\n|5m_vs_6m(win rate)|28.2|57.9|68.7|73.4|74.3|72.5|74.6|\n|MMM2(win rate)|4.8|25.1|67.5|79.3|85.2|91.4|90.2|\n## Methods:\n>1. ...whether baseline methods are implemented optimally...\n\nFor fairness, all parameter-sharing (PS) methods (e.g., FuPS and SNP) were implemented in the same repositories (e.g., pymarl).  PS implementation for pymarl (located in rnn_agent.py) is the same as pymarl2 and pymarl3. From this point, pymarl2 and pymarl3 are not optimized pymarl. Moreover, we also evaluate distributional and risk-sensitive agents, where are not included in pymarl2&3.\n\nWe show in the following table that for pymarl3 QMIX experiments, GradPS outperformed FuPS+ID and FuPS. This shows that our method works in pymarl3.\n\n|Env (pymarl3)|GradPS|FuPS+Id|FuPS|\n|-|-|-|-|\n|Predator-Prey Medium (return)|187.2|165.8|139.6|\n|5m_vs_6m (win rate)|73.4|68.9|65.7|\n|MMM2 (win rate)|85.8|74.2|72.3|\n## Experimental:\n>2 ...the grouping mechanism and sensitivity to hyperparameters...\n\n**We have described the grouping method in Section 5.1** and provided more details in Appendix C. We describe the hyperparameters and present more results as follows.\n\n$T$ is the period for identifying futile neurons. An excessively large T may delay futile neuron detection. \n\n$K$ denotes the number of groups. Setting K too large will lead to low sample efficiency.\n\n$\\rho$ indicates the recovery probability for a group. In short-term tasks, even $\\rho$=0 has a negligible effect on performance. \n\nSensitivity analysis for $T$, $K$, and $\\rho$ is shown in the left, middle, and right of Figure 10 in Appendix D.5.3, respectively.\n\nThe $\\alpha$ determines futile neuron identification. Our sensitivity analysis (see table) shows $\\alpha$=0.2 yields promising results.\n\n|Env.| $\\alpha$=0.1 | $\\alpha$=0.2 | $\\alpha$=0.4\n|--|--|--|--|\n|5m_vs_6m(win rate)|72.3|74.5|67.6|\n|MMM2(win rate)|78.3|82.6|67.6|\n|Predator-Prey Medium(return)|167.4|173.9|178.8|\n## Weaknesses:\n>1. Section 5.2 and 5.3 is hard to read.\n\nWe will revise them to increase readability.\n\n### Method: \n>1. ...seems a kind of dormant neuron...\n\nDormant neurons have low normalized activation scores, while futile neurons suffer from gradient conflicts unrelated to activation. A non-dormant neuron can be a futile neuron.\n\nWe have listed the differences as follows.\n\n| Feature|Dormant Neuron|Futile Neuron|\n|-|-|-|\n|Definition|Normalized Average Activation Value falls below $\\tau$ |Neuron Efficiency is below $\\alpha$|\n|Normalized activation score|Small|Any|\n|Gradient conflict|Unclear|Large|\n|Gradient|Near Zero|Any|\n|Parameter Update|Near Zero|Slowed down by gradient conflict|\n\n>2. ...the grouping method is not mentioned in the main paper...\n\n**We have described the grouping method in Sec 5.1 of the main paper.**\n\n### Experiment: \n>2. ...Lack of comparison with naive baselines...\n\nWe compared with **SOTA partial PS methods**: SNP, which prunes a shared network for individual policies, and Kaleidoscope, which learns agent-specific masks. Our method outperforms both, as shown in experiments.\n\nFollowing the reviewer's suggestion, we also compared with Naive LastPS (sharing all but the last layer), where our approach demonstrates significantly better performance, as it is depicted in following table.\n\n|Env|GradPS|Naive LastPS|\n|-|-|-|\n|Predator-Prey Medium(return)|173.9|74.7|\n|5m_vs_6m(win rate)|74.5|54.6|\n|MMM2(win rate)|82.6|16.2|\n## Suggestions:\n>Put more details in the method section.\n\nWe will add more details to improve its readability.\n\n## Questions: \n>2. Is the introduction of \"futile\" necessary?\n\nIn PS networks, gradient conflicts produce \"futile neurons\" - neurons hindered by conflicting updates. As detailed in our comparison table, these are systematically identifiable through their distinct characteristics.\n\n>3. why...the execution time of GradPS increase..., is the same as the fullps baseline...\n\nIn GradPS, the number of parameters per agent differs from that in FuPS, as demonstrated in Appendix D.6. GradPS requires additional parameters for cloning neurons and storing gradients. For computational overhead, GradPS requires about 1.1$\\times$ the training time of FuPS; it requires significantly less (40-50%) time than SNP and NoPS.", "claims_and_evidence": "The main claims of improved diversity and performance are generally supported by experiments. However, convergence plots are missing, making it hard to assess long-term stability. More explicit reproducibility analysis (e.g., variability across random seeds) would also help validate the claims.", "methods_and_evaluation_criteria": "The proposed GradPS algorithm is innovative and appropriate for the MARL setting. Experiments involve relevant benchmarks and comparisons. However, it is unclear whether baseline methods are implemented optimally, such as from existing strong repositories, potentially biasing results in favor of GradPS.", "theoretical_claims": "The paper does not make theoretical claims requiring proof validation. Definitions of gradient conflict and futile neurons are clearly presented and intuitive.", "experimental_designs_or_analyses": "Experiments cover multiple relevant benchmarks. Still, convergence properties are not shown explicitly, and some analyses (Sections 5.2/5.3) lack clarity. Specifically, the grouping mechanism for neurons and sensitivity to hyperparameters (e.g., number of groups K) need clearer explanation.", "relation_to_broader_scientific_literature": "Related to MARL and deep learning phenomenon.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Weaknesses:\n\nClarity:\n1. Section 5.2 and 5.3 is hard to read.\n\nMethod:\n1. The \"futile\" neuron seems a kind of dormant neuron. The necessity of this new definition is not convincing\n2. The grouping method which is important is not mentioned in the main paper.\n\n\nExperiment:\n1. The performance after convergence is missed.\n2. Lack of comparison with naive partial parameter-sharing baselines, e.g., the agents share the whole network but the last layer.", "other_comments_or_suggestions": "Put more details in main paper, especially in the method section.", "questions_for_authors": "1. Are all algorithms implemented based on the same optimized repo such as pymarl2[1] and pymarl[3]?\n2. Is the introduction of \"futile\" necessary?\n3. Why does the execution time of GradPS increase, since the number of parameters for each agent is the same as the fullps baseline?\n\n\n\n[1] Hu, Jian, et al. \"Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning.\" arXiv preprint arXiv:2102.03479 (2021).\n\n[2] Jianye, H. A. O., et al. \"Boosting multiagent reinforcement learning via permutation invariant and permutation equivariant networks.\" The Eleventh International Conference on Learning Representations. 2022.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "egsMRJQgef", "reviewer": "Reviewer_EAb8", "summary": "This paper addresses the balance between parameter sharing and behavioral diversity in MARL from the perspective of gradient conflict and resolution. First, the concept of gradient conflict in MARL is introduced, and experiments verify its impact and patterns on multi-agent policy training. Subsequently, a method is proposed to group agents at the neuron level by detecting gradient conflicts. Finally, the effectiveness of the proposed method is validated in SMAC and particle-based environments.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments, we address your concerns as follows. \n## Claims: \n>1. ..\"neuron-based\"..\n\nGradPS is a neuron-based method **from the angle of gradient conflict**. We agree that pruning-based PS can be viewed as neuron-based too.\n\n>2. ..a momentary gradient direction..\n\nWe agree that gradients indicate temporary improvements. We will soften the claim regarding the direct connections between gradient and diversity/skill. \n\n>3. ..neuron efficiency decreases\n\nIn Dec-POMDPs, agents with identical tasks may take different actions due to partial observations, leading to gradient conflicts from divergent optimal actions.\n\nIn Figure 3 (Left), these agents share the same goal (removing all enemies) and with the same type. Neuron efficiency decreases with more agents due to higher partial observation diversity. Please refer to our response to Suggestion 1 of Reviewer MPio for experiments regarding agents with identical task.\n\n## Methods:\n>1. ..the functional heterogeneity of agents..\n\nThe relationship between gradient conflicts and functional differences is worth exploring. Even agents with an identical task can lead to gradient conflicts due to partial observation issues, as analyzed above. **Thus, analyzing the learning efficiency of MARL for gradient conflict is needed.** \n\nWe do not link gradient conflicts with role (functional) differences. However, according to our experiments (Fig. 10), our approach is able to discover hidden roles. \n\n>2. .. more hyperparameters..hinders its practical application.\n\nOur work is practical, requiring (only 4) no more hyperparameters than existing grouping methods without the need of pre-training (unlike SePS/AdaPS). The table below compares hyperparameters across PS/grouping methods.\n|Method|Hyperparameter Count|Description|\n|-|-|-|\n|SePS|4|Clusters count, VAE training step, latent dimension, KL weight|\n|AdaPS|4|Clusters count, Drop threshold, VAE training trajectory, KL weight|\n|ADMN|4|Module count, Layer number, Module outputs size, Combination weight|\n|MADPS|5|Obs subset size, sampling times, Fusion/Division threshold, Fusion/division interval|\n|SNP|7|Actor/Critic Pruning ratios $\\times$ 3, Subnetworks number|\n|Kaleidoscope|6|Reset probability, Ensembles, Actor/Critic diversity, Actor/Critic reset interval|\n## Theoretical Claims:\n> ...formulas and definitions...\n\nWe will fix the notations. \n\n## Experimental:\n>1. ..compare with other grouping methods.. AdaPS, ADMN [1], or MADPS [2]\n\nADMN and MADPS are **not open-source**. Reproduction attempts at such a short time would risk introducing unfair biases. GradPS performs better than AdaPS as shown in the following table.\n|Env|GradPS|AdaPS|\n|--|--|--|\n|Predator-Prey Medium|173.9|94.7|\n|5m_vs_6m|74.5|42.9|\n|MMM2|82.6|52.2|\n>2. ..compared with other restoration methods..\n\nWe compare two restoration methods, which use the average value of the group or the value of a random clone to directly replace the neuron weights. Our method is better.\n|Map/GradPS|Our|Average Restoration|Random Restoration|\n|--|--|--|--|\n|Predator-Prey Medium|173.9|154.2|142.1|\n|5m_vs_6m|74.5|64.2|54.7|\n|MMM2|82.6|53.5|35.7|\n## Essential References:\n>..some diversity-based MARL works..\n\nThanks, we will discuss all of them in related work. \n\n## Weaknesses:\n>.. computational cost and practicality.. effectiveness in deeper networks.\n\n\nWe have shown in Appendix Table 10 and 11 that the increased overhead is small. It is much smaller than partial PS methods (e.g., SePS and NoPS). \n\nOur work is practical for multi-layer agents. In Appendix D.5.2, for a 4-linear-layer agent, there are more futile neurons in the first linear layer than the other layers. Although there are more layers, such a method (QMIX-4-layer) performs weaker than standard QMIX, due to futile neurons. Applying GradPS in the first linear layer leads to higher performance gain than applying it to the other layers, justifying our first-layer focus.\n\nMoreover, we show that GradPS works in different agent networks (distributional or risk-sensitive) in Appendix D.4. Figure 7. \n\n\n## Suggestions:\n>..gradient-conflicted neurons and MARL elements..\n\nIt is necessary to study the fundamental connection between futile neurons and essential MARL elements following ADMN and MADPS.\n\n## Questions: \n>1. ..gradient be a tensor..the overall knowledge..\n\nNeuron gradient is not a vector (tensor) gradient, as defined in Definition 1. The neuron gradient is the gradient with respect to the output for a neuron, which is just a scalar (positive or negative). A holistic view of all neurons can better represent the whole knowledge. \n\n>2. ..the precise meaning..\n\nWe will soften the claim regarding the direct relationship between gradient conflicts and policy diversity.\n\n>3. ..gradients depend not only on network input/output structure..\n\nGradient conflicts correlate with diverse agent observations/actions and are influenced by factors such as loss functions and environmental stochasticity.", "claims_and_evidence": "The paper's structure is generally sound, supported by relevant evidence. However, some arguments may lack rigor or contain inaccuracies. These are pointed out below in order of appearance:\n\n1. （minor）Line 40 claims the work is \"neuron-based\" to distinguish it from others. This may be inaccurate, as other works (e.g., network pruning in PS) can also be considered neuron-based. \n\n2. （major）The paper makes multiple direct connections between an agent's policy/acquired skills and its gradients. This connection may be problematic. Gradients indicate only a temporary direction of improvement for the neural network. First, this improvement might not align with the optimal direction. Furthermore, a momentary gradient direction does not immediately constitute a stable policy or skill.\n\n3. (minor) Lines 181-184 state that neuron efficiency decreases with an increasing number of agents. This is clearly not a universal conclusion. In MARL scenarios where agents perform identical “tasks”, increasing the number of agents might actually increase neuron efficiency.", "methods_and_evaluation_criteria": "This paper constructs a strong PS algorithm from the perspective of gradient conflict and resolution. The overall idea is suitable for this problem. However, concerns about the proposed method's necessity and practicality are the primary reasons I cannot give this paper a high score.\n\n(Major) Regarding necessity, the proposed method uses gradient conflict to identify operable neurons and divide agents. However, I believe it does not address the core issue. This method dynamically assigns “roles” to agents by identifying gradient conflicts, but the essence should be to identify the functional heterogeneity of agents in MARL tasks.\n\n(Major) Regarding practicality, the proposed method still relies on two hyperparameters: the number of groups and the threshold for detecting gradient conflicts. This even introduces more hyperparameters compared to existing grouping methods (e.g., SePS), which significantly hinders its practical application.", "theoretical_claims": "This paper does not involve complex theoretical proofs. I have carefully reviewed all theoretical definitions and related claims. \n\nI believe there are just some minor issues with the notation used in the formulas and definitions.\n\n For example, in Equations 1 and 2, the definitions of *gradient conflict* and *Neuron Efficiency* lack a subscript for neuron $n$. In Equation 3, the superscript in the definition of *Futile Neuron* could be easily mistaken for an exponent symbol.", "experimental_designs_or_analyses": "The overall design of the experimental section in the paper is reasonable, but there are some issues with its specific implementation.\n\n（Minor）Regarding the choice of baselines, I appreciate the authors' selection of SePS, SNP, and Kaleidoscope for comparison. However, considering that the proposed method performs dynamic grouping, it would be beneficial to also compare it with other dynamic grouping methods. Examples include AdaPS, ADMIN [1], or MADPS [2], which are mentioned in the Related Work section.\n\n（Minor）Regarding the ablation study, since the authors mention a specially designed parameter restoration approach in lines 302-308, this approach should be compared with other restoration methods.\n\n[1] ADMN: Agent-Driven Modular Network for Dynamic Parameter Sharing\nin Cooperative Multi-Agent Reinforcement Learning. IJCAI 2024.\n\n[2] Measuring Policy Distance for Multi-Agent Reinforcement Learning. AAMAS 2024.", "relation_to_broader_scientific_literature": "This paper offers a novel perspective on diversity learning in MARL, utilizing the concepts of gradient conflict and futile neurons.", "essential_references_not_discussed": "Since the abstract and introduction both mention the relationship between parameter sharing and diversity learning, the related work section should also include works on diversity-based learning. Although Section 3.1 already covers some diversity-based MARL works, some are still missing. There is a considerable amount of work in this area, including but not limited to: CDS [1], RODE [2], LIPO [3], MADP [4], FOX [5], and DICO [6].\n\n[1] Celebrating diversity in shared multi-agent reinforcement learning. NeurIPS 2021.\n\n[2] Rode: Learning roles to decompose multi-agent tasks.ICLR 2021.\n\n[3] Generating diverse cooperative agents by learning incompatible policies.ICLR 2023.\n\n[4] Measuring Policy Distance for Multi-Agent Reinforcement Learning. AAMAS 2024.\n\n[5] Fox: Formation-aware exploration in multi-agent reinforcement learning. AAAI 2024.\n\n[6] Controlling behavioral diversity in multi-agent reinforcement learning.ICML 2024", "other_strengths_and_weaknesses": "**Other Strengths**:\n\n1. The paper investigates diversity learning in MARL from a novel perspective: gradient conflict.\n2. The paper proposes a neuron-level parameter allocation method.\n\n**Other Weaknesses**:\n\nPracticality: Besides the aforementioned concerns about hyperparameter usage, I have concerns about the computational cost and practicality of neuron-level agent assignment. The method requires calculating an assignment matrix of at least *Agent\\*HiddenDim* size, which can be computationally expensive. My careful review of the code revealed that the agents' policies are restricted to shallow networks. Its effectiveness and computational load in deeper networks, which are common in real-world scenarios, warrant further investigation and may be limited.", "other_comments_or_suggestions": "Considering the concerns regarding necessity and practicality, I recommend that the authors emphasize investigating the fundamental connection between gradient-conflicted neurons and essential MARL elements, instead of concentrating solely on presenting a new algorithm.", "questions_for_authors": "Beyond the specific concerns already raised, I have the following questions, which may influence my assessment of whether my potential concerns become definitive shortcomings.\n\n1. Regarding the gradient calculation for individual neurons across different agents in Figure 1, I have several questions.\nShouldn't the backpropagated gradient during neuron training be a tensor, rather than a scalar positive or negative value?\nCan the sign of a single neuron's gradient represent the \"direction\" of overall knowledge?\nThe overall knowledge should be a non-linear combination of all neurons, so,  wouldn't a holistic view of all neurons better represent the knowledge?\n\n2. Regarding lines 138-140 (\"These findings suggest that although the knowledge learned by agents may be diverse, their differences may cause different gradients for neurons.\"), what is the precise meaning of this statement? \n\n3. In lines 274-->236 (\"Gradient conflicts arise from the diverse observations and actions of individual agents.\") Observations and actions are inputs and outputs of the network. However, gradients depend not only on network input/output structure but also on the loss function. I recommend a more thorough derivation, as this information should also encompass elements of the MDP, such as state transitions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wSJN2qTiUS", "reviewer": "Reviewer_MPio", "summary": "This paper studies the gradient conflict in parameter-sharing networks and proposes a Gradient-based Parameter Sharing (GradPS) method to resolve futile neurons in the PS network. It dynamically creates multiple clones for each futile neuron. For each clone, a group of agents with low gradient-conflict shares the neuron’s parameters, which are updated according to the gradients of each agent group. GradPS performs better than the other parameters sharing (PS) methods for the SMAC and predator-prey benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for viewing our work as interesting. Thanks for your valuable comments regarding the novelty of the futile neuron phenomenon and neuron gradient conflicts in MARL. We will improve our work based on your suggestions. We address your concerns as follows. \n\n## Experimental:\n\n>1. How do you determine the futile threshold?\n\nWe perform a simple parameter sensitivity experiment to determine the futile threshold $\\alpha$. The experimental results for SMAC (5m_vs_6m and MMM2) and Predator-Prey are shown as follows. \n\n|Env.| $\\alpha$=0.1 | $\\alpha$=0.2 | $\\alpha$=0.4\n|--|--|--|--|\n|5m_vs_6m(win rate)|72.3|74.5|67.6|\n|MMM2(win rate)|78.3|82.6|67.6|\n|Predator-Prey Medium(return)|167.4|173.9|178.8|\n\nThe sensitivity experiment shows that the performance for $\\alpha=0.2$ is good. Thus, we fix the futile threshold of all the experiments as 0.2. It is possible that such a threshold should be flexible for different environments. Moreover, better thresholds, such as deviation from the average futile ratio, could be used. We want to explore these in the future.\n\n>2. ...the computational overhead of this method...\n\nIn the appendix, the execution time (wall-clock training time) is shown as a ratio over the FuPS approach. The experimental results show that the additional computational overhead brought by GradPS is small; it is only slightly higher than FuPS. The other methods (e.g., SePS, NoPS) require significant computational overhead. \n\nIn the following table, we present the wall-clock training time (in hours) for each algorithm, averaged over five random seeds. GradPS achieves lower computational overhead than other parameter-sharing methods (SNP, NoPS). \n\n|Execution time（h)|FuPS|GradPS|FuPS+id|SNP|SePS|NoPS|Kaleidoscope|\n|--|--|--|--|--|--|--|--|\n|Predator-Prey Small(1M)|2.45|2.74|2.55|2.72|5.23|5.45|4.37|\n|Predator-Prey Medium(1M)|5.11|5.65|5.23|5.34|6.86|7.18|6.32|\n|Predator-Prey Large(1M)|5.89|6.32|5.84|6.62|9.92|11.65|9.05|\n|5m_vs_6m(2M)|9.21|9.74|9.33|10.64|12.56|14.21|11.86|\n|MMM2(2M)|12.94|13.83|13.02|14.95|18.42|19.34|15.53|\n|27m_vs_30m(2M)|23.12|24.75|23.45|25.77|36.53|42.32|31.45|\n\n## Weaknesses:\n\n>1.Analysis of the futile neuron threshold $\\alpha$ is missing.\n\nWe have discussed $\\alpha$ in the above response.\n\n>2. The algorithm lacks sufficient theoretical analysis.\n\nWe employ a simple yet effective method to mitigate gradient conflicts while encouraging future research to develop more solutions to gradient conflict issues in MARL. We will perform an in-depth theoretical analysis of GradPS and other parameter-sharing methods.\n\n## Suggestions\n\n>1. Under what circumstances does gradient conflict usually occur? If agents share a common goal...\n\nThanks for raising this question; we will discuss it in the discussion section. Gradient conflicts commonly occur in multi-agent systems. Even if agents share a common goal, they may have different partial observations, be located in different positions of the game, and their actions may be different, which may lead to different agent gradients (e.g., divergent actions). For example, when capturing one prey, it is possible that the best action for agent 1 is to move up, while the best action for agent 2 is to move down. The optimal actions for these two agents are different; this will cause gradient conflict, too. \n\nTo verify the above analysis, we train two predators to capture one prey in a 5 $\\times$ 5 map, and the spawn point of the agents and the prey is fixed. At each time, there is only one prey. Experiments show that the proportion of futile neurons is about 20%, which suggests that there are gradient conflicts, even if the two agents are pursuing the same prey. \n\n>2...situations where this approach is not effective...\n\nIn a highly stochastic environment, SMACv2, where agent types are randomized each episode, GradPS fails to deliver performance improvements as the gradient conflict patterns vary due to the randomness of agents themselves.", "claims_and_evidence": "This paper is well-motivated. This paper shows the futile neuron phenomenon in MARL and the neuron gradient conflict of neurons by experiment on SMAC. I think the proposed method is interesting.", "methods_and_evaluation_criteria": "The proposed method GradPS addresses the challenge of gradient conflicts in parameter-sharing (PS) networks for MARL. By dynamically cloning futile neurons and grouping agents with low gradient conflicts, GradPS effectively balances the trade-off between sample efficiency and policy diversity.\n\nOverall, the method and evaluation are well-designed.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "This paper conducts evaluations on SMAC and predator-prey benchmarks. Experiments show that the proposed method outperforms the state-of-the-art PS methods. Ablation studies validate parameter sensitivity.\n\nSome issues:\n\n1. How do you determine the futile threshold $\\alpha$ ?\n\n2. I am concerned about the computational overhead of this method. While I noticed the Execution Time Table in the appendix, the paper does not provide any explanation for it.", "relation_to_broader_scientific_literature": "This paper tackles the issue of policy homogeneity in parameter-sharing networks for multi-agent reinforcement learning from the angle of neuron gradient conflict among agents. This paper highlights the futile neuron phenomenon and the neuron gradient conflict of neurons in MARL, which have been explored in multi-task learning but not in MARL previously.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths \n+ Originality: This paper studies the gradient conflict among agents, a concept not previously explored in depth in MARL. The proposed gradient-based PS method is novel and addresses the policy homogeneity in parameter-sharing networks from the angle of neuron gradient conflict among agents. \n+ Significance: GradPS can learn diverse behaviors through multiple clones to avoid gradient conflict, and enjoys good sample efficiency by sharing the gradients among agents of the same clone neuron.\n+ Clarity: The paper is well-structured and well-written. \n## Weaknesses\n+ Analysis of the futile neuron threshold $\\alpha$ is missing.\n+ The algorithm lacks sufficient theoretical analysis.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "+ Under what circumstances does gradient conflict usually occur? If agents share a common goal, such as pursuing the same prey, is there still gradient conflict or futile neuron? \n+ Could you discuss situations where this approach is not effective or does not provide performance gains?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YDsuY1WssP", "reviewer": "Reviewer_P57Y", "summary": "This paper investigates parameter sharing techniques in cooperative multi-agent reinforcement learning (MARL). The authors observe gradient conflicts among multi-agent policies and propose a new partial parameter-sharing scheme. This method exhibits superior performance on benchmarks such as SMAC and PredatorPrey.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and suggestions, your comments are very important to us. We will improve our work based on your suggestions. We address your concerns as follows. \n\n ## Methods: \n>1. ... visualize the group assignment pattern...\n\nThank you for your suggestion. It is very valuable, and we will include it in the revised manuscript. The following table shows an example of the grouping of **a neuron** in the 5m_vs_6m environment. Before 0.5M steps, the neuron is not futile. At 0.6M timesteps, the neuron is futile, and the agents for the neuron were grouped into (1,2,3) and (4,5). The grouping is restored at 1.2M timesteps. At 1.6M timesteps, the agents were grouped into (1,3) and (2,4,5). \n\n|Timesteps|0.2M|0.4M|0.6M|0.8M|1M|1.2M|1.4M|1.6M|1.8M|2M|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Neuron|/|/|123/45|123/45|123/45|/|/|13/245|13/245|13/245|\n\n>2. ...report the training time and memory usage.\n\nAppendix D.6 shows the number of parameters (including factorization networks) used by different methods. We provide more results as follows.\n\nWe present the wall-clock training time (in hours) for each algorithm, averaged over 5 random seeds. The computational overhead of GradPS is small. It takes less training time than SNP, SePS and NoPS, most of the time.\n\n\n|Time（h)|FuPS|GradPS|FuPS+id|SNP|SePS|NoPS|Kaleidoscope|\n|--|--|--|--|--|--|--|--|\n|Predator-Prey Small|2.45|2.74|2.55|2.72|5.23|5.45|4.37|\n|Predator-Prey Medium|5.11|5.65|5.23|5.34|6.86|7.18|6.32|\n|Predator-Prey Large|5.89|6.32|5.84|6.62|9.92|11.65|9.05|\n|5m_vs_6m|9.21|9.74|9.33|10.64|12.56|14.21|11.86|\n|MMM2|12.94|13.83|13.02|14.95|18.42|19.34|15.53|\n|27m_vs_30m|23.12|24.75|23.45|25.77|36.53|42.32|31.45|\n\nThe following table depicts the number of parameters for each agent network compared to the full parameter-sharing approach. The additional memory overhead of GradPS is lower than that of SePS and NoPS. \n\n|Parameters|FuPS|GradPS|FuPS+id|SNP|SePS|NoPS|Kaleidoscope|\n|-|-|-|-|-|-|-|-|\n|Predator-Prey Small|100%|145%|102%|123%|198%|624%|218%|\n|Predator-Prey Medium|100%|146%|103%|124%|197%|1034%|217%|\n|Predator-Prey Large|100%|147%|104%|124%|196%|1540%|216%|\n|5m_vs_6m|100%|140%|102%|116%|297%|519%|226%|\n|MMM2|100%|176%|102%|128%|296%|1036%|218%|\n|27m_vs_30m|100%|195%|105%|136%|293%|2762%|215%|\n\n\n>3. ...$\\alpha$ sensitivity analysis...\n\n$\\alpha$ controls how futile neurons are identified. We have performed a sensitivity analysis for three environments; their results are depicted as follows.\n\n\n|Env.|$\\alpha$=0.1|$\\alpha$=0.2|$\\alpha$=0.4|\n|-|-|-|-|\n|5m_vs_6m(win rate)|72.3|74.5|67.6|\n|MMM2(win rate)|78.3|82.6|67.6|\n|Predator-Prey Medium(return)|167.4|173.9|178.8|\n\nThe results show setting $\\alpha=0.2$ is a good choice. We agree that such a threshold may be environment-dependent, and we would like to explore it in the future. \n\n## Questions: \n#### Method:\n> ...How gradient conflict retained..., introduce significant overhead...\n\nThe accumulated gradients are stored in a tensor (T $\\times$ agents $\\times$ dim), representing the historical gradient values of neurons across each agent over the past T periods. These saved gradients are used to compute neuron efficiency, not for back-propagation. It introduces little computational overhead. Moreover, we have shown in the above parameter table that the memory overhead of GradPS is low.\n\n#### Experiment:\n>1. ...higher futile neuron percentages do not seem to correspond to lower performance. ...\n\nWe appreciate your insight. MARL performance is influenced by multiple factors, and futile neuron is just one of them. We agree that higher neuron percentages do not necessarily lead to lower performance. \n\n>2. ... a cost comparison...\n\nWe have already shown the execution time and memory overhead in the above table.\n\n>3. ...relationship between the clone ratio and the heterogeneity..\n\nThe cloning ratio $K$ is not directly correlated with agent heterogeneity. $K$ is a neuron-level value, it is not an agent-level value. We will explore whether the aggregate $K$ of all the neurons can represent agent heterogeneity. \n\n>4. ...$\\rho$  does not impact performance, though it seems to tie to the computation overhead...always set a large $\\rho$\n\n$\\rho$ is the probability of restoring the parameters of a neuron from grouped to shared status. We have conducted experiments that set $\\rho$ to a large value (from 0.05 to 0.5), and we find that increased $\\rho$ does not impact the computational overhead significantly. \n\n|Time(h)|$\\rho$=0.05|$\\rho$=0.2|$\\rho$=0.5|\n|-|-|-|-|\n|5m_vs_6m|9.74|9.66|9.68|\n|MMM2|13.83|13.64|13.94|\n\nGradual parameter recovery requires sufficient time to prevent rapid performance degradation; thus, a large $\\rho$ may not be the optimal choice, as shown in the following table.\n\n|Env.|$\\rho$=0.05|$\\rho$=0.2|$\\rho$=0.5|\n|-|-|-|-|\n|5m_vs_6m(win rate)|74.5|73.2|63.6|\n|MMM2(win rate)|82.6|80.2|57.6|", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "1. The motivation is clear and well-supported by the toy example in Figures 1-3.\n2. The experiments demonstrate the superior performance of the proposed GradPS compared to baselines on benchmarks.\n\nOverall, the experiments are thorough enough to support the claims made in this paper. However, I would like to suggest a few additional experiments that could potentially strengthen the paper even further:\n1. It might be insightful to visualize the group assignment pattern throughout the training process. This could offer valuable insights for future research on PS in MARL.\n2. Considering that cloning and optimizing weights separately may introduce some overhead, it would be helpful to report the training time and memory usage. \n3. \\alpha also appears to be a quite important hyperparameter. it would be beneficial to include a parameter sensitivity analysis for this variable.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "As adequately discussed in the related work section, this work bridges the neuron learning efficiency and MARL.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "This paper proposes a novel PS technique for MARL. The method is well-motivated by the observation that MARL often experiences gradient conflicts among agents, causing futile neurons and leading to degraded performance. The proposed method is clearly explained and well-supported by experiments. However, it might introduce extra overhead, and some experimental results may need further explanation (see the Question Section for details).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Method-related:\n1. How is gradient conflict retained during the interval T? Will this introduce significant overhead in terms of time and space?\n\nExperiment-related:\n1. From the analysis in Section 4, fewer futile neurons should lead to better expressiveness and performance. While the proposed GradPS provides the best performance and the lowest futile neuron percentage, baselines with higher futile neuron percentages do not seem to correspond to lower performance. Are there any possible reasons for this?\n2. Could the authors provide a cost comparison (training time and memory requirements) for the proposed methods and baselines?\n3. Is there any pattern or relationship between the clone ratio and the heterogeneity of the agents?\n4. The authors state that \\rho does not impact performance, though it seems to tie to the computation overhead. In this case, is it safe to assume that we should always set a large \\rho?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a new method for improving parameter sharing in MARL by improving diversity in the shared network. It does this by cloning and updating 'futile' neurons that have conflicting gradients when updated by different agents. 

There are some concerns about the significance and practicality of approach but it appears to be novel and beneficial. This line of work should be interesting to the community. 

The paper should be improved to clarify details about the method and experiments. The author response was helpful in this regard, but the paper should be updated with the additional details and additional experiments.
Decision: Accept (poster)