Bi-Level Dynamic Parameter Sharing among Individuals and Teams for Promoting Collaborations in Multi-Agent Reinforcement Learning
Authors: Yan Liu
Ying Tiffany He
Fei Yu
Zhong Ming
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-Agent Reinforcement Learning
Reinforcement Learning

Abstract: Parameter sharing has greatly contributed to the success of multi-agent reinforcement learning in recent years. However, most existing parameter sharing mechanisms are static, and parameters are indiscriminately shared among individuals, ignoring the dynamic environments and different roles of multiple agents. In addition, although a single-level selective parameter sharing mechanism can promote the diversity of strategies, it is hard to establish complementary and cooperative relationships between agents. To address these issues, we propose a bi-level dynamic parameter sharing mechanism among individuals and teams for promoting effective collaborations (BDPS). Specifically, at the individual level, we define virtual dynamic roles based on the long-term cumulative advantages of agents and share parameters among agents in the same role. At the team level, we combine agents of different virtual roles and share parameters of agents in the same group. Through the joint efforts of these two levels, we achieve a dynamic balance between the individuality and commonality of agents, enabling agents to learn more complex and complementary collaborative relationships. We evaluate BDPS on a challenging set of StarCraft II micromanagement tasks. The experimental results show that our method outperforms the current state-of-the-art baselines, and we demonstrate the reliability of our proposed structure through ablation experiments.

Under review as a conference paper at ICLR 2023
BI-LEVEL DYNAMIC PARAMETER SHARING AMONG INDIVIDUALS AND TEAMS FOR PROMOTING COL- LABORATIONS IN MULTI-AGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Parameter sharing has greatly contributed to the success multi-agent reinforce- ment learning in recent years. However, most existing parameter sharing mech- anisms are static, and parameters are indiscriminately shared among individuals, ignoring the dynamic environments and different roles of multiple agents. In addi- tion, although a single-level selective parameter sharing mechanism can promote the diversity of strategies, it is hard to establish complementary and cooperative relationships between agents. To address these issues, we propose a bi-level dy- namic parameter sharing mechanism among individuals and teams for promoting effective collaborations (BDPS). Specifically, at the individual level, we define virtual dynamic roles based on the long-term cumulative advantages of agents and share parameters among agents in the same role. At the team level, we combine agents of different virtual roles and share parameters of agents the same group. Through the joint efforts of these two levels, we achieve a dynamic balance be- tween the individuality and commonality of agents, enabling agents to learn more complex and complementary collaborative relationships. We evaluate BDPS on challenging set of StarCraft II micromanagement tasks. The experimental results show that our method outperforms the current state-of- siee-of-the-att baselines, and we demonstrate the reliability our proposed structure through ablation experiments.
INTRODUCTION
In many areas, collaborative Multi-Agent Reinforcement Learning (MARL) has broad application prospects, such as robots cluster control (Buşoniu al.) 2010), multi-vehicle auto-driving (Bhalla et al. 2020), and shop scheduling (Jiménez 2012). In multi-agent environment, agent should observe the environment's 's dynamics and understand the learning policies of other agents to form good collaborations. Real-world scenarios usually have large number of agents with different identities or capabilities, which puts forward higher requirements for collaborations among agents. Therefore, how to solve the large-scale MARL problem and promote to form stable and complemen- tary cooperation among agents with different identities and capabilities are particularly important.
To solve the large-scale agents issue, we can find that many collaborative MARL works adopting the centralized training paradigm use the full static parameter sharing mechanism (Gupta et al., 2017]. which allows agents to share parameters of agents' policy networks, thus simplifying the algorithm structure and improving performance efficiency. This mechanism is effective because agents generally receive similar observation information the existing narrow and simple multi- agent environments. In our Google Research Football (GRF) (Kurach et al., (2200) experiments, we can find that blindly applying the full parameter sharing mechanism does not improve the perfor- mance of algorithms because the observation information is very different due to the movement of different players. At the same time, because the full static parameter sharing mechanism ignores the identities and abilities of different agents, it constantly limits the diversity of agents' behavior policies (Li et al. 2022] Yang et al., (022). which makes it difficult to promote complementarity and reliable cooperation between agents in complex scenarios.

---

Under review as a conference paper at ICLR 2023
Recently, in order to eliminate the disadvantage of full parameter sharing, single-layer selective parameter sharing mechanism has been proposed (Christiamos et al. 2021; Wang et al. (022). that is, extracting deep features from agents' observation information by encoder and clustering them to achieve the combination of different agents in order to select different agents for parameter sharing.
Although the single-level selective parameter sharing mechanism can promote the diversity of agents' policies, it causes the relationship between agents that do not share parameters simulta- neously fragmented. So that agents cannot establish complementary cooperative relationships in a broader range. More importantly, designing an effective selector is the key to sharing selective parameters, especially for the single-level dynamic selective parameter sharing mechanism, which needs to be done within several rounds of the selection operation. Most methods only use the real- time observation information of agents, which loses attention to agents' history and is not conducive to correctly mining the implicit identity characteristics of the agents. For football team, special training for different players, such as shooting training for the forwards and defense training for the defenders, will be carried out. However, winning game requires special training and coordination between players of different roles. That is, not only do we need to share parameters with agents of the same role, but also need to combine agents of different identities to ensure that they can form robust and complementary collaboration on a larger scale.
To address these issues, in this paper, we propose bi-level dynamic parameter sharing mechanism among individuals and teams (BDPS). The advantage functions of agents can be expressed as the advantages of taking action relative to the average in the current state. We consider that the advantage function can better represent the actual roles of agents in current identity and grouping. In order to more accurately identify the role of agents, at the individual level, we compute the long-term advantage information of the agents as the key to virtual role identification and use the variational autoencoders (VAE) (Kingma & Welling 2014) to learn the distribution of the role characteristics of the agents, and obtain more accurate virtual role directly by sampling the distribution of role features. To alleviate the split of the relationship between different virtual role agents by the single- layer dynamic parameter sharing mechanism, we further use the graph attention network (GAT) (Velickovic et al., 2018) to learn the topological relationships between different roles based on the roles obtained at the individual level, so as to achieve the combination of different identity agents at a higher level and a broader range. Through the method we designed, we achieve dynamic and selective parameter sharing for agents at two different levels, individual and team, and achieve the goal of stabilizing complementary collaboration among agents in a more extensive scope while achieving the diversity of agents' policies.
We test BDPS and algorithms using different parameter sharing mechanisms on the StarCraft II mi- cromanagement environments (SMAC) (Samvelyan al., 2019) and the Google Research Football (GRF) (Kurach et al., 2020). The experimental results show that our method not only outperforms other methods with single-level selective parameter sharing mechanism and full parameter sharing mechanism in general on all super hard maps and four hard maps of SMAC, but also performs well in the experimental scenarios in the used GRF In addition, we carried out ablation experiments to verify the influence of different parameter sharing settings on the formation of complementary cooperation between agents, which fully proves the reliability of our proposed method.
2 BACKGROUND
2.1 DECENTRALIZED PARTIALLY OBSERVABLE MARKOV PROCESS
A full cooperative MARL task can usually be modeled as a decentralized partially observ- able markov process (Dec-POMDP) (Oliehoek & Amato 2016). We can define tuple M = SN A, R, P. P. 0,7) global state, and 7 E [0, 1). At each time step t, each agent i E A receives a local observation o E f according to the observation function O (s, i). takes an action a, A to form joint action E AA". and gets a shared global reward r = R (s. a). Due to the limitation of partial observability, each n; (a./T,
7) to represent it, where N is the finite set of n agents, s E S is finite set of
its own local action-observation history T; E T (Ω x A) Agents together aim to maximize the expected return, that is, to find a joint policy π = {T.,..., A.) to maximize a joint action-value function Q"
agent conditions its policy
Ess.. 00.00
1EE 7're|'n 1s0 8,00 a, TT.
2

---

Under review as a conference paper at ICLR 2023
2.2 CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION
In many MARL settings, partial observability problems can be solved by centralized training with decentralized execution (CTDE) paradigm (Lowe al. 2017 Foerster al.) 2018 Rashid al., 2020 Wang et al.] 2 Son et al. 2019). which is currently the mainstream of MARL methods. The centralized mode adopted training, and after the training, agents only make decisions based on their local observation and the trained policy network. Thus, the problems of unstable environment and large-scale agents can be overcome the same time to a certain extent.
2.3 VARIATIONAL AUTOENCODER
Variational autoencoder (VAR) (Kingma & Welling, 2014) is a generative network model based on Variational Bayesian (VB) (Fox Roberts (012) inference. Two probability density distribution models are established by the inference network 9o and the generative network pę. The inference network is used for the variational inference of the original input data æ to generate the variational probability distribution of hidden variables The generative network restores the approximate probability distribution of the original data according to the generated implicit variable variational probability distribution. VAE model can be divided into the following two processes: the approx- imate inference process of posterior distribution of hidden variables: go (z|æ) and the generation process of conditional distribution of generated variables: Pg (z) Pg (ø(z).
In order to make de ()) and the true posterior distribution Pa (z|zr) approximately equal, VAE uses KL-divergence to measure the similarity between them:
DKL(ge (z|z>) Pe (z|z)) Po (x) + (xg()()) [log qs (=|z) log (x, 2)]
where the term log (xr) is called log-evidence and it is constant. Another term is the negative evidence lower bound (ELBO). The VAE with additional GRU network is widely used in sequence anomaly detection (Su at. al. 2019)), In this paper, we use it to identify the identity feature distribution mapped behind the advantage information sequence of agents for a period of time better guide them in choosing appropriate parameter sharing partners.
2.4
GRAPH ATTENTION NETWORK
Graph attention network (GAT) (Velickovic et al., 2018) is a network architecture based on an at- tention mechanism, which can learn different weights for different neighbors through the attention mechanism. For calculating the output characteristics of node i, GAT first trains a shared weight matrix W for all nodes to obtain the weight of each neighbor node of node i. Then, according to the weight, the attention coefficients between node and its neighbor nodes are calculated. Finally, these attention coefficients weighted and summed to obtain the output features h of node i:
exp (02 (a" (Whallhs)) Whj
M = 01
Σ JEN.
EREN, exp (02 (aT [Wh; hx]))
(1- (1-Heud)
where di and d2 represent nonlinear functions, Nr represents the first-order neighborhood of node i, a" represents the transpose of the weight vector a and is the concatenation operation.
In recent years, MARL with communication has widely used GAT determine the explicit com- munication goals of agents (Niu et al., 2001]] [eraj et af.] (202)). In this paper, we use the GAT to establish the topological relationship between different virtual roles, and further combine the agents under different virtual role mappings to form complementary and reliable cooperative relationships.
3 OUR METHOD
In this section, we introduce the proposed BDPS in detail. BDPS mainly comprises individuals and teams, as shown in Figure[[ We get inspiration from people: person may play multiple roles and can freely switch roles in different scenes, which promotes the stable development of people, and choose sacrifice some network parameters for agents to
the same is true for agents. Therefore, maintain multiple roles and groups at the same time. And according to the role selector U and the group selector V. choose the most suitable role and group for the agents.
3

---

Under review as a conference paper at ICLR 2023
8ole
e²
Rol Natworks
Indivdiull
A 2
GRUM
De
GRU
Mirg
Hole Selector
Gropp
Group
MLP
GRUP
MLP
Group Networks
Team Level
Broody
dr
Doodde
R... M
GRU
GAT
Dity GRU Call
GRU Netwook
Selector
Figure 1: The overall BDPS architecture.
First, we explain the relationship between agents and individual virtual roles and groups.
Given n (n22) agents, k ((25k<n) roles and m (25m5k) groups, we have the following mapping relationships: fo :N N K and fy + M, where K. and M represent finite sets of roles and m groups, respectively.
Definition
From the Jv, we can the agents' groups depend on agents' roles, we give the time relationship for agents to update the groups and roles: Ty e Tu. where Tv represents the update period of the group, Ty represents the identity's 's update period and Z
3.1 INDIVIDUAL, LEVEL DESIGN
Different from the existing methods of determining the roles of agents based on real-time observa- tion information, we consider the impact of the long-term cumulative advantages of agents on the definition of the agents' roles, because the advantage information can better represent the current state of agents than the observation information. At the same time, unlike the existing methods that only use the encoder to learn the role characteristics agents for clustering, we learn the long-term advantages of agents through the VAE to obtain the distribution information of role. Because in many cases, the role an agent will not change just because makes a unique action. At the in- dividual level in Figure [ first maintain the role selector U select appropriate roles uy for agents, then apply parameter sharing to agents according to the same role.
3.1.1 CHOOSE APPROPRIATE ROLES FOR AGENTS
Firstly, the role selector U inputs the pre-calculated agents' advantage information sequence Ar-Test-1 = {A.-Te: A.-Tu+I, A.-13 into GRU network to capture complex temporal dependence between agents' advantage information in the virtual roles' update period. Secondly, when the time meets the periodic condition of updating virtual roles, we take the implicit output in- formation 1 at this time as the characteristics of the agents and input it into the Encoder Eg of the VAE. Finally, we obtain the virtual roles uy of agents from the k-dimensional Gaussian distribution output by the Encoder E.-
For agent its advantage information at time t can be obtained by A max (Qi) Q, where Q represents the agent i's local Q-function time f, and unlike the classical calculating advantage function = Q V, ours A; 2 0 is convenient for us to use the Decoder De to reconstruct the input information. When the virtual role ub of agent needs to update at time t, its virtual role can be obtained from the VAE's bottleneck:
u )
(3)
where zy JAU dg eg, ep N T).
4

---

Under review as a conference paper at ICLR 2023
3.2 TEAM LEVEL DESIGN
The purpose of introducing grouping is to eliminate the fragmentation of the relationship between agents caused by the single-layer parameter sharing mechanism, and establish the cooperative relationship between agents different roles in a wider range. As shown in Figure [ the team level mainly composed of the group selector V and grouping cooperative networks. The group selector V is realized through an additional reinforcement learning task. We introduce GAT for this task to find the correlation between different dynamic roles and promote the role composition in the grouping results to be more comprehensive and complete. In the part of parameter sharing, we add additional implicit information hy from the individual role level to give full play to the positive impact of the individual identities on the team results.
3.2.1 COMBINE DIFFERENT ROLES TO ACHIEVE GROUPING
The group selector takes the virtual roles uy as inputs. This input information is encoded into the feature vector of the agents via multi-layer perceptron (MLP). Subsequently, these feature vectors form a set a "e.,.".,,"""" to be input into the GAT. We use the self-attention mechanism in the GAT to calculate the attention coefficient Bij between agents and use it as an essential basis for grouping:
ejj LeakyReLU (a" Wa weellaa
(4)
where i E N. In our method, considering the similar observation information and close location arrangement between agents, and there no rigid separation restriction between roles of different agents, we allow the agent's attention to come from all undead agents (including itself) at the current moment. Of course, to facilitate the comparison of the attention coefficient between agents, we also use the softmax function for normalization.
K 11o
x
Σ JEN;
oi W x/
(5)
k=
where k represents the number of attention heads, which is consistent with the number of virtual roles we set. We hope to find the influence of different agents' roles as the dominant factors on the formation of teams by agents. For other parts in Equation [ Mr represents the first-order neigh- borhood of agent i, and 0g = softmax, (esz) represents the normalized attention coefficient that indicates the importance of agent j's features to agent
new set of feature vectors 2; = ={ """,,,,,""""" x." """""""""""...
is obtained after the GAT, which is input into the GRU network to get the hidden state affecting the grouping of agents. When the time increment is equal to the period for agents to update groups Tv. we use the hidden state to output the groups' value and use the E gredddy select groups By for agents.
3.3
OVERALL ORJECTIVES
3.3. GET THE LOCAL Q R-FUNCTIONS REQUIRED BY AGENTS
In previous sections, use the role selector U the individual level and the group selector V the team level to map agents from actual individuals to virtual roles uy and groups vv, respectively. The bi-level parameter sharing mechanism is also established for agents using roles and groups information obtained. Although contributions from hidden states at the individual level are used in calculating the local Q-functions of agents at the team level, still may not completely discard explicit efforts at the individual level of agents. So we give the local Q-function:
Q (T. a) = Q" (T, a) + QV (T.a).
(6)
3.3.2 TRAIN THE MODULES INCLUDED IN OUR METHOD
Starting from the individual level, we need to train the role selector U to select appropriate virtual roles for agents. The training objectives of the Role Selector U include the corrected reconstruction item and the KL-divergence item, and our goal is to minimize these two items:
Lu (E. 05 Av) = Lmme (Au: AU) + ADxI. [go (Zty (NN llpe (21o )]
(7)
5

---

Under review as a conference paper at ICLR 2023
where Lise (:) is the mean squared error term for calculate the reconstruction loss of the advan- tage sequence, ly is the length of the Au, À is scaling factor and Au is the agents' advantage information reconstruction sequence, which is obtained by the Decoder's output huy
For the group selector V at the team level, we used the QMIX (Rashid et al. 2018) and RODE (Wang et al. 2021b) methods because we introduced an additional deep reinforcement learning task to generate groups for agents:
Te yaaxo Qia (T' d', uyy 5') QYos (T, a, uc,s) )
21
Lv (0,)
Σ Σ
(8)
where b the batch size of transitions sampled from the replay buffer.
In order for agents to global rewards to learn local Q-functions, input the local Q-value into the mixing network of QMIX (Rashid et al., 2018) again to estimate the global action value Qtat (T,
(TD(Ow==>)
E (r + +maxx Qtor ('',',8) - Qroe (T, a, s)))
4 EXPERIMENTS
In this section, demonstrate and evaluate the advantages of proposed BDPS using the chal- lenging tasks in the StarCraft II micromanagement enviroments (SMAC) and Google Research Foot- ball (GRF). We not only compare our proposed method with QMIX (Rashid al., 2018) which adopts full parameter sharing mechanism and no parameter sharing mechanism, but also further compare with several baseline methods aimed at promoting the diversity of agents' policies, such as CDS (Li et al. 2021), EOI (Jiang & Lu, (202)) and RODE(Warr et al.) (202b). Finally, we ablate our method in SMAC, verifying the true utility of the components in our method.
4.1 PERFORMANCE ON GOOGLE RESEARCH FOOTBALL (DEC-POMDP)
In the official multi-agent example, agents are allowed to observe all information on the field, which contrary to our research hypothesis. We provide an observation space setting for agents in the Dec-POMDP version. The observation space of agents dynamically change with their motion vectors. See Appendix A.2 for details.
First, we compare method and baseline methods in the GRF of the Dec-POMDE version provide. Compared with the experimental scenario provided by CDS (Li et al. 2021). we use the algorithm to control all agents the left team, the built-in scoring and checkpoints reward settings of GRE, and set that the agents in the left team cannot be moved by the built-in Al of the system when they do not touch the ball.
As shown in Figure 2 we select two scenarios Loademy_3_es___ waw.1_wtthheeeeeee and academy-per paannngppsss.... saaosppasssdddddaeeeee www.dshhodddo....oooooooo wwn._shhot.uttthkkee... withhkeeper for comparison. We compare the actual reward re- ceived by the agents with the goal score in both scenarios. In general, our method is able to achieve better results than other baseline algorithms in both scenarios. Among them, the full parameter sharing version of QMIX is better than the non-sharing version, and other improved baseline algorithms based on full parameter sharing QMIX can also achieve goals, but the effect is not obvious.
As we mentioned in Section [ the full parameter sharing mechanism ignores the identities and capabilities of agents, which makes them lose the diversity of behavior policies. As a result, full parameter sharing cannot improve the effectiveness with large differences in information. For the single-layer dynamic selective parameter sharing, due to the hard cutting of the relationship between agents that do not participate in parameter sharing, although better results can be achieved by relying on the selective parameter sharing mechanism, the agents cannot form a sufficiently stable cooperative relationship, which makes the effect fluctuate significantly. See Appendix for the specific results.
the algorithm when agents observe scenes

---

Under review as a conference paper at ICLR 2023
academy with keeper
academy and shoot with keeper
Test retummm mean
-1.0
9.0
10.0
Tran stops (M)
Train Stops (M)
academy, vs With keeper
academy pass, and shoot keeper
Test score reward
0.9
0.6
0.6
-0.3
-0.3
-0.6
-0.6
0.0
1.0
2.0
30
4.0
5.0
6.0
7.0
8.0
9.0
10.0
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
Ours
GMIX/UU)
(MiXinn)
Figure 2: Performance comparison with baselines on Google Research Football.
27m, vs, v5.30m
Gh, 8z
corridor
100
100
100
Textt win rate (%0
80
40
40
20
20
20
0
1.0
Train sllepe (M)
Train slepe (M)
Train sleps (M)
MMM2
vo__v__44gg
Toaz win rata (%)
100
100
100
60
-0
40
0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
2.0
Train
Train
Ours
CDS
EDI
QMIX/UI)
RODE
QWiX(no)
Figure 3: Performance comparison with baselines on three super hard maps and three hard maps. we show results the whole benchmark.)
(In Appdendix E.
In both scenarios, we can see that RODE differentiated the roles of the agents by limiting their action space, but this limitation also prevented the agents from achieving further training results. Through the experimental results, we can find that these baseline algorithms drop the ball in these two acudemyy-scenriio, indicating that the agents do not form a sufficiently complementary collab- orative relationship.
4.2 PERFORMANCE ON STARCRAFT II
As can be seen from Figure 3 our method is superior to other baseline methods on these maps, es- pecially in maps with larger number of agents, where our method is able to maintain its advantage. Of course, EOI and RODE also show good stability these maps, and the full parameter sharing version of QMIX is still better than the no parameter sharing version.
Specifically, our method has more advantages than the CDS, which emphasizes the balance of per- sonalities and commonalities of agents. As we mentioned, exploring the dynamic balance between individual personalities and commonalities is essential. Still, this balance is difficult to define, so we do not give factors in Equation O that adjust the different importance at the individual and team levels. This balance is dynamic and hides in the process of sharing dynamic parameters we design. We will describe it in detail in the ablation experiment in Section 4.3

---

Under review as a conference paper at ICLR 2023
4.3 ABLATION STUDY
In this section, we conduct ablation studies to understand the actual utility of each level in our bi- level dynamic parameter sharing mechanism. In addition showing the winning rate of maps in SMAC, we calculate the entropy difference between individuals and teams to quantify the advan- tages of different components.
100
100
Ours
Ours (K- 3)
Text win rate (1)
80
(only Ind.)
Toxt win rate (%)
80
Ours (5-
60
QMX (ful share)
60
OMI (no share)
Ours (k
0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
stops (M)
stops (M)
Figure 4: Comparison of different parameter sharing mechanisms in 5m_vs_6m.
Figure 5: Comparison of different virtual roles in MMM2 (Only individual level).
We first the homogeneous agents' map 5m 6m in SMAC conduct ablation research to analyze the advantages of dynamic parameter sharing over full and no parameter sharing. As shown that the method of applying parameter sharing has apparent advantages
in Figure 4 over the method without parameter sharing. Of course, our method is better than QMIX's full static parameter sharing. Specifically, the single-level dynamic parameter sharing has a weak advantage over the full static parameter sharing. Our bi-level dynamic parameter sharing is more advantageous than the single-level dynamic and full static, which is exactly the goal of our proposed method.
we also compare the impact of different numbers of virtual roles on the collaboration of agents in the heterogeneous agents' map MMM2. From the curve the figure, we can find that the winning rate in the MMM2 map increases first and then decreases with the number of virtual roles, which indicates that blindly increasing the number of virtual roles can adversely affect the collaboration of agents.
As shown in Figure 5
To quantify the difference between our single-level and bi-level dynamic parameter sharing, we consider analyzing from the perspective of information theory. First, we introduce the evaluation indicator: Entropy difference.
Entropy describes uncertainty. We describe the capabilities of agents in different layers by compar- ing the entropy of action value between the two layers.
1 _
Al
Σ pl log a2
Σ pd log
(10)
Piy
PU
where, pt = softmax (r', a: u and pú = softmax Qu (r),at_1))
Emiogy afference (hatg)
0.1
5m vs. Bm
Entropy dffleence (rntt))
0.05
MMV2
AH (k- 5)
Atd 4)
-0.05
-0.2
AM (K 2)
-0.1
-0.
-0.15
-0.2
0.0
1.3 Train stops (M)
2.0
1.0 Tran stops (M:
2.0
Figure Comparison of the entropy difference of two levels in 5m 6m and MMM2.
As shown in Figure 6 we can that in 5m_vs_6m, the information entropy the team level is always smaller than that at the individual level, and the difference between the two is increasing. The change of entropy shows that the agents the team level are more orderly than strategies
8

---

Under review as a conference paper at ICLR 2023
formed by agents only at the individual level, which also confirms that team-level design find commonalities among agents with different virtual roles to establish correct collaborations between agents. For MMM2 in Figure our conclusion is still valid. No matter whether & = 3 or k = 5, the entropy difference between the team level and the individual level is constantly expanding with the training, and the entropy of the team level is constantly developing in direction smaller than that of the individual level. And this phenomenon is consistent with our winning rate in Figure 4 and Figure
RELATED WORK
5.1 PARAMETER SHARING
Parameter sharing plays an important role in MARI.. Tan (1933) first studied the positive role of "sharing" in promoting collaborative agents in classical reinforcement learning algorithms. Gupta et al. (2017) proposed a parameter sharing variant of the single agent DRL. algorithms, which intro- duced the parameter sharing mechanism into homogeneous MARL. Obviously, parameter sharing has been widely used as the implementation details of homogeneous multi-agent algorithms,, such as QMIX (Rashid et al. 2018), QTRAN (Son et al. 2019), Qatten (Yang et al.) 2020). etc. Re- cently, Terry et al. (2020) applied parameter sharing extensions to heterogeneous agents algorithm by padding based method, demonstrating again the important use of parameter sharing for multi- agent algorithms. Of course, recent works have pointed out that full parameter sharing mechanism tend to make the behavior strategies of agents more same, Christianos et al. (202I) proposed a se- lective parameter sharing mechanism to eliminate the limitations of full parameter sharing.
5.2 INDIVIDUALS AND TEAMS
Focusing on the expected development of individuals and teams will not only help agents to maintain a variety of policies but also help them form a more stable collaboration. At the individual level, Wang et al. (2020 2021b) focus on discovering the character traits behind the agents. Diang & Lu (2021) studies the identifiability of agent trajectories and fixed identities. [u al. (2019) proposes the use of internal rewards to stimulate diverse behaviors among agents. At the team llbbll et al.)
(20T)) randomly groups agents into related and unrelated groups, allowing agents to explore only specific entities in their environment. |Wang et al. (2022) implements dynamic grouping method by extracting the potential intentions of agents as tags. Li et al. (2021) divides agents' value functions into shared and unshared parts focus on the agents' personality and commonality. In this paper, we focus on the impact of both individual and team levels on the complementary collaboration of agents. Of course, our approach is not to overlap the two layers of individuals and teams but to fully extend and combine the individual identities acquired by the agents.
6 CONCLUSION AND FUTURE WORK
In this paper, we proposed BDPS, a novel bi-level dynamic parameter sharing mechanism in MARL. By maintaining role selector and a group selector, BDPS provides a new solution for agents to select the partners for parameter sharing in timely and dynamic manner both individual and team levels. And we integrate the roles and groups of agents into whole to achieve dynamic balance between their personalities and commonalities. Our experiments on SMAC and GRF show that BDPS can significantly facilitate the formation of complementary and reliable collaboration between agents.
Additionally, although we defined that agents have the opportunity to be in any role or grouping, they cannot explicitly utilize past knowledge when they were in other roles or groups Therefore, how to make explicit use of the role knowledge they have learned and how to make use of other similar knowledge are issues that we need to explore further. We will present this paradigm of knowledge flow across roles, groups, and agents in future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Stefano Albrecht and Subramanian Ramamoorthy, A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems. arXiv preprint arxiv: 2015.
Stefano Albrecht and Peter Stone. Reasoning about hypothetical agent behaviours and their pa- rameters. arXiv preprint arXiv: /906. 1064, 2019.
Sushrut Bhalla, Sriram Ganapathi Subramanian, and Mark Crowley. Deep multi agent reinforcement learning for autonomous driving. In Canadian Conference on Artificial Intelligence, volume 12109, pp. 67-78. Springer, 2020.
Lucian Buşoniu, Robert Babuška, and Bart De Schutter. Multi-agent reinforcement learning: An overview. Innovations in Multi-Agent Systems and Applications-I, pp. 183-221, 2010..
Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano Albrecht, Scal- ing multi-agent reinforcement learning with selective parameter sharing. In International Confer- ence on Machine Learning, volume 139, pp. 1989-1998. PMLR, 2021.
Yali Du, Lei Han, Meng Fang, Liu, Tianhong Dai, and Dacheng Tao. Lüir: Learning individ- ual intrinsic reward in multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 32, pp. 4405 4416,4416,222 2019.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Charles W Fox and Stephen Roberts. tutorial on variational bayesian inference. Artificial intelligence review, 38(2):85-95 2012.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In Autonomous Agents and Multiagent Systems, volume 10642, pp. 66-83. Springer, 2017.
Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wist factorization for agett reinforcement learning. In Pro- ceedings the 38th International Conference on Machine Learning, volume 139, pp. 4596-4606. PMLR, 2021.
Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 4992-909400 5001. PMLR, 2021.
Yailen Martínez Jiménez. A generic multi-agent reinforcement learning approach for scheduling problems. PhD, Vrije Universiteit Brussel, 128, 2012.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. In 2nd International Con-
ference on Learning Representations, 2014.
Karol Kurach, Anton Raichuk, Piotr Staúczyk, Michal Zajkac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4501-4510. 2020.
Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Cele- brating diversity in shared multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3991-4002, 2021.
Ryan Lowe. Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-compeittiy environments. Advances in Neural Infor- mation Processing Systems, pp. 6379-6390, 2017.
Yaru Niu, Rohan R Paleja, and Matthew C Gombolay. Multi-agent graph-attention communication and teaming. In International Conference on Autonomous Agents and Multiagent Systems, pp- 964-9733 2021.
10

---

Under review as a conference paper at ICLR 2023
Frans A Oliehock and Christopher Amato. A Concise Introduction Decentralized POMDPs. Springer, 2016.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International Conference on Machine Learning, pp. 4295-4304 PMLR, 2018.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi- reinforcement learning, Advances in neural information processing systems, 33:10199- 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference Autonomous Agents and MultiAgent Systems, pp. 1186-2188, 2019.
Esmaeil Seraj, Zheyuan Wang, Rohan Paleja, Matthew Sklar, Anirudh Patel, and Matthew Gombo- lay. Heterogemeous graph attention networks for learning diverse communication. arXiv preprint arXiv:2108.09566 2021.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning, volume 97, pp. 5887-5896. PMLR, 2019.
Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2828 2837, 2019.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In International Conference on Machine Learning, pp. 330-337. PMLR, 1993.
Justin K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXi::2005.13625, 2020.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, 2018.
Han Wang, Yang Yu, and Yuan Jiang. A cooperative multi-agent reinforcement learning algorithm based dynamic self-selection parameters sharing. Chinese Journal of Intelligent Science and Technology, 4(1):75, 2022.
Jianhao Wang, Zhizhou Ren, Terry Liu. Yang Yu, and Chongjic Zhang. QPLEX: duplex dueling
multi-agent q-learning. In 9th International Conference on Learning Representations, 2021a.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi aalti-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference Machine Learning, volume 119, pp. 9876-9886. PMLR, 2020.
Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng. Shimon Whiteson, and Chongjie Zhang Rode: Learning roles to decompose multi-agent tasks. 9th International Conference on Learn- ing Representations, 2021b.
Mingyu Yang, Jian Zhao, Xunhan Hu, Wengang Zhou, and Houqiang Li. Ldsa: Learning dy- namic subtask assignment in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2205.02556, 2022.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv: 033999 2020.
11

---

Under review as a conference paper at ICLR 2023
A MULTI-AGENT ENVIRONMENTS
In this paper, we use three multi-agent environments, Stareraft Il Multi-agent Challenge (SMACySanvelyan et al., 2019). Google Research Football (GRF) (Kurach et al. (020) and Level- Based Foraging (LBF) Papoudakis et al. (2021); |Albrecht & Ramamoorthy (2015); Albrecht & Stone (2019), to conduct verification experiments.
A.1
SMAC
The starcraft multi-agent challenge is a fully collaborative, partially observable set of multiagent tasks. This environment implements various aacro-managemenn tasks based on the popular real- time strategy game StarCraft II. Each mission is a specific battle scenario in which group of agents, each of whom controls single unit, fight against army controlled by the built-in Al in StarCraft II.
Compared with the LBF, SMAC provides abundant battle scenes for agents. We conducted comparative experiments with baseline methods on hard and super hard maps. The characteristics of relevant maps shown in Table
Table I: Maps used in our experiments.
Map Name bane_vs_bamd vs_bane
Number of Controlled Agents
Map Type
24
Hard
Symmetric
Heterogeneous
MMM2
Hard
Asymmetric
Heterogeneous
5m_vs_6m
5
Hard
Asymmetric
Homogeneous
38_vs_50 5z
3
Hard
Asymmetric
Homogeneous
2c.vs_6Azg
2
Hard
Asymmetric
Homogeneous
27m_w vs 30m
27
Super Hard
Asymmetric
Homogeneous
6h_vs 8z
6
Super Hard
Asymmetric
Homogeneous
corridor
6
Super Hard
Asymmetric
Homogeneous
3s52.vs_3s6z
8
Super Hard
Asymmetric
Heterogencous
A.2 GRF
In Google Research Football (GRF) tasks, agents need to cooperate according to the rules of football matches to win games or related training tasks. The reward settings of GRF can be divided into two types. One is that only goals can be rewarded (scoring), and the other is that when an agent reaches specific coordinate point (checkpoints), it will get certain reward. Compared with the types of reward, the setting of the first type is more sparse. In the official multi-agent example, agents are allowed to observe all information on the field, which contrary our research hypothesis. We provide an observation space setting for agents in the Dec-POMDP version. The observation space of agents can dynamically change with their motion vectors, as shown in Figure 7 Considering the size of the field in the environment, we set the visual distance of the agents to 0.84, and the forward visual angle to 200". The agents can observe the position and direction of the ball, and within the range of viewing them, we also allow the agents to observe the position of their teammates and the position of their opponents.
In our experiment, we control all the players on the left side and set their movement state to be lazy (when not touching the ball, there no built-in Al intervention), and the built-in AI in the system completely controls all the players on the right side.
A.3 LBF
The LBF a multi-agent collection task built on a grid world where each agent and object as- signed level. The criteria for successful collection by agents is the sum of the levels of the related agent needs to be equal to or greater than the level of the item. The agent will receive a reward at the relative item level if the item is successfully collected. LBF provides more sparse reward environment and collaboration constraints than SMAC, which puts forward higher requirements for the effectiveness of multi-agent algorithms.
12

---

Under review as a conference paper at ICLR 2023
Direction of motion vector
Visual range
Visual range
100'
Agent' location
Figure 7: The Observation Space Design of Agents in GRF
The difficulty of the LBF can be adjusted by the number of agents, the size of the grid world, the type of items, and the hard collaboration constraints. In this paper, we select 15x 15-4p-5f supplement to the ablation experiment to verify whether the different components of our method are still valid a more challenging collaborative setting. The 15x 15-4p-5f scene indicates that agents in 15 15 grid world are involved in collecting 5 items.
Figure 8: 15x 15x15-4--5f in LBE.
B CASE STUDY IN LBF
As discussed in Appendix A.3 the constraints from level and the sparse reward environment pose a greater challenge for agents to establish complementary and reliable collaborations. order to verify the effectiveness of our proposed dynamic parameter sharing mechanism for the formation of complementary relationships between agents, we verify the performance of different parameter shar- ing mechanisms in 15x 15-46 1f- including single-layer dynamic parameter sharing (ours with only individual level), bi-layer dynamic parameter sharing (ours). full static parameter sharing (classic QMIX) and no parameter sharing (QMIX with non-sharing) all.
13

---

Under review as a conference paper at ICLR 2023
Teat minemmeee
0.8
Ganiontying stwe) OMIX (ne alaee)
0.4
0.2
4.0
8.0
12.0
16.0
20.0
Train steps (M)
Figure 9: Comparison of different parameter sharing mechanisms in 15x 15x15-4p-5f.
As shown in Figure 9 our method still yields the best return, demonstrating that our approach promotes more collaboration among agents. Although single-layer dynamic parameter sharing can promote agent collaboration, it can not always form complementary collaborative relationships. The result again confirmss that forming complementary collaborations among agents requires a dynamic balance of personality and commonality, which is hidden in dynamic parameter sharing at both the individual and team levels.
C ABLATION STUDY IN GRF
As shown in Figure we additionally verified the performance the single-layer dynamic param- eter sharing mechanism (only individual level) in two GRF scenarios.
acadamy. va 1 with koaper
academy pass and shoot with koeper
Test retum mean
00006000
3.0
2.0
1.0
-1.0
0.0
4.0
5.0
9.0
10.0
Train steps (M)
Train stepss (M)
academyy with Keeper
acadenty wotooothhhheeeeeeeeeeeeeeeeeeeee weeper
shoot
Tea sodee reward
1.2
0.6
0.6
.0.3
-0.3
10.0
Train stops (M)
Train Stops (M)
Ours
Cura (only Ind)
QMIX(ful)
QQMIX)o)
Figure 10: Performance comparison with baselines on Google Research Football.
Unlike the experimental results of SMAC and LBF, the algorithm of single-layer dynamic parame- ter sharing is better than the QMIX algorithm of full static parameter sharing and even can achieve the result of bi-level parameter sharing academy 3 us with keeper. The experimental phe- nomena is because, in the GRF environment, agents have more extensive range of actions, which makes them have a unique observation space than the other two environments, which proves that the full parameter is more effective because agents have similar observation spaces Chrissianos al. 2021).
We need point out that the single-layer dynamic parameter sharing mechanism in the two scenar- ios is not as stable as our bi-level dynamic parameter sharing mechanism. This is because, on the one hand, agents unstable due to changes in shared partners; on the other hand, agents that do not share parameters cannot establish complementary cooperation relationships, which again proves that approach is correct in considering both individual and team levels.
14

---

Under review as a conference paper at ICLR 2023
D THE EFFECT OF THE NUMBER OF VIRTUAL ROLES
During the experiment, we found that setting different numbers of virtual roles would affect the per- formance of the entire algorithm. As shown in Figure I1 show that in the SMAC environment, we only consider the impact of different numbers of virtual roles on the experimental results.
5m VS. 6m
3o V5 5z
MMM2
100
130
100
k=
Tust (%)
Test Win rale (%)
80
40
40
20
20
20
0
1.5
2.0
Train steps (M)
Train stegs (M)
Train steps (M)
comndor
355z 3s02
Sh.
100
90
Teal win all (%)
80
wit rata
Test rate
80
60
40
20
0.00.51.01.8
4...
0.00.51.01.52.02.5553
Train steps (M)
Train scaps (M)
Train staps
Figure 11: Compare the effects of different numbers of virtual roles (only individual level).
As we analyzed in the text, setting too many virtual roles will not improve the performance of the algorithm.
E FULL EXPERIMENTAL RESULTS IN SMAC
comddo
Gh.ve
355z. Jefz
100
100
100
Test onn rate (%6
80
80
20
2.0
3.0
0.0
3.0
4,0
1.0
2.0
5.0
Train steps (M)
Train steps (M)
Train steps (M)
542g
Test rite (%)
100
100
80
80
20
0
0
2.0
Train steps (M)
Train stops (M)
Train steps (14)
5m, va, 6m
bone, vo, bone
So. va. Sut
100
100
Test wn cale
80
40
20
0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Train
Ours
CDS
EDI
QMIXquul
RODE
Figure 12: Performance comparison with baselines on SMAC.
15

---

Under review as a conference paper at ICLR 2023
F HYPERPARAMETERS SETTING
In our method, we design a role selector and a group selector to guide agents in selecting appropriate partners for dynamic parameter sharing the individual and team levels. The core of the role selector is a gated recurrent unit variational autoencoders (GRU-VAE). Besides the GRU unit, the encoder and decoder parts that makeup VAE are composed of simple linear layers and activation functions. The design of the group selector can be divided into graph attention network with a 128- dimensional output state and other corresponding dimensions of the network layers linear layer and GRU unit with 64-diimnsional hidden state). To expand, we use the softmax function to get the roles of agents at the individual level through the bottleneck of the GRU-VAE. For the grouping of agents at the team level, considering that our grouping task is an extra-designed reinforcement learning task, we use the same E - greedy method as agents' selection of actions.
We need to state that, in addition to the newly designed components of our method, we maintain as much agreement as possible with the baseline method QMIX on the structure, parameters, and optimization methods that make up the rest of our method, to fully demonstrate in ablation experiments that the real utility of the different components that make up our method comes entirely from our design.
G EXPERIMENTAL DETAILS
For our method and baseline algorithms used, we tested 2MM and 5M steps on all hard maps and all super hard maps of SMAC using five random seeds, respectively. Due to the use of the ε - greedy method, we & to be linearly annealed from 1.0 to 0.05 in 70K time steps in all hard maps and three super hard maps and kept constant during the rest of the training period. We set the annealing time for the remaining super hard map 6h_vs_8, 8z to 500K.
In GRF, we respectively tested 10M and 8M steps on academy 1.w...3......tthheeeeeeeee and academy pass and shoot with keeper using three random seeds. We set ε be linearly an- nealed from 1.0 to 0.05 in 100K time steps in both scenarios. For part of the observable space we designed, contains the location information of the agent itself, the position, direction and attribu- tion information of the football, as well as the player information and opponent information within the visual range.
To test our method components' effectiveness in the 15x 15-4p-5f scenario of the LBF environment, we set the experimental test step as 20M and the annealing time of as 200K. The setting of e remains the same in our method and bascline algorithms.
For the section on the virtual roles and groups setup in our method, order to ensure that the collaborative relationship between agents is sufficiently complementary, we set the number of agents n, the number of virtual roles k, and the number of groups m satisfying the relationship n 2 k 2 2. Of course, we obtained plots of the number of virtual roles and the win rate in our ablation experiments, the number of virtual roles could not be increased blindly. Blindly increasing the number of virtual roles not only brings vast amount of parameters, but also makes complementary collaborations between agents impossible. In the SMAC and the LBF, the number of virtual roles we set shown in Table 2 In response agents' virtual roles and groups update periods, we have Tv = e-Ty. To guarantee that the virtual roles and groups are updated normally in each episode, we require that the update periods of the virtual roles and the groups satisfies To, Tv len (episodde) We set update periods for virtual roles and groups based on the average length of a episode. In general, we set the update periods of 5, 7, 8, 10 or 14 with e = 1 or
Our experiments are completed at Ubuntu 18.04 with GPU of NVIDIA GTX 3090, CPU of Intel 19-12900k and memory size 128G. In addition, the SMAC version is 10, the experimental version of 15x 15-4p-5f in LBF is v2 and the version of the GRF is 2.10. We will open source my algorithm and experimental environments code on time.
16

---

Under review as a conference paper at ICLR 2023
Table Number virtual roles and groups in different maps.
Environment Name
Map Name
Number of Roles
Number of Groups
bane vs bane
MMM2
5
5m_vs_6m
3s_vs
SMAC
3c_vs_64zg
2
27m_vs_30m
6h 8z corridor
3
3557_V83ss6z
3
LBF
x 13-00--
1.vs_1_with keeper
GRF
pass shooottt with keeper
2
17


Reviews:
[{"review_id": "tkdFsR-y0P", "reviewer": "Reviewer_UyCu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Bi-level Dynamic Parameter Sharing mechanism (BDPS) in MARL. The core idea is assigning different agents to different roles based on the long-term cumulative advantages and grouping multiple roles to a set of teams via the Graph Attention Network. And the authors verify the proposed method in some typical MARL benchmarks.", "summary_of_the_review": "This paper study an important and an interesting problem in MARL. But the quality of the paper should be significantly improved. So I recommend a reject based on the current version of the paper.", "strength_and_weaknesses": "**Weaknesses**\n  * The motivation is not clear enough. From the paper, I do not see why using the advantage function to represent the roles of agents is better and why incorporating a two-level selective parameter sharing is important.\n  * The writing of the paper could be significantly improved. Lots of sentences are too long and many sentences are unclear, e.g., \"we add additional implicit information $h_U^t$ from the individual role level to give full play to the positive impact of the individual identities on the team results.\"\n  * Two important single-layer selective parameter sharing baselines ([1] and [2]) are mentioned in the introduction but are missing in the experiments.\n  * The experimental evaluations in SMAC is not convincing. Recently, [3] and [4] have verified that the optimized QMIX (completely sharing the parameters among all agents) could achieve 100% win rates on all Easy, Hard and Super Hard scenarios of SMAC. Therefore, SMAC may not be a good testbed to validate the benefits of the dynamic parameter sharing mechanism. The authors could verify the algorithm in more complex environments.\n  * minor:\n    * Typos:\n       * \"And according to the role selector and the U group selector, choose the most suitable role and group for the agents V.\"\n       * In equation (6), the local Q-function improperly takes the joint action **a** as input.\n\n**Reference**\n  * [1] Scaling multi-agent reinforcement learning with selective parameter sharing.\n  * [2] A cooperative multi-agent reinforcement learning algorithm based on dynamic self-selection parameters sharing.\n  * [3] Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning\n  * [4] API: Boosting Multi-Agent Reinforcement Learning via Agent-Permutation-Invariant Networks", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nb19Yn8EMR", "reviewer": "Reviewer_diDZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method for dynamic parameter sharing over individual and group levels.\nResults show an improvement over standard baselines on two challenging benchmarks (Google Research Football and StarCraft).", "summary_of_the_review": "While the results are reasonably good, I came away from the paper unsure why most of the specific design decisions were made, and the experiments did not provide a significantly granular ablation study to justify those decisions. The paper could improve by providing both more intuitive explanations for the decisions as well as improved ablations.", "strength_and_weaknesses": "# Strengths\n* The method addresses a challenging and important problem.\n* The results seem to indicate that it is effective in improving upon standard baselines (e.g. QMIX).\n\n# Weaknesses\n* It's unclear what the VAE over \"advantages\" is representing, especially since the advantages are computed with agent utility values which are **not** Q-values in the traditional sense since they do not respect the Bellman equation. These agent-wise utilities are fed into a mixing network to produce a $Q_\\text{tot}$ which is a true Q-value.\n* The choice to select the argmax of the VAE latent as the role (Eqn 3) seems unmotivated.\n* If I understand correctly, the group selector requires information from all agents (the set of all virtual roles). As such, the method does not fall within the decentralized execution paradigm as suggested by section 2.2, and this should be more explicitly stated.\n* The group selector appears to be a hierarchical controller which operates over a dilated time scale, but in order to learn it we would need to have a separate mixing network for the group-selection actions. This is vaguely alluded to in the text, but is not shown in Figure 1. Furthermore, the Q-functions in Eqn 8 would have to be defined over the domain of group-selection actions, $\\bf v$, but they are not which is especially confusing since the equation is maximizing over those.\n* The results for RODE are not consistent with the literature. I haven't checked the others carefully, but those stood out since it thoroughly outperforms QMIX in the original paper.\n* The ablations aren't explained clearly. More granular ablations are required to justify the design decisions (e.g. using attention, VAE, etc.)\n\n# Questions\n* What is meant by \"advantage information can better represent the current state of agents than the observation information\"? If anything these filter out information about the agents' observations to only that which is relevant to the global expected returns.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eG5rwnpDiJ", "reviewer": "Reviewer_CKA2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a bi-level dynamic parameter sharing mechanism (BDPS) to achieve better coordination among agents in cooperative multi-agent tasks. The key idea is rather than blindly sharing parameters among all agents, they share parameters among agents based on their roles and groups.  At the agent level, they determine the roles of different agents based on the advantage sequence information and agents with the same role share parameters. At the team level, they determine the groups of different roles using a graph attention network and agents within the same group share parameters. ", "summary_of_the_review": "I struggled with understanding the main method proposed in this paper. I found many descriptions about the proposed method confusing and hard to understand. Just to list a few examples here:\n- \"we give the time relationship for agents to update the groups and roles: T_V = e · T_U.\" Why the update periods of the group and individual roles follow this relationship? How do you decide the value of T_V  and T_U?\n- \"because the advantage information can better represent the current state of agents than the observation information.\" The advantage function basically describes how much better or worse an action selection is compared to the agent’s performance under current policy. Why it can better represent the current state of agents than the observation information? What is the intuition behind this? \n- \"use the \\epsilon−greedy to select groups v_V for agents.\" It is not clear to me how do you really select groups for the agents (e.g., how to select groups when you are being *greedy*). Why is this a good way of selecting groups for agents (with probability \\epsilon, you are just randomly choosing groups for agents)?\n- \"For the group selector V at the team level, we used the QMIX (Rashid et al., 2018) and RODE (Wang et al., 2021b) methods because we introduced an additional deep reinforcement learning task to generate groups for agents.\" This is not a good explanation of the motivation for using QMIX and RODE. It is also not clear to me how RODE is used in the proposed method. \n- In Equation 8, how is $Q_{tot}^{V}$ generated? Through a mixing network? I cannot find it in the overall BDPS architecture (Figure 1). There is the global action value function $Q_{tot}$ in Equation 9, does this mean there should also be a $Q_{tot}^{U}$, which is never mentioned (I believe)?\n\nIn addition, I'm not fully convinced that the current way of combining agents of different roles into groups in BDPS is beneficial. Simply using $\\epsilon$−greedy strategy to select groups for the agents seems random and naive to me. I do not see how randomly assigning groups sometimes can help boost the performance. Also, the result of the single-layer dynamic parameter sharing mechanism (only individual level) in one GRF scenarios shows that its final performance is very similar to that of BDPS. How does the single-layer dynamic parameter sharing mechanism perform in all SMAC maps tested?", "strength_and_weaknesses": "- Strengths:\n\t- The proposed method seems to achieve some good experimental results in different cooperative tasks.\n- Weaknesses:\n\t- The proposed method is not very well motivated and the key insights/intuitions behind it are not very clear.\n\t- Many details of the proposed method are not clear to me, making it hard for me to judge the technical contributions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MrGqImk_B6", "reviewer": "Reviewer_n4Px", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on parameter-sharing issues in multi-agent reinforcement learning. The core idea of this paper is that previous methods of parameter sharing only consider the sharing between different agents, which is a single-level sharing. This work proposes a bi-level parameter-sharing mechanism that not only shares parameters among agents but also among teams. To achieve this, they defined roles based on the long-term cumulative advantages and share parameters of agents in the same team. Empirically, they evaluated the proposed method in SC2 and Google Footballs. And the results show some strengths over baselines. ", "summary_of_the_review": "The idea of bi-level parameter sharing is novel and interesting, but this paper can be further improved especially in terms of soundness, clarity and ablation studies. ", "strength_and_weaknesses": "**Strengths**\n\n1. The problem this paper considers is very important. Parameter sharing could largely improve the efficiency of policy learning in MARL. \n\n2. The literature review is sufficient. Previous works are discussed and compared empirically. \n3. The motivation for this work is easy to understand. Previous works only considered single-level parameter sharing, and this paper novelly considers bi-level sharing.\n\n\n**Weaknesses**\n\n1. The proposed method is not sound enough. (1) In Definition 1, it seems that a role can only belong to one team, but why is that? And what are the definitions of roles and teams? (2) Individual-level parameter sharing depends on advantages, which are calculated using $Q$ values. What if these $Q$ values are not accurate at the beginning of training? Will the false role selection further hamper policy learning? (3) Advantages are computed as $A=\\max{(Q)}-Q$, then agents with similar observations will always have the same role. What if two similar agents need to take different sub-tasks? \n2. The quality of empirical evaluations needs to be improved. (1) Figure 2 is not clear. Do not use dotted lines for curves with similar performance. (2) In Figure 3, the performances of CDS and RODE in some tasks are very different from their original paper. (3) The ablation studies are not enough. More results are expected. What is the functionality of individual-level/team-level parameter sharing? How do the roles and teams evolve in an episode or in the process of training?\n3. The results are not significant. This paper tested two tasks in Google Football, but the proposed method only outperforms others in one task. \n4. This paper does not provide any discussions of limitations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject