Role Diversity Matters: A Study of Cooperative Training Strategies for Multi-Agent RL
Authors: Siyi Hu
Chuanlong Xie
Xiaodan Liang
Xiaojun Chang
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-agent Reinforcement Learning

Abstract: Cooperative multi-agent reinforcement learning (MARL) is making rapid progress for solving tasks in a grid world and real-world scenarios, in which agents are given different attributes and goals. For example, in Starcraft II battle tasks, agents are initialized with the various move, defense, and attack abilities according to their unit types. Current researchers tend to treat different agents equally and expect them to form a joint policy automatically. However, ignoring the differences between agents in these scenarios may bring policy degradation. Accordingly, in this study, we quantify the agent's difference and study the relationship between the agent's role and the model performance via {\bf Role Diversity}, a metric that can describe MARL tasks. We define role diversity from three perspectives: policy-based, trajectory-based, and contribution-based to fully describe the agents' differences. Through theoretical analysis, we find that the error bound in MARL can be decomposed into three parts that have a strong relation to the role diversity. The decomposed factors can significantly impact policy optimization on parameter sharing, communication mechanism, and credit assignment strategy. Role diversity can therefore serve as a flag for selecting a suitable training strategy and helping to avoid possible bottlenecks on current tasks. The main experimental platforms are based on {\bf Multiagent Particle Environment (MPE) }and {\bf The StarCraft Multi-Agent Challenge (SMAC)}, with extensions to ensure the requirement of this study are met. Our experimental results clearly show that role diversity can serve as a robust description for the characteristics of a multi-agent cooperation task and help explain the question of why the performance of different MARL training strategies is unstable according to this description. In addition, role diversity can help to find a better training strategy and increase performance in cooperative MARL.

Under review as a conference paper at ICLR 2022
ROLE DIVERSITY MATTERS: A STUDY OF COOPERA- TIVE TRAINING STRATEGIES FOR MULTI-AGENT RL
Anonymous authors Paper under double-blind review
ABSTRACT
Cooperative renifiegeettrnnforremmentt learning (MARL) is making rapid progress for solving tasks in grid world and real-world scenarios, in which agents are given different attributes and goals. For example, in Starcraft II battle tasks, agents are initialized with the various move, defense, and attack abilities according to their unit types. Current researchers tend to treat different agents equally and ex- pect them to form a joint policy automatically. However, ignoring the differences between agents in these scenarios may bring policy degradation. Accordingly, in this study, we quantify the agent's 's difference and study the relationship between the agent's role and the model performance via Role Diversity, metric that describe MARL tasks. We define role diversity from three perspectives: policy- based, trajectory-based, and contribution-base to fully describe the agents' dif- ferences. Through theoretical analysis, we find that the eπror bound in MARL can be decomposed into three parts that have strong relation the role diversity. The decomposed factors can significantly impact policy optimization on param- eter sharing, communication mechanism, and credit assignment strategy. Role diversity can therefore serve flag for selecting suitable training strategy and helping to avoid possible bottlenecks on current tasks. The main experimen- tal platforms are based on Multiagent Particle Environment (MPE) and The StarCraft Multi-Agent Challenge (SMAC), with extensions to ensure the re- quirement of this study met. Our experimental results clearly show that role diversity can serve as robust description for the characteristics of a multi-agent cooperation task and help explain the question of why the performance of differ- ent MARL training strategies is unstable according to this description. In addition, role diversity can help to find better training strategy and increase performance in cooperative MARL.
reinforcement
INTRODUCTION
Recently, multi-agent reinforcement learning (MARL) has captured people's attention due to its impressive achievements in the field of super human-level intelligence in video games [3, 6, 52, 57], card games [7, 25, 44, 59], and real-works applications [62-64]. These achievements have benefited substantially from the success single-agent reinforcement learning (RL) [14, 15. 30, 42, 43] and rapid progress of MARL from both the competitive and the cooperative side.
On the competitive side, existing works focus on game theory among adversary agents with guaran- teed policy convergence via theoretical analysis [5, 17, 27, 29,56], 56]. Whereas the achievements on the cooperative MARL are more based on empirical results in cooperative multi-agent system (MAS) [4, 21, 28, 39, 46, 51, 61]. One key problem of cooperative MARL is that whether one algorithm better than another depends on the MARL tasks as showed in Fig. 1a. Current researches focus on developing algorithms on the tasks they good at but lack the study of why the performance declines on other tasks [16, 35, 55, 55, 58]. Even adopting the state-of thhe--art algorithms does not guarantee strong performance [13, 35, 48, 55, 58]. This may due to the varying characteristic (e.g agent's 's attributes and goals) of MARL tasks and scenarios, one single algorithm is not able to cover them all, which means we have to change the training strategy according to the current scenario.
From this perspective, we need to find metric to describe different MARL tasks and use this description to help determine the best strategy combination showed in Fig. 1b. Considering that

---

Under review as a conference paper at ICLR 2022
Task 1
Tasic 1
Bele Divarrity
Algoriites Task
Peftemanee
MARL Algecitan
Algurittee
Tasis
Communicttoon
Pedtermacee
Communication
MARL Algocther:
Cnellt Anigrmer
Task
Tlask n
Iille Divardity
aiqgereet
Alpuriam
Task
Perfonaace
(b)
Figure 1: (a) Performance of MARL algorithm 1 & 2 combined with different parameter sharing, communication, and credit assignment strategies varies from candidates to n. (b) Using Role Diversity to describe each task, we can ensure the best combination of different strategies.
the main component of MAS is the agents, we propose new definition Role in MAS to quantify the agents' difference and use Role Diversity describe MAS. We then analyze how the role diversity impacts the MARL both theoretical and experimental. For theoretical analysis, we use the decomposed estimation error of the joint action-value function Qtot in MARL to discuss how role diversity impacts the policy optimization process. The experiment further verifies the theoretical analysis that the role diversity is strongly related to the model performance and can serve as good description of a MAS. As shown in Fig. I, with the role diversity description of each task, we can now avoid possible bottleneck of a MARL algorithm with the combination of different parameter sharing, communication, and credit assignment strategies. With the definition of role diversity and the analysis its impact on MARL, we can also explain the question of why the model performance varies across different tasks.
Role diversity are defined from three aspects: policy-based unjectory-baseee and contribution-bart in Sec. which are measured by action distribution, observation overlap, and Q/state value diversity. Through theoretical analysis, we find each type of role diversity has different impact to different terms of decomposed estimation error: algorithmic error, approximation error, and statistical error (Sec. 4). We conduct comprehensive experiments covering three main topics in MARL: parameter sharing communication, and credit assignment in Sec. 5 and provide a of guideline on choosing MARI. training strategy in Sec. 6. The main experiments are conducted on MPE [28] and SMAC [39] benchmarks, covering a variety of scenarios with different role diversity. The impact of role diversity is evaluated representative MARL algorithms including IQL [49]. IA2C [31], VDN [48] QMIX [35], MADDPG [28], and MAPPO [58], covering independent learning methods, centralised policy gradient methods, and value decomposition methods. The experiment results prove that the model performance of different algorithms and training strategies is largely dependent on the role diversity. Scenarios with large policy-based role diversity prefer parameter sharing strategy. Communication is not needed in scenarios with large trajectory-based role diversity. Learnable credit assignment modules should be avoided when training on scenarios with large contribution- based role diversity.
The key contributions of this study are as follows: First, the concepts of the role and role diversity are defined to describe MARL tasks. Second, a theoretical analysis of how role diversity impacts MARL policy optimization with estimation error decomposition built. Third, role diversity is proven to be strongly related to performance variance when choosing different training strategies, including parameter sharing, communication, and credit assignment on the MARL benchmarks. Finally, set of guidelines for selecting training strategy based on role diversity is provided.
2 RELATED WORK
Researches the development of cooperative MARL algorithms are mainly in three aspects: pa- rameter sharing, communication and credit assignment.
For parameter sharing, the common approach in cooperative MARL is to fully share the model pa- rameters among the agents [16, 35, 48, 55]. In this way, the policy optimization can benefit from the shared experience buffer with samples from all different agents, providing higher data efficiency. However, has also been noted recent works that parameter sharing is not always a good choice [8, 34, 50]. In some scenarios, selective parameter sharing strategy, no parameter sharing.
2

---

Under review as a conference paper at ICLR 2022
significantly benefit agent performance and surpass the full parameter sharing. However, the question of why different parameter sharing strategies have different impacts on different scenarios remains open. In this study, we find that the role diversity can serve strong signal for selecting the parameter sharing strategy.
The communication mechanism is intrinsic part of the multi-agent system (MAS) framework [18, 19, 23, 28, 47]. It provides the current agent with essential information of other agents to form a better joint policy, which substantially impacts the final performance. In some cases, communication restrictions exist, which hinder us from freely choosing communication methods [28, 39]; in most cases, however, the communication is available and is optional on when to communicate and how to ingest the shared information [19, 45J. We present comprehensive study on the relationship between role diversity and information sharing via communication mechanisms and demonstrate that role diversity determines the necessity of communication.
For the credit assignment, most cooperative MARL algorithms adopt Q-learning or policy gradient as the basic policy optimization method, which is combined with an extra value decomposition module [16, 35, 48, 55] or shared critie function [13, 28, 58] to optimize the individual policy. Some other works find that leveraging the reward signal is unnecessary; however, optimizing the individual policy independently (independent learning. IL) can still get a strong joint policy [34, 49]. It then becomes slightly difficult to decide which credit assignment method (including IL) is better as there is no single method in cooperative MARL that is robust and always outperforms others (compared to PPO [43] or SAC [14] in single-agent RI. different tasks. In this study, we contend that role diversity is the key factor that impacts the performance of different credit assignment strategies.
In the next section, we present the role definition from three aspects including policy-basede trajectory-based, and contribution-based, and propose the measurement of different role diversity types to describe a MARL task.
ROLE DIVERSITY
Using role to describe the characteristic of the agents in the MARL context has been proven to be effective in many recent works [8. 24, 54, 55]. However, the definition of the role concept remains largely unclear. In work [55]. the role is defined the higher-level option in the hierarchical RL framework [20]. In work [8], the role is defined as the environmental impact similarity of a random policy. These definitions
intuitive and cannot accurately describe the role difference. In this study, we attempt to define the role in a more comprehensively way from three different aspects: policy-based, trajectory-based and contribution-based More specific scenario- based illustration can be found in Fig. 7 and Fig. 8. With our refined role, a strong relationship between role diversity and the MARL optimization process can be built and the performance variance can be further explained.
3.1 POLICY-BASED ROLE
In MARL, different agents output different actions based on its current status. As common sense would indicate, actions taken at the same timestep can indicate different roles [55]. However, there many exceptions. For instance, if we have two soccer players passing ball to each other repeatedly [21], although the action is different at each time step, the roles of these two soccer players can be very similar from the perspective of the whole soccer game. Therefore, it is not sufficient to distinguish the role difference based on a single timestep. Instead, we contend that this difference should be defined based on a period. As this role purely based on policy distribution n we refer to it as a policy-based based role.
Specifically, we define the policy- role as the statistics of the actions' frequency over a period, which is n steps backward and forward from the current timestep. Here, is the time interval that half the length of the total time. More details can be found in Fig. 8a, where we provide a real scenario from SMAC. Policy-based based role difference can be represented as follows:
T4n Σ
1
2n
t=T-n
where T represents the current timestep, n is the time interval, a is the agent index, π is the policy distribution. We adopt symmetrical KI. divergence measure the distance of different policy-based
3

---

Under review as a conference paper at ICLR 2022
18
Focus on one
Find one's best strarogy
3
Find good posilion
30
20
30
52
(a) Policy-based role one episode 4m_vs_3z
23
pigees
33
48
30
2m, VS 32
1slmth1M vs. 5z.
(b) Observation overlap percentage curve
(c) Q value of different agents curve
Figure 2: (a) An illustration of how policy-based role difference varies in one game(4m 3z). A detailed explanation of how varies can found Sec. 3.1. (b) Instance curve of the trajectory- based role difference in different battle scenarios according to observation overlap. Trajectory- role difference is larger in 3s_vs_5z but smaller in 4m_vs_5m. (c) Contribution-bass role diversity in different battle scenarios represented by Q value. The contribution-basee role diversity is larger in Is lelmlhlMMwi Mv but smaller in 4m-vs_3z.
roles. The total role distance A agents can be computed as follows:
dy
Σ
[KL
(L(r
Σd 00=00 a1=
AA-1
where de represents the policy-based role distance at timestep T, A represents the total agent num- ber, and KL represents
-Libllr divergence.
We also provide a case study of how policy-based role diversity varies in Fig. 2a. From a real battle scenario (4m 3z) taken from SMAC, find three stages including Find good position, Focus on enemy and Find each one's 's best strategy. In stage 1, agents try to find their own best location; the role diversity large. stage agents focus on the same enemy target; the policies become similar and the role diversity is decreased. In stage 3, the formation of the agents is broken up by the enemies. Policy-based role diversity again increases as each agent required find its best strategy to deal with its current situation.
3.2 TRAJECTORY- BASED ROLE
A policy-based role only considers the role diversity from the action distribution perspective. A slight action difference may not diversify policy-based role; however, this difference can be enlarged by time, which eventually results in de facto behavior difference. The most significant phenomenon caused by this is the agent's 'ss trajectory. Again, consider a soccer game where players repeatedly pass a ball to each other; these players may have similar policy-based role, but their trajectories are different. This difference is enhanced by the partial observation setting that exists in many popular cooperative MARL environments [39, 46, 61] as the partial observed input from different agents shares less common pattern when the vision scope is smaller. Therefore, trajectory- -based role diversity an important supplement to policy-base role diversity.
define the trajectory based role the record of the agent's movement. However, the determination of the extent to which two trajectories differ is not a straightforward matter. To measure this difference,
Generally speaking,
indirect metric called observation overlap percentage.
4

---

Under review as a conference paper at ICLR 2022
Using observation overlap to measure the trajectory- role difference is: I. easy to compute, II. able to utilize a constant scale from to and III. strongly related to observation scope, which means that two trajectories can have varying role distances. For instance, in an A-agent multi-agent system, the trajectory difference can be computed as follows:
fraj dT
Σ
ddo.a:
(3)
A Σ
Og =0
Here, d:a-as is the observation overlap percentage of the total area between agent dg and agent di at timestep T. An example of calculating the observation overlap percentage in SMAC [39] can be found in Appx. C. We provide the observation overlap percentage curves in Fig. 2b to demonstrate how trajectory-based role diversity varies in different scenarios.
3.3 CONTRIBUTION BASED ROLE
To help ensure the authenticity of the modern MAS, agents are initialized with different attributes in recently proposed MARL environments [21, 28, 39, 46]. For instance, in [21], the roles of forward and goalkeeper are quite different and characterized by different observation scopes, action spaces, and reward functions. The type differences are easy to notice but hard to define, as is the role distance between them.
Here, we use an indirect variable called contribution to measuring the type difference. In cooperative MARI, the final target obtain an optimal joint policy consisting of A individual policies. To do this, the credit assignment strategy is proposed in [13, 35, 48] to leverage the reward signals to each agent achieve individual policy optimization. A good credit assignment strategy should be able to leverage the reward signal in a manner equal to each agent's contribution to the global reward. In this way, the Q value (Q function in off- off-policy RL) or state value (critic function in on-policy RL) of each agent can be estimated based on the leveraged reward signal. From this perspective, the Q value or the state value can regarded as the agent's contribution to the team. Generally speaking, we use the Q value or state value to measure the contribution of single agent. Contribution--ased role diversity can be computed as follows:
Eap May
gront
Σ
Σ
Σo
max (Vaar(Vos: Vay)
9p=0 da
Here, v is the Q value or state value of the policy output and de do is the absolute value difference between the agents' output. In addition, role diversity from to 1. We provide the Q value (mean & max) curves in Fig. 2c to demonstrate that the contribution-bass role diversity can vary a lot in diffferent scenarios.
max function to keep the range of contribution-basee
More detailed role definition, connection, and application discussion can be found in Appx. C, D and E. In this paper, all the role diversity values and curves including dG de and doont come from VDN[28] and separated training with no communication for robust performance and training- efficient.
4 THEORETICAL ANALYSIS
In this section, we use a simple scenario as an example to illustrate the role diversity. Suppose each agent makes individual observations and the learning procedure of all agents is independent. We provide finite-sample analysis for the estimation error of the joint action-value function and identify the terms corresponding to the role diversity. We denote Qi and Q as the optimal joint and individual Q-function respectively and write || - as the Lp norm with respect to probability measure μ. Motivated by [48] and [35], we consider a simple case:
Qiot R F(Q"== TQ with E An, Q = =Q' 27),
where n is the number of agents and A, is the (n - 1)-dimensional probability simplex. Here, the credit assignment function F is weighted sum of Q i [n] with non-negative weights. We then study the excess risk that the error gap between the estimated and optimal solution:
Err
11Qror w Tu
11Qror (w*)² "
5

---

Under review as a conference paper at ICLR 2022
where w and w are the optimal and estimated weights and Q is the output of FQI algorithm at the iteration t. We further denote Q as the space of individual Q-functions and write w (Q) = su inforee IIQ' TQll2 In Sec. B.3, we prove that
Err
vn ||
w|ll
Varn (Qr)((Q))
11
40
No
47
( - O(V + (1-
+ (1 71- M,
(5)
N
where N is the sample size and T is the number of iterations. In addition, Qy.v is the concentration coefficient and No represents the 1/N-covering number of Q. Please refer to Sec. B.1 for the detailed definitions.
The first term on the RHS of (5) reflects the benefit of credit assignment that is strong related to the Contribution-Based Role (Sec. 3.3). When Var, is non-negligible, minimizing l|w* w|l can significantly decrease the excess risk. The second term that involves w( ()) stands for the approximation error caused by functional approximation in Q. It depends on the concentration of the sample and the scale of the hypothetical space. The remaining two tems are statistical error and algorithmic error. If the sample size is sufficiently large and the learning time is long enough, they can be arbitrarily small. In Sec. B.5, we assume Q is a sparse ReLU network and TQ is a composition of Hölder smooth functions, and analyze the convergence rate as N,T 00.
Next, we demonstrate that the variance term Var, (Q") is related to both the Policy-Basee Role (Sec. 3.1) and the Trajectory-Base -Based Role (Sec. 3.2). We consider the case in which all agents share one estimated Q-function and denote the optimal share Q-function as Q". Sec. B.4 proves that
| Var (Q-(z, u)
Err share
( - Q²))
+ Vn ||
w|
1,44
1,a
4 p
In Ny
47
(1 7)2 V (Q) + (V
+
(1 - 7) M,
(6)
n.
where Ng is the 1/ number of Q and
Q' u) = (Q) uz), Q" (zz, U2)..... Q" un)).
The first term on the RHS of (6) stands for the bias caused by parameter sharing. If all Q are the same, the bias will disappear. Therefore, the Policy-Based Role related this bias. Second, the variance Var,, (Q") here is caused by the trajectory diversity. To reduce this term, we should should ensure that all agents have similar observations. In addition, the Trajectory- the concentration of all agents' support set. It is therefore natural to group the highly overlapped agents into one sub-joint agent via communication mechanism. This can be compared to the separate case in (5), where approximation eиror and the learning error are the same. In Sec. B.5, we show that the parameter sharing improves the convergence rate of the statistical error via sample pooling, while the communication decreases the convergence rate by activating more input variables.
Role measures
5 EXPERIMENT
In this section, we mainly demonstrate how model performance varies with role diversity and how to adjust the training strategy in the context of cooperative MARL. The experimental results show the following: 1. that the performance of different parameter sharing strategies is strongly related to the Policy-Based Role (Sec. 5.1). 2. that the benefit brought by differrntt communication mech- anisms can be easily affected by the Trajectory-Based Role (Sec. 5.2). 3. that the performance of the credit assignment method, or the centralized training strategy, is largely dependent on the Contribution Based Role (Sec. 5.3). that the choosing of training strategies should be deter- mined by the scale of role-diversity for different scenarios. The main experimental platforms are MPE [28] and SMAC [39]. Extensions are made to fulfill the requirements of parameter sharing and the communication mechanism, these include separated training of policy in Sec. 5.1 and informa- tion exchange among agents in Sec. 5.2. All results come from eight random seeds. More details regarding the experimental settings can be found in the appendix.

---

Under review as a conference paper at ICLR 2022
Benchmark
Scenario
Role Diversity
Warm-up
No shared
Pardy shared
Shared
SimpleSpread
14.1
-598.3
4137,07-1429 +43.4/+57.3
4179077176.
+15.17+19800
Tag
17.8
3.8
+47.0/660.9
+48.8/459.2
MPE
Adveessry
10.7
+5.27
+6.27
Double Spreadd
17.6
7.3
+47.877 +532
+28.67 +34.6
+3.67 +15.9
Double Spread-
19.5
220
+29,57 +1224
+12.07991.3
+11.74555 45.3
Aov.vs. Sm 35 Vs Sz
1577.1
6.5
+3.6/
+4.77450
+547 +6.1
18.
5.4
+7.37+1L0
+6.67
+827 +118
3.17122
6.0
11.27v1111
115.571556
+18.
4m.
3.3719. 19.3
4.4
+3.87+127
+163/+14.7
+5.47 +8.4
SMAC
4m 37
3.8/12.1
1.2
+1244712.1
+12.7+122
+12.
3.2732.5
4.
+2.27+455
lelstr_ys,
8.7/22.0
11.8
64.1/=6.1
43.7/45.9
42.7745.4
24773322
162
43.474 3.4
43.6743.6
+3.7733.6
Is1mihIM
277/188
8.2
+7.8/+11.6
46.541009
45.3/+10.0
1s1mib1M.vs.55
6.2/22.5
6.2
+6,4/+9.1
+1.2/+8.5
+3.7/46.11
Table 1: Performance of three parameter sharing strategies on different scenarios. Warm-up up refers to the reward value point where the strategies start to differentiate. + represents the additional reward gained based on warm-up performance. The left side and right side of the represent the reward gained at the half training steps and the full training steps respectively. The best performance in each scenario is marked in bold red. More detailed analysis can be found in Sec. 5.1.
5.1 PARAMETER SHARING
Policy-based role influences the convergence speed and final performance of different parameter sharing strategies in cooperative MARL. The scenarios we choose from the MPE and SMAC bench- marks are simple but diverse, covering policy-based role diversity that ranges from small to large. In table. 1 and Fig. 3, we provide policy-based role diversity and the model performance curve on these chosen scenarios. For the SMAC benchmark, adopt two metrics count rz in Eq. 1. The first metric is real policy diversity, which treats each action as an independent. The second way is se- mantic policy diversity, which distributes the actions to different groups according their semantic type (e.g. move & attack). There is no semantic policy-based based role in scenarios chosen from the MPE benchmark as all the actions are of the same semantic type (move). The policy- -based role diversity is then calculated according to Eq. 2. The base MARL credit assignment strategy we choose in table. 1 is VDN[48], combined with a fully/partly/no parameter sharing strategy. For details of how the partial parameter sharing strategy works, please refer to the Appx. F. We also evaluate other pop- ular credit assignment strategies including IQL[49), IA2C, MADDPG[28] MAPPO[58], MAA2C and QMIX[35] combined with a fully/no parameter sharing strategy in Fig. 3. IA2C and MAA2C are the extension of A2C[31] on multi-agent scenarios.
Table. 1 outlines the performance of three parameter training strategies including No shared, Partly shared and Shared with the base credit assignment method VDN[48]. Detailed framework and settings can be found in Appx. F. As the policy-based role diversity increases, the performance of the Shared strategy is degraded in terms of both convergence speed (half training steps) and the final reward (full training steps). One interesting phenomenon also emerges: the same agent type (e.g. 3s_vs_5z, 4m_vs_42 does not always indicate small policy-based role diversity, and vice versa (e.g. IslmIhiM_vs. _v__3,,_ which means it is hard to define the role before identifying an adequate policy. Fig. 3 shows the model performances of two parameter sharing strategies including No shared and Shared with different credit assignment methods. For policy gradient-based methods, we extend the training steps from the standard 2M to 20M (10 times) as the convergence speed of policy gradient- based methods (e.g. MAPPO, MAA2C) slower than Q value-based methods. From Fig. 3, we find that different credit assignment methods have a slight impact on parameter sharing strategies but the trend in which no parameter sharing strategy achieves performance improvement continues to be present as the policy-based role diversity increases.
Here we conclude that, scenarios with large policy-based role diversity prefer no parameter sharing strategy, and vice versa. Suitable parameter sharing strategy helps obtain faster convergence speed and higher final performance.
5.2 COMMUNICATION
More information is better. This principle is common sense in many areas including computer vision or natural language processing. With bigger dataset and more detailed and accurate annotations, the model can be better optimized. However, in reinforcement learning, the data is sampled us- ing the current policy: in most cases, this policy learning starts with randomly initialized neural network. In this way, the provision of more information may introduce a burden for the policy op- timization and can moreover further degrade the sampled data quality, which gives rise to vicious
7

---

Under review as a conference paper at ICLR 2022
20.0
TextTre Tee.iee-eteoooo Se
13-0
50.0
THE
charn
cland
24
Sag
Skpp
Skop
(a) 38_v5_55
(b) 4m-_s
(c) Sellszzss_Icls3
(d) 4m vs_5m
TheT Tpee Tye:/e/bwort
1000
0.
15
20
0.0
05
3.0
15
20
Skip
Sapp
(c) 38_vs_5z
(1) 4m.
(g) IcIslz.vs.ell334
(h) 4m.vs vs.55mmmm
Figure 3: Performance curves include Q value- baste---asd(firstt row) and policy gradient- bsseent-bbaddsscoodd
row) credit assignment with Shared and No shared parameter sharing strategies.
50
15
35
tadk Siat
-10
Observahon Ovedap 8
Performance Improvement
quatter stogs
stags
Obencvation
Reward
Overlap
%
(a) Model performance with or without communi- cation
(b) Model performance with various vision scope
Figure 4: Histogram of (a) model performance when adopting communication mechanism compared baseline (w/o communication). (b) model performance with different vision scope (6-9-18), where scope 9 is the standard setting in SMAC. Grey dots represent the observation overlap. Larger the overlap, the smaller the trajectory-based role diversity. Detailed analysis can be found in Sec. 5.2
circle. Therefore, it is critical to determine when and how to accept the extra information provided via communication mechanism in the context of cooperative MARL. As discussed in Sec. 4, the pattern of different agents' support sets for policy optimization can determine whether or not the extra information is needed. Notably, the similarity of these support sets is largely dependent on the trajectories of different agents, corresponding to the trajectory-basse role diversity defined in Sec. 3.2. Small trajectory-basse means that forming a concentrated input preferred for policy optimization. Experiment results in table. 7 and Fig. 4 further prove that scenarios with larger observation overlap are more suitable for communication. Detailed setting of communication mechanism on SMAC can be found in Appx. G
role diversity corresponds to a similar support set pattern, which
To prove that small vrjecttoyybbaee role diversity prefers obtaining extra information via commu- nication and vice versa, we conduct extra experiments to study the relationship between the pattern of the input observation (support set) and the model performance by shrinking the vision scope (r in eq. 14). The results can be found table The results show that the model performance is strongly related to the vision scope and the trajectory-base -ased role diversity determines whether the small or role diversity prefers large vision scope, indicat- ing the similar pattern of support set is better. Large trajectory-based role diversity prefers small vision scope, which means enlarging the pattern difference benefits the policy optimization. This further prove that extra information provided by communication forms similar pattern of support set which preferred in scenarios with small trajectory-basse based role diversity.
large vision is preferred. Small trajectory-
5.3 CREDIT ASSIGNMENT
The performance of different credit assignment methods is strongly related to the contribution- based role. We mainly focus three representative value-based MARL algorithm: VDN[48], QMIX[35] and [QL[49]. and compare their performance on different scenarios that have different contribution-based role diversity measured by Q value according Eq. 4. The result can be found
8

---

Under review as a conference paper at ICLR 2022
obs everlip
perfermance
obs overlap
seppe
perlenmanee
15.87 19.5
6.77927
1647 1957197 19.7
461.ys,5m
0.47
637 1L77 TOU
GN7
8471537188
1757
1slmihiM.ss..
0.25
K 15.7 15.7
0.40
12.77 TEAI TTX
18
7.87117715
18
12471537777
6V
6.67 37.7
6/
6.07 1717
IsimihiM.ve
0.18
3s 35.v5.5
0.21
5A0 1297 1O
397897102
3.27907
Table 2: Different vision scopes (6-9-18) impact the model performance. The scope should be larger than 6, which is the attack scope for agents. rrepresents the scope with the best performance. Detailed analysis can be found in Sec. 5.2.
docniit
brakand
Jana GRIS
lelsie.vn-leisle wa.55
1237159717
185/125/122
112714/1122
987012/119
BOY AOTIEDEEE
Sa
0.105
1787
697124/ 1334
SE/KT/RS
4274575T
101719911 1867
KETTSTTH
(stmiii5 dz
84715H7 155
4S1STT61
T.7K57KS
6371281
33751765
Table 3: Policy-based role diversity influences the performance of different parameter sharing strate- gies on the MPE[28] and SMAC[39] benchmarks. The best performance in each scenario is marked V. Asterisks denote the algorithms that are not significantly different. Q diversity curves can be referred to Fig. 18.
in table. In small Q diversity (= 0.5) scenarios, QMIX significantly outperforms VDN with both shared and no shared strategies. With the increase of Q diversity, the performance of QMIX starts to degrade. In scenarios where agents have significantly different Q value distribution (Fig. 18 1slm
VDN significantly outperforms QMIX. As for IQL. the performance is not as good as VDN and QMIX in most scenarios. However, IQL is not sensitive to Q diversity and can perform well in easy scenarios like lslmlhIM 3z. Combined with theoretical analysis in Sec. 4, we can conclude that QMIX is not suitable for a large contribution-base, role diversity sce- nario because of the additional value decomposition module, which is a sum function in VDN and a learnable neural network in QMIX. The neural network fails to minimize the approximation error for Qto. and is an extra burden when the reward function (or the contribution to the global reward) is diverse. IQL has no such problem as it treats Qioe as the individual Q value. From this part, we conclude that using credit assignment methods with learnable value decomposition module should be avoided in scenarios with large contribution-ban role diversity.
6 FINDING A BETTER TRAINING STRATEGY
Based on our theoretical and experimental analysis, we contend that role diversity is a strong can- didate metric to aid in selecting the proper training strategy for cooperative MARL. Specifically, if the policy-based role diversity is large, we should choose a no parameter sharing strategy, and vice versa. If the trajectory-basse based role diversity large, we should avoid communication or other unnecessary information sharing, and vice versa. If the contribution-basee role diversity is large, the fixed credit assignment method or independent learning method is preferred, and vice versa. In this way, we can avoid the possible performance bottlenecks in the context of cooperative MARL. Notably, however, the question remains as to how we might obtain an accurate measurement of the role diversity before the training. In this study, all the role diversity comes from VDN and separated training with no communication because we require the bascline method our study to be robust and training-effician The fact however remains that, in some cases, roles are different in ways that are easy to notice and require no trained policy.
7 CONCLUSION
In this paper, we define the role and role diversity to describe cooperative MARL task and explain the question of why the model performance varies in different scenarios. We claim that strong relationship between the role diversity and model performance exists and we prove it through both theoretical analyses MARL error bound decomposition and experiments conducted MARL benchmarks. The experiment results clearly show that the role diversity significantly impacts the model performance of different training strategies and this effect is ubiquitous in various environ- ments and algorithms. Finally, we provide a guideline on choosing the proper training strategies for cooperative MARI. based on the role diversity description.

---

Under review as a conference paper at ICLR 2022
REFERENCES
[1] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-lange language navigation: Interpret- ing risually-groundee navigation instructions in real environments. In CVPR, 2018.
[2] Martin Anthony and Peter L. Bartlett. Neural Network Learning Theoretical Foundations. Cambridge University Press, 2002.
[3] Bowen Baker, Ingmar Kanitscheider, Todor M. Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. ICI.R, 2020.
[4] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216, 2020.
[5] Tamer Başar and Geert Jan Olsder. Dynamic noncooperative game theory. SIAM, 1998.
[6] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06633 2019.
[7] Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365 (6456):885-890, 2019.
[8] Filippos Christianos, Georgios Papoudakis, Muhammad A. Rahman, and Stefano V. Albrecht. Scaling multi-agent reinforcement learning with selective parameter sharing. In ICML, 2021.
[9] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. J. Mach. Learn. Res., 6:503-556, 2005.
[10] Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q-leaming. In L4DC, 2020.
[11] Amir Massoud Farahmand, Rémi Munos, and Csaba Szepesvári. Error propagation for ap- proximate policy and value iteration. In NIPS, 2010.
[12] Amir-massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesvári, and Shie Mannor. Regularized policy iteration with nonparametric function spaces. The Journal of Machine Learning Research, 17(1):409-4874, 2016.
[13] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI, 2018.
[14] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie; Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, 2018.
[15] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining im- provements in deep reinforcement learning. In AAAI, 2018
[16] Wan Ju Kang David Earl Hostallero, Kyunghwan Son, Daewoo Kim, and Yung Qtran. Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In ICML, 2019.
[17] Junling Hu and Michael P Wellman. Nash gelearning for general-sun anneral-umm stochastic games. Jour- nal of machine learning research, 1(Nov):103--1065 2003.
[18] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooper- ation. In NeurlPS, 2018.
[19] Daewoo Kim, Sangwoo Moon. David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning. In ICLR, 2019.
10

---

Under review as a conference paper at ICLR 2022
[20] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. NIPS, 2016.
[21] Karol Kurach, Anton Raichuk, Piotr Stanezyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly. Google research football: A novel reinforcement learning environment. In AAAI, 2020.
[22]
Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Analysis of classification- based policy iteration algorithms. Journal of Machine Learning Research, 17:1-30, 2016.
[23] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint WXX::20066002111: 2020.
[24] Hoang Minh Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In ICML, 2017.
[25] Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang, Li Zhao, Tao Qin, Tie- Yan Liu, and Asiao-Wuen Hon. Suphx: Mastering mahjong with deep reinforcement learning. arXiv preprint arXiv:2003. 2020.
[26] Long Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Mach. Learn., 8:293-321, 1992.
[27] Michael L Littman. Friend-or-foe q-learning in general-sum games. In ICML, 2001.
(28) Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiiee environments. In NIPS, 2017.
[29] David Henry Mguni, Yutong Wu, Yali Du Yaodong Yang, Ziyi Wang, Minne Li, Ying Wen, Joel Jennings, and Jun Wang. Learning in nonzero-sum stochastic games with potentials. In ICML, 2021.
[30] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529533,, 2015.
[31] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukeuoglu. Asynchronous methods for deep reinforce- ment learning. In ICML, 2016.
[32] Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008
[33] Frans A Oliehoek, Christopher Amato, al. concise introduction to decentralized POMDPs, volume 1. Springer, 2016
[34] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Bench- marking multi-agent deep reinforeement learning algorithms in cooperative tasks. In NeurIPS, 2021.
[35] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi- agent reinforcement learning. In JCML, 2018.
[36] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Uni- fied, real-time object detection. In CVPR, 2016.
[37] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NIPS, 28:91-99, 2015.
[38] Martin A. Riedmiller. Neural fitted Q iteration first experiences with a data efficient neural reinforcement learning method. In ECMI, 2005.

---

Under review as a conference paper at ICLR 2022
[39] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt. Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043,, 2019.
[40] Brumo Scherrer, Mohammad Ghavamzadeh, Victor Gabillon, Boris Lesner, and Matthieu Geist. Approximate modified policy iteration and its application to the game of tetris. Journal of Machine Learning Research, 16: 16:1629-167, 2015.
[41] Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics, 48(4):1875-1897, 2020.
[42] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICMI,, 2015.
[43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: /rXivv:/77.066: 707.06347, 2017.
[44] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint 1rXiv:1712.01815, 2017.
[45] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In ICLR, 2019.
[46] Joseph Suarez, Yilun Du, Clare Zhu, Igor Mordatch, and Phillip Isola. The neural MMO platform for massively multiagent research. CoRR, abs/21 2021.
[47] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In NIPS, 2016.
[48] Peter Sunchag. Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decome decomposiionn networks for cooperative multi-agent learning based on team reward. In AAMAS, 2018.
[49] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In ICML, 1993.
[50] Justin Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Re- visiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005. 13625, 2020.
[51] Karl Tuyls, Shayegan Omidshafiei, Paul Muller, Zhe Wang, Jerome Connor, Daniel Hennes, Ian Graham, William Spearman, Tim Waskett, Dafydd Steel, et al. Game plan: What ai can do for football, and what football can do for ai. Journal of Artificial Intelligence Research, 71:
41-88, 2021.
[52] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki. Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grand- master level in stareraft ii using multi-agent reinforeement learning. Nature, 575(7782):350- 354, 2019.
[53] Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye, and Chongjie Zhang. Towards un- derstanding linear value decomposition cooperative multi-agent q-leamnnng arXiv preprint 2rXiv:2006.00887 2020.
[54] Tonghan Wang, Heng Dong, Victor R. Lesser, and Chongjic Zhang. ROMA: multi-agent reinforcement learning with emergent roles. In ICMI., 2020.
[55] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng. Shimon Whiteson, and Chongjie Zhang. RODE: learning roles decompose multi-agent tasks. In ICLR, 2021.
12

---

Under review as a conference paper at ICLR 2022
[56] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In ICML, 2018.
[57] Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, al. Mastering complex control in moba games with deep reinforcement learning. In AAAI, 2020.
[58] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
[59] Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu. Douzero: Mastering doudizhu with self-play deep reinforcement learning. In JCML, 2021.
[60] Kaiqing Zhang, Zhuoran Yang. Han Liu, Tong Zhang, and Tamer Basar. Finite-sample anal- ysis for decentralized batch multi-agent reinforcement learning with networked agents. IEEE Transactions on Automatic Control, 2021, doi: 10. 119/TTAC.202..3449222
[61] Lianmin Zheng, Jiacheng Yang. Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. Magent: many-agent reinforcement learning platform for artificial collective intelligence. In AAAI, 2018.
[62] Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, and Yizhou Wang. Towards distraction- robust active visual tracking. In ICML, 2021.
[63] Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang. David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, et al. Smarts: Scalable multi-agent rein- forcement learning training school for autonomous driving. arXiv preprint 0000::2000009776, 2020.
[64] Fengda Zhu, Siyi Hu, Zhang, Haodong Hong, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Main: A multi-agent indoor navigation benchmark for cooperative learning. 2021.
13

---

Under review as a conference paper at ICLR 2022
A PROBLEM FORMULATION
Multi-agent Reinforcement Learning A cooperative multi-agent task is a decentralized partially observable Markov decision process [33] with tuple G = (S, A, P.r. 2, O, 7). Let S denote the global state of the environment, while A represents the set of n agents and U is the action space. At each time step I, agent a E A {1, ..., n selects an action u E U. forming a joint action which in turn causes a transition in the environment represented by the state transition function 21 x x [0, 1]. All agents share the same reward function r(s, u) : S XL" R while E [0. 1) is discount factor. For any state-action pair, the reward is bounded by M, i.e. |r| < M. We consider a partially observable scenario in which each agent makes individual observations z E Z according to the observation function (s,,a) : A + 2. Each agent has an cetion-observatton history that conditions a stochastic policy Te. creating the following joint action value: Q uz) Be [R Ze, un]. where Rt
U?"
P(s' is u)
-00 - 7 is the discounted return.
Centralized training with decentralized execution Centralized training with decentralized execu- tion (CTDE) is commonly used architecture in the MARL context. Each agent is conditioned only on its own
history to make a decision using the learned policy. The centralized value function provides centralized gradient update the individual function based on its output. Therefore, stronger individual value function can benefit the centralized training.
B PROOFS
In this section, we present more detailed results and the proof' for the theoretical analysis in Sec. 4. In Sec. B.I, we denote more notations and state the concentration property of Markov decision pro- cess. Sec. B.2 presents two useful lemmas about the error propagation and one-step approximation respectively. In Sec. B.3, we consider a simple example of the decentralized and cooperative MARL and provide the saitee analysis for the estimation error of the joint action-value function. We use the value decomposition [35, 48] and the finite-sample results for single-agent RL [10]. For more related results about MARL, please refer to [60] and [53]. Sec. B.4 studies the parameter sharing case that all agents share one deep Q network. In Sec. B.5, we assume Q is a sparse ReLU network and TQ is composition of Hölder smooth functions. Then we discuss the convergence rate of the statistical error as the sample size tends to infinity. According to Sec. B.3, B.4 and B.5, one can find that each type of role diversity have different impact to the decomposed estimation error. Furthermore, we explain the benefits of the training options, e.g. parameter sharing (Sec. 5.1), communication (Sec. 5.2) and credit assignment (Sec. 5.3), and discuss how these options impact the convergence rate of approximation error and statistical error.
We summarize our results as follows:
The parameter sharing strategy introduces a bias term by constraining the diversity of indi- vidual action-value functions, which corresponds the policy-based role diversity. At the same time, it speeds up the convergence rate of statistical error by pooling training data.
The communication mechanism reduces the variance caused by the trajectory-based role diversity but slows down the convergence rate of approximation error by introducing more active input variables.
When the contribution-base role diversity is nonnegligible, the credit assignment sig- nificantly reduce the estimation error of the action-value function.
B.1 MORE NOTATIONS AND ASSUMPTIONS
We denote the joint optimal action-value function by
Dio (z, u) = (Z. Zn. U1 un).
where is the global state of the environment, the action of each agent u, and & is the observation of the agent i generated from the emission distribution z; S). We further denote the individual optimal action-value function by Q and write
{ui ... u,) is the action set that collects
Q² u) (Q](21,
u1),.... ... (a)))..,(h((.... (%,(2n,,))
14

---

Under review as a conference paper at ICLR 2022
the vector of all agents' action-valls value functions. According to the value decomposition assump- tion, the joint optimal Q-function Qio: can be approximated with
F(Q")(z, u) (Z1 ui),..., Q7 un)),
where F F is credit assignment function. The VDN method [48] approximates the joint sum of individual action-value functions that condition only on individual observations and actions. Then decentralised policy arises simply from each agent selecting actions greedily with respect to its Qt Since n is fixed integer, we write the hypothetical space of credit assign functions that only contains one function as:
1 n n
F F=(((Q)
- Qi with Q = Qn))..
The QMIX method [35] generalizes the value decomposition scheme and prove that if
aQ u) 2 0, for L1 visisn Az E Z", Vu Ut"
(Q. u.)
then the global arg max performed on joint Q-function yields the same result as a set of individual arg max operations performed on each agent Q-function, that
arg max,, Qı arg max Q2 U2)
arg max Q(z, u)
arg max Qn (7 un))
Motivated by VDN and QMIX, consider simple throughout this section:
(F(Q) w QQ w A, and Q (Q1..... Qn)),
where A, is the dimensional probability simplex.
Suppose the individual action-value function estimated by the fitted-Q iteration (FQI) algorithm [9, 38]. At the iteration < é S T' we write Qú, and Ti, as the output of FQI algorithm and the corresponding greedy policy respectively. Let Q be the Q-function corresponding to Ti,.- Then the joint action-value function is estimated by F(Q.), where
Qz u) = (Q (Z1, u1), 2 (Z2, Uz),... QTn.. un)).
To proceed further, give the following assumption that controls the similarity between two prob- ability distributions under the Markov decision process.
Lei μ.v E P(Z U) be two probability measures that are absolutely continuous with respect to the Lebesgue measure on Z x U. Let {me] } be a sequence of joint policies for all the agents, with n; Z P(U) for all time Suppose the initial state-action pair (zo, uo) has distribution and the action u, is sampled from the joint policy ne For any integer we denote by psm pam-1... parp the distribution of {(4,, 1 under the policy sequence { Then, the m-th concentration coefficient is defined
Assumption 1.
mm pam-1 pm 11)
27 1/2 1/2
x(m; μ1 v)
sup
E,
du
where d(P"mmpt 1... p p dv is the Radon-Nikodyn derivative of pT p.m 1 .... .......ppp with respect to v and the supremum is taken over all possible policies.
Furthermore, let be the stationary distribution the samples {lz,, u,}} from the Markov decision process and let be a fixed distribution on S x U. We assume that there exists constant &n such that
(1- 71-7)2.
Σ 7 m- k(m; U, S 6p...
m21
15

---

Under review as a conference paper at ICLR 2022
To proceed further, denote Q the space of individual Q-functions and let
w @ - sup inf
IQ' - -TQ
Qeo Q'EQ
where || 112,,> as the Le norm with respect to a probability measure v. In the following, we take v as the independent data sampling distribution in the FQI algorithm, e.g. experience replay [26].
We say a collection ,Q. Q*) } S @ is an ô-cover of Q for each Q E Q, there exists Qh such that ||Q Q*1 I 5. The d-covering number of with respect to is
(Q, 6. - II) = in E N there is an d-cover of Q with respect to -1I..
In the following, we take the (2 x U) norm on Q by
IQ - Q'lll= = sup
1Q(z, - Q'
(x.
For the sake of simplicity, rewrite N(2, 6, |I-1)) as Ng. In Sec. B.3, we study the estimation error of the joint action-valus function and prove that the statistical error depends on In No/N where No the // 1/N-cover of Q and N is the sample size. For the parameter sharing settings, the sample size increases while the cover number also increases due to the smaller Thus, we still do not know whether the parameter sharing improves the convergence rate of the statistical error. So we present a fine-grain analysis to discuss the convergence rate in Sec B.5.
B.2 USEFUL LEMMAS
Lemma 1. (Theorem 6.1 [10]). For each agent [n], we denote (Qis) as the iterates of FQI Algorithm. Let Ti, be the one-step greedy policy with respect to Qi,, and let Qu be the action-value function corresponding Tia Under Assumption we have
2 max (1 7) ET(
4T
|| - -
+ (1 71
M.
Proof: Please see Appx C.1 of [10] for complete proof.
This lemma quantifies the propagation procedure of each agent action-vait value functions through each iteration of FQI Algorithm. The first term on the RHS is the one-step statistical error and will not vanish even when the iteration goes infinity (T oc). For more related error propagation results, please refer to [11, 12, 22, 32, 40].
Lemma 2. (Theorem 6.2 in [10]). Let [(zij [g))) be N i.i.d. random variables. For each j E [ let rey Fij and 2iy be the reward and the next state corresponding to (Zij, Ugz). In addition, for any fixed Q1,--1 E Q, we define Yi = rij + + max, 1..-1(1/2) u). Based on {(zig, unj Ytş) }e(])
define Qi, as
Q.1 arg min
1
Q(1z,, - -Yr)
QEO
N
==1
Then for any 8 O, have
I - Quella < 4w( + C M In
M8
+ C
(8)
(1-7)² N
1 1-7'
where w (Q) = supqeq inforeo ||Q' - - and N₂ is the 8-covering number of Q with respect to the norm Ioo.
Proof:
Please Appx C.2 of [10] for complete proof.
16

---

Under review as a conference paper at ICLR 2022
B.3 INDIVIDUAL Q-FUNCTION
Theorem 1. We consider the separated strategy that each agent has its own action-value function and reward. All agents' learning process is independent. Suppose (oisT are the output of FQI Algorithm for the agent i. Let Ts be the one-step greedy policy with respect to Qi and let Q be the action-value function corresponding to Tt, We rewrite
Q = -Q _==.. (5...) and Q (Q.
Recall that 0 S < is the discount factor, the reward function is bounded, i.e, Ir(s, u)l S M, Q is the space of individual Q-functions and ( = supqeo infqreo l|Q' TQ Then, under Assumption 1, we have
11 F(Qi)l1a
11Qtot (w") TQ
+vn x || - w/l x
Varn (Q*
1.
40p.r7 vw(() + oly
In Noo)
47
N (1 7)2 M,
(1-
where Ny is the 1/N- covering number of Q with respect to the norm
||
20 and
- Σ
(1 u.) -1
1 Σ 0z
Var, (Q)
71 1=1
n
Proof: It easy that
1Qutt F(Q
1 F" =0))
F(Qi))li.
S 11Qrot F" F² (110)==(())) + +1=F(1I) -
Here the first term at the RHS of (9):
1Qi - (Q')))12, = l1ii - (w") ||ا,
(10)
represents the best achievable estimation error under the value decomposition assumption. Next we consider the second term in the inequality (9):
|| (Q") - F(Q") ||,, [1(w") Q - w TwQ 1|,
Il(w w Q'
For any given (Zi,..., Zn) and u ..y.....,uu,)
n Σ (wi -w: (2; u:)
(w" - (w'-w) T")Q""Z,.uu
Σ (w] w:) (2, - - ())
n
The second equality holds since E/= (w: - -w,) xe 0 for any constant c. By the Cauchy-Schwarz inequality, we have
2
12
1 n m
(w" (w--w) (w--w)TQp (x)
S
Σ
(wG w: Σ
Σ Qi Q7 u:)
= ||w - -wl x Var (Q (Q.
where
Var, (Q', u)) = 1 7
Σ
(1(2)))))
1 n
Σ Q7 ui)
1=1
17

---

Under review as a conference paper at ICLR 2022
is the variance of the output vector of Q" given and Plugging the positive upper boundary of
(w" w)T Q" into the expression of [ (Q") F(Q*) (l, we obtain that
- |
I| w| v Varn (z, uj)
1.
-vax || - -wl x | Var (Qr (2,
u))
(I1)
1,
Here the term |ly Var (2, u)) 1 stands for the diversity of the agents. Finally, we deal with the third term on the RHS of (9). Notice that
[I((Q)) - ((Q)) =
(i - Q7 112
n
Wss 11 - Q, 11
Therefore, it suffices to consider the deep Q-learming procedure of each agent separately and to give upper bound of ||
Q || for each [n]. According to (7) and (8),
[ Qieell, + 4~T+1
||Q; - Q77 11,
S
max
(1 - 7)2 M
(1 7)²
rE[T]
2
M
In + C 1 1-7 + (1 (1- 7)2 M
4,
<
4w(2) + +0
7)2
(1 71-
4
2M Y
In C 22 2. RVMMOuT v8
4-
((Q) C
(1 71- M.
7)²
(1 - 73-
(1 7/5/2
We take = 1/N and write No as the -overinn number of Q. Then, we have
40p.07
In No
+ 47
|1 - 17) 11i, I (1- (1->)² (w(@) + ( +1
(1 -1- M.
N
Furthermore,
4
In No
4T
[I((Q)) FQQ
(1 7)2 VW ((()) +(
+ (1 - 7)² 7)-
M.
(12)
Combining the results of (10), (11) and (12), we know
11 - F(Q:)111, I 11 (w") (w')TQ"ll
+vnx x || - w| x
v Varn (Q)
1:31
40
ww( + olV Lmn
47 (1 ()- M.
INN +
7)
Remark. The term
Vn ||w" w|l
|1
Var, (Q" (Q (z, u))
1,2
shows the benefits of learning credit assignment, where w stands for the best credit assignment scheme. Here we assume w is given and do not take the modelling and learning of w into con- sideration. In practice, is the output of a credit distribution network and learning procedure also influence the convergence properties of individual Q-fuctioons. On the other hand, Var, (Q) corresponds to the contribution-based role diversity in Sec. 3.3. Therefore, when the variance is nonzero, good credit assignment w can the estimation error. For the parameter sharing case in the next section, we decompose the variance term into the sum of a bias and a variance caused by policy-bacc role diversity and the trajectory-basse based role diversity respectively. This decomposition does not always hold. Here, we assume that all agents' learning processes are independent and that each agent has its reward function. In practice, these assumptions idealistic and limited.
18

---

Under review as a conference paper at ICLR 2022
B.4 SHARED Q-FUNCTION
Theorem 2. We consider the parameter sharing strategy that all individual agents shares one action- value function. Suppose (QrhogieT are the output of FQI Algorithm. Let ne be the one-step greedy policy with respect to Qt, and let Q be the action-value function corresponding to Tt. We further denote Q as the optimal shared action-value function and write
Q,(z, u) Q²(z.u) (1, Q (Z2, U2), ... Q' (Z,s Un)).
(Q (z), QTH (Z2, u2)..... Q (Zn: Un)), (Zn:
Recall that 0 S < is the discount factor, the reward function is bounded, i.e. Ir(s, u)] S M, Q Then, under
is the space of individual Q-functions and ( = supqeo infqree 11 IQ Assumption 1, we have
11 - F(QU)lI I 11Qior - -(w") TQ" +
Σ w ((Q2-QQ))
Q")
1,14
+v x
- -
x |
Varn (Q (z, u))
1,14
40p,u7 7)2 ( + OlV
In No
+ 4- --
7)2 M.
n.
where No is the -conNi-covering number of Q with respect to the norm
|ox and
Var, (2,
1
(7)
n Σ @² u:)
1
n
Proof: Similar the arguments in (9), we have
11 ((Qu)
S 1 F² + (Q'))="(Q'')ILA
+IFF' (Q') - f(Q*)ll,, + - -
(13)
The term liQ - F (Q") caused by the value decomposition is the same that in (9). So (10) still holds. For the second term on the RHS of (13),
[IF'(Q") (Q"))
ll(w" T(Q' Q") 111,
a
W
@*)
1.
which is the bias term caused by the parameter sharing. Next, we turn to a turn that is related to the trajectory-baan -based role diversity. Similar to (11), we know
(F*(Q)) (z, (w" TTQQQT TTQ (2,
Σ (w. =w.)
1=1
Σ (wr wi) Q- (²((i,,u) - - (
i=1
= vn x || - w|l x / Vaar (Q u)).
On the other hand, by (7) and (8).
20
T,T+1
||Q" - Q=e 1a
max 6E12"
[ - -illz, + (1 7) M
7)2
2
M In InNs
C² MR
+ T+1
<
( C
M
(1 - 7)²
(1 7)- nN
1- (1 - 7)2 71-
40
2M 2MouNY N
+ C² 2/ 2 v8 +
47 T+1
<
- 7)² v ((()) + C
(1 71- M.
(1 7)3
nN
(1 (1- 3//2
19

---

Under review as a conference paper at ICLR 2022
We take ò (/nN)) and write No as the 1/(n number of Q. Then, we have
1| - Q- s
(1- (w(2) + +Ol +0
40p.v7
In N
+
(1 - M.
47 T+1
n.
Therefore,
4 (1 7)2 VW (/((() (2) OVV
In No n.V
+ 47
[[((Q)) - F(Qull,
7)2 M.
Summarizing the above results, we have
|| F(Q:)llu
11 - (w"))QQ"l Σ wi(Q - Qq")
1,14
+ x l| - (w"-w/)
x
Varn (Q- u))
4 7)3 (w + O(V
In No
+ 47 M.
nN
(1 7)2
B.5 CONVERGENCE RATE
Similar to the Theorem 4.4 in [10], we assume that Q belongs to family of sparse ReLU networks and TQ can be written as compositions of Hölder smooth functions. Here T the optimal Bellman L+1 operator. We start with the definition of a (I. + 1) layers and {d, } width ReLU networks:
f(x)
aW (WW.- + 01) + v2)... + UL):
where a is the ReLU activation function, and W; and ty are the weight matrix and the bias in the 1-th layer, respectively. The family of sparse ReLU networks defined as
{F f:
M
Wi 1 [E[L+1]
max
Σ
Willo s, max je(dz+1]
I I
I-
1=1
where Wi is the parameter matrix that contains Wt and by and ʃj the j-th output of f. On the other hand, the set of Hölder smooth functions is
100 s 80x f(v)]
C, H) J:D
Σ a:/a
10 Sllx +
Σ
sup
H
| y11 11
a:||a||1=1
where D is a compact subset of R', I-] stands for the floor function and 0a = mm g ... 8ah with (a1, ..,.),.,,))) Furthermore, we write G([p,, 44. Bj, Hejl)la)) as the family of functions that be decomposed into the composition of sequence of Hölder smooth functions {gr {gg))elgl-
That is, for any function f G((p,, ty 33, H, )
f = 9, o 9 ................................ 92 o 91
where for any k E [pj41] and j E [q]. the k-thh component of the function gj is a Hölder smooth function, i.e. gik E G, (ay,,,]' b, ",,,H,, H,). For simplicity, we take Pj+1 = 1. Here we assume that the input of yjz is dimmensioonal where 1j can be much smaller than pj. More specific, the deep Q network we used is a sparse ReLU network for any given action u. Therefore, we rewrite the space of individual Q Q-functions Q as
Fo= Sx
(4) (4L, (
for any EU-
Furthermore, for any Q E Fo. we assume TQ composition of Hölder smooth functions and belongs to the following family:
9n = ( S S
RR: f(,u) 901r.14.8 H3))ew)) for any E U).
20

---

Under review as a conference paper at ICLR 2022
To proceed further, denote
b 23j 11
a" max JEgg
3j 3g
II 1=f+1
min(3t, 1), and Bg = 1.
Now we are ready to state the following result.
Theorem 3. Suppose the assumptions of Theorem 1 hold and for any Q E Fo. TQ E Go. where T is the optimal Bellman operator. The sample size is sufficiently large such that there exists constant & > O satisfies
Σ
(ty + +3 +1)+t,,
Ein(t, +3), max Pj jElg
S (1n
max
iE(a]
The network architecture of the Q-function is well designed such that
S (In r min dj S max dj N and 8 Na (In Nys
jELL]
jELL]
for some constant 5* > +++ 25. The number of iterations is sufficiently large, such that
T a"1a""n")
where C" is a constant. Then, under Assumption 1, we have
11 - 1(Q
$ l1Qioe (w*) TQ"
+Vn x || - -wll x
Varn (Q"
+0 (InN))+22 N minsele zaytes
Proof: This is direct conclusion reached by Theorem 4.4 in [10]. That is, for any agent i E [n].
T+1
|0 - 2 11,,
S O (11 (1a2)11/2) + (1 7)2 M.
The approximation error in Theorem that involves w Q) is bounded above via Theorem in [41]. The upper bound for the cover number No is derived from Theorem 14.5 in [2]. Please refer to Section [10] and Sec. B.3 for a complete proof.
Remark: Note that
T+1 4
M
+ as T T 00,
(1 7
which is the algorithmic error that converges to zero linear of T. In Theorem 3, we assume T' is sufficiently large such that this error is negligible comparing to the statistical error. If we ignore the logarithmic term, the convergence rate of the statistical error is about
max N jE
Here 88 and ty are parameters of the functional space of TQ. Therefore, the parameter sharing (Sec. 5.1) keeps B and t, unchanged and increases the sample size N to nN by pooling train- ing data. In addition, t, is the number active input variables gj- Thus, the communication mechanism (Sec. 5.2) slows down the convergence rate by enlarging 'y-
21

---

Under review as a conference paper at ICLR 2022
C OBSERVATION OVERLAP PERCENTAGE CALCULATION
C.I OVERLAP PERCENTAGE CALCULATION GAMES
In this part, we demonstrate how to calculate the observation overlap percentage in SMAC [39]. As the partial observable area is circular, and the coordinate system a 2D map with axis and Y, the observation overlap one battle scenario can be computed as:
V (Tag Tas) (Tas Too) + (Nao ya.) (yar - yoo
(1 2 /2
8 vp. (p 1--18 1) 1p
(p- r) (p
(14)
0 = cos (1/(2 (2) r
00 (/ (²) dr
Here r is the vision scope. Notice that if < 2r, dr equals zero no overlap exists. We provide the observation overlap curve in Fig. 2b to show how trajectory-based role distance varies in one game.
C.2 OVERLAP PERCENTAGE CALCULATION IN REAL WORLD SCENARIO (SEMANTIC)
Bathruom (tarpet)
Kitchen
I want to find the bathroom with a shower and toilet. The bathroom is opposite the bedroom. There bed cabinets In the bedroom. On the right side of the bedroom is the living room. There sofa, television and coffee table in living room, The opposite the living room the kitchen, The kitchen has dining table, dining choirs, and slove, The balcory right of the kitchen, there chairs and green plants on the balcony.
upent's from balcomy to living room: Trajectury: green plants 5000 coffee table tolevision
Green agent's from living room to kitchen: Trajectory falevssion, sota dining table, stove
Bedroom
Living Room
Orange agent's ' from bedruom to bedroom 1: Trajectory: bed, cabinets, bed
Bedroom 2
Bounding Box
Trajectory
Shared Detected Object
Figure 5: A multi-agent visual language navigation task. Agents are initialized in different locations and the target description given. Agents need to cooperate with each other find the target location according to the description as soon as possible.
In this part, we demonstrate how to apply the observation overlap concept and trajectory-based role diversity calculation (Eq. 3) to real-world scenarios. Different from game scenarios like SMAC and MPE, the observation of real-world tasks is usually an image. For example, in the vision language navigation task (VLN[I)), agents take real indoor scene pictures as the input, combine them with language description to locate the target as shown in Fig. 5. Considering the learning efficiency, ob- ject detection techniques like YOLO[36] and FasterRCNN]33 are used VLN to help extract the objects from the scene pictures semantic information. The semantic information can be recorded as part of the agents' trajectory, enabling agents to use the past information for future decisions. Under the multi-agent setting, agents required cooperate and find the target together. There- fore, trajectory overlap should be avoided, which means that large trajectory- preferred in this task, and policies that cause trajectory overlap should be punished. Directly using scene pictures as input or its feature pattern can bring large noise in observation overlap calculation. Instead, using semantic information from the detected object can significantly reduce the noise and serve as a good observation history representation. As shown in Fig. 5, the red dotted frames indicate that the blue agent and green agent share some similar observation semantic in their trajectories. In this way, the trajectory-based role diversity of multi-agent VLN task can be calculated the same as Eq. 3. Only the de..0: is replaced by the observation semantic overlap, which is the shared detected object percentage in total detected objects. In this way, without knowing the exact trajectory, still manage to calculate the trajectory distance. And using overlap to represent the trajectory- based role diversity, keep this metric in a fixed range from to 1.
role diversity is
22

---

Under review as a conference paper at ICLR 2022
C.3
OVERLAP PERCENTAGE CALCULATION REAL WORLD SCENARIO (RAw)
Agent
Obervvtion
Pretrained Backbone
Feature vector
Overaap calculation ways)
Veclor wise
Overlap cosine similarity
Channel wise
0 and M
Overiap = M/(N+M)
Figure 6: Using real images observation overlap percentage calculation. Two methods are pro- posed including vector-wise cosine similarity and channel-wise threshold-bes basdd similarity percent- age. A detailed discussion can be found in Sec. C.3
is also possible that we can get observation overlap directly based on real image MARL tasks. As showed in Fig. 6. Passing the input image pre-trained CNN/Transformer backbone and getting its feature, we can use cosine similarity or channel-wiss similarity to compute the overlap between different observation features dl. dfo in Eq. 3. However, these methods can bring large noise this metric. Moreover, how to stabilize the reinforcement learning with real pictures as input is still under investigation.
addition, it rare in MARL tasks that the only information provided in the training stage is one single image. Location and communication are necessary auxiliary information help learn the coordination of agents in most MARL tasks. Therefore, simply using the raw image to calculate the observation overlap can be choice, but not the best choice.
D TYPES OF ROLE
We present two illustration figures for different types of role based on MPE [28] and SMAC [39]. Fig. is based on MPE. Grey circles and black circles represent agents and goals respectively. Dashed arrows in different colors represent different actions. Larger circles receive more rewards when they reach the goal.
(a) policy-
(b) trajectory sajeetooy--based based role
(c) contribution based role
Figure 7: An illustration of different role based on MPE.
Fig. 8 is based SMAC. detailed explanation can be found in the caption.
E CONNECTIONS OF DIFFERENT ROLES
Is there any redundancy in the definition of different kinds of role diversity in Sec. 3? Here discuss the connections of different role diversity.
From the theoretical perspective, the contribution- based role diversity is compound description of role diversity. It corresponds to the variance term in (5). For the parameter sharing case, we decompose this variance into sum of two terms: bias term corresponds to the policy-based role
23

---

Under review as a conference paper at ICLR 2022
122
H+0
++30
10
Store
Mow
Clservatoo
120
Agent
Tojotbey of apuat
Target aaemy
Tojedtay agen?
1833
Boll: nove-tttackccoeeee Fight
Tejedtyy of aeget
(a) Policy-based role difference in period from t-2 (b) Trajectory based role difference in a period to 1+2. Action statistic shows only two different roles from (-2 to L+2. The area covered by the grey dot- among three agents.
ted line is the observation overlap of agents and There is overlap for agent 1.
Atribatac: Lig Delrece
Aiteck Gourd
(Pr Madeel
Denage SHTTI22
0.5
large:
Grand//
Citriba avonnd Mosion
Acauc it
Tarpet Grond
Garge
150
DPC
Dekrve 2V
10 Prb
(c) Contribation-baseed role difference depends on agents' original attributes including attack method and defense.
Figure 8: Illstratioon of policy-based role, trajectory-basee based role, and contribution bassed role on real scenarios from SMAC.
Secnario
Semanic Policy Diversity
Real Policy Diversity
Trajectory Diversity (overlap)
Cocmrbttion Diversity (max Q)
4mn_ys_
0.47
0.13
27
3.3
19.3
0.31
0.06
33
121
0.35
0.25
132
0.31
0.61
27
158
0.25
0.75
225
0.18
0.82
Table 4: Different role diversities on different scenarios from SMAC. The minimum value of one column is labeled in green and the largest value is labeled in red. Detailed analysis can be found in Appx. E.
diversity and variance term corresponds to the trajectory-basee role diversity. Therefore, under the simple scenario Sec. 4, we can find clear relationship between different role diversity.
From the experiment perspective, the decomposition in (6) may not hold because of the more compli- cated settings. We have discussed this issue in the remark on Page 18. Here we collect all different role diversity data in Table. 4. We find scenarios like 3s 5z have relatively small diversity in the policy-based role while the observation overlap of trajectory diversity small. Scenarios like 4m-vs_5m have small policy-based role diversity while the observation overlap is large. Contribute- based role diversity can not inferred from the policy diversity and trajectory diversity, and is more depending the agents' behavior difference.
In conclusion, the relationship between different roles exists MARL training theoretically, while this relationship is not significant in the experimental perspective due to more complicated set- tings of the real MARL tasks. Yet the strong relation between different role diversity and the MARL training process still exists with conflict with the conclusion of this paper.
24

---

Under review as a conference paper at ICLR 2022
F PARAMETER SHARING
Four different parameter sharing strategies are tested in our experiment including shared, no shared, partly shared, and selectively shared[8]. For partly shared, we only shared the GRU cell across dif- ferent agents while keeping the embedding layer of the policy function model separated for each agent. For selectively shared strategy, we reproduce the grouping results following[8). An illustra- tion figure can be found in Fig. 9.
Envirnment
Envtronzent
Eovinvame
Envinnmentt
Seporatedd
Fully Slaned
Purtly Slardd
Selecthody Staret
Figure 9: An overview of how knowledge sharing works with the MARL framework. Fully shared, partly shared, shared (separated), and selectively shared[8] strategies shown here. The same color indicates the same policy function part across different agents. Dash line represent only sharing no gradient backpropagation.
F.1 SELECTIVELY SHARING THE PARAMETER
Here we provide the selective parameter sharing strategy result in Table. 5 as a supplement for Ta- ble. 1. The main purpose for doing so is to verify whether this method can serve as general solution for parameter sharing strategy choosing issue. Selective parameter sharing strategy partitions the agents into the different groups automatically with an encoder-decoder model. However, the parti- tion process is before the MARL training stage, which can not fully catch the policy difference. And according to the grouping result column in Table. 5, the selective parameter sharing strategy tends to divide agents by their initial attributes. This works well in scenarios like Is halmhhllMvv... and 1clslz vs Icls3z but ignores the fact that the same type of agents may evolve to different functions during MARL training, which is the weakness of the selective parameter sharing strategy.
Benchmark
Scemario
Role Diversit
Worm up
No shared
Shared
Selectively shured
Groppng resulos
4m
1379
63
+3.7 +4.
+5.77 +6.1
+5.07
all shared
3z
+837
+8.27
all Shared
517122
all shared
SMAC
1127
4007+12
+057=12
shared
Tela,, Tolat
427/434
Tes1s+1
247132
16.2
+347+33
+347+3.
+347=33
Im- Ih-IM
141mthiM va 4z
27/153
478/+1166
+5./+100
+7.87
Ise Im
62/2225
62
-64/-9.
+37/46.1
+647-9.1
Ise Iasl Ih-IM
Table 5: Model performance including selective parameter sharing as a supplement to Table. 1. The grouping result is provided in the last column.
G COMMUNICATION FRAMEWORK
The communication mechanism is important for MARL. The information shared can be location, action, and partial observation as showed in Fig. 10a. In many cases, communication is optional where the agent should learn when to communicate and how to ingest the information (dash line in Fig. 10b). In our experiment, we only consider the observation sharing method where the support set of policy functions contains both self partial observation and the aggregated observation information from other agents. The aggregated information is obtained by getting the mean value of other agents' observation and concatenate with the self partial observation. This means the support set of policy functions is now much similar to the global state.
25

---

Under review as a conference paper at ICLR 2022
ErV
inbeaction
communicction
(a) Communicate with others: where am, what (b) Communication is optional during optimization. see and what will do
Figure 10: Communication works as a supplement part for MARL, under the CTDE framework. (a) The sharing information can be current status or future policy, as the extra information for the decision making. (b) Learning when and how to communicate critical to help policy learning.
H CREDIT ASSIGNMENT
Credit assignment is the key part module for cooperative MARL, especially for the value decomposition- method as it leverages the reward signal to each agent by approximate the Qtot- Then the learned individual policies combine to form a joint policy interacting with the MAS.
Joial Policy
Credit ssinment
Individual Rew ard
Individual Reward 2
Individual Reward
Figure 11: Credit assignment method focuses on assigning the proper individual reward from the total reward to update.
1 EXPERIMENT TABLE & CURVE
Readhoark
Sumoda
Sharin mwdh
TANC
MADORG
MARIN
MAAT
OMNN
Teg
200
Acveneyy
Baial
BRTFROTTH HOOP
RAST I84 TR4
RERTIMETING TESETICK
KITSATIC
XSY 387363711 112
Deehiipras-44
DET AETADERIINN
EETERTEST
247377337
SturiSu
SEPEITES
N2T DETTET 113
107407 39
SMAC
43753713
C2 200
12571597144
dund
187017635
Table 6: Policy-based role diversity influence the performance of different parameter sharing strate- gies on the MPE [28] and SMAC [39] benchmarks.
26

---

Under review as a conference paper at ICLR 2022
abs overlpp
baseline
baseline
5m
66711,5711
40.5
0.37
7.5/16/195
1971987994
00.00.1
leista
123/159/179
124715.77KX
0.21
54/129/164
5471247153
43/132//5.1
84/160/198 19.8
7971537190
Table 7: Model performance with and without communication. The performance is recorded in the "warmup performance half steps performance final performance' pattern. Detailed analysis can be found in Sec. 5.2.
14
Bloodd
1.0
1.5
0.5
1.0
Sey
Stap
Ship
(a) 3s_vs._5z
4m VS 3z
(c) 4m_vs_4z
(d) 40_44__55m
That Tre 500000
Partt
50
1.0
1.5
0.3
0.5
1.0
20
00
0.5
3.0
1.5
20
Skp
(e) lclslZ.ss.els32
(0 lslm/hlM.vs.32
(g) Mssm
(h) Is1mIhIM vs Sz
Figure 12: Policy learning curve with different parameter sharing strategies.
20
373
Tec Spw.k
225
32
The
1.5
(a) 38 vs 5z
(b) 4m vs 3z
(c) 4m__s_42
(d) 4m_vs_55mm 55-5m
20
t 2
Tea Din Reveet
373
Tee
2.5
1.0
1.5
00
0.5
3.5
25
Skpp
Sep
Skop
(e) Iclslz.vs.Icls32
( IsImIhIM.vs.33 3z
(g) lslm/hiM.vs.4
(h) IsImlhlM.vs.51
Figure 13: Policy learning curve with and without communication on different scenarios.
27

---

Under review as a conference paper at ICLR 2022
25.0
$7.5
One Eww.ottoooeeeooooooooooee Rovarrt
Toat Doe Eatodde
14
12
1.1
0.5
20
0.0
1.0
3.5
Step
Stpp
(a) 3s_vs_5z
(b) 4m_vs.55m
(c) IcIsl2.vs.Icls3z
20
Teel Ore Bioechh Rement
27.5
Teat Ore Rowedd
17.5
34
12.5
12.5
30
7.5
7.5
0.0
0.5
1.5
Stap
(d) Is 3z
(c) lslm/hlM_vs.4r
(f) 1sImthiM_vs.5
Figure 14: Policy learning curve with different vision scope (6-9-18).
23
33
33
12
32
(a) 38. VS. v8.5z
(b) 4m VS 32
(c) 4m vs. 42
(d) 4m v8.5m
2.0
20
1.0
10
2.5
(0) vslslz.ss..clss
(f IsImIhIM.vs vs.3z 3z
(g) IslmlhlM.vv vs..4222222 (h) IslmIhlM.v vs
Figure 15: policy trained using VDN[48] and no parameter sharing. We also provide the curve of game progress(equals to the enemy health), ally in scope and ally alive. All values are normalized from 0
Observation overlap curve of one episode game on different battle scenarios. The
to 1.
28

---

Under review as a conference paper at ICLR 2022
15.
(a) 5z
(b) 4m vs.
(c) 4m vs
(d) 4m vs.
(e) Velssz
(l lsIm/hlM.vs.33
(g) IsImlhIM.vs.
(h) IsIm/hIM.vs.5
Figure 16: Policy based role diversity(real) in one episode.
(a) 3s_vs_5z
4m_vs_3z
4m_vs_44
(d) 4m_vs_5m
X
15
(c) vsss
(f) I
(g) ls
(h) lslm/hlM.vs.51
Figure 17: Policy based role diversity(semantic) in episode.
0.5
(a) 38_v8_52
(b) 4m vs_33 33.3%
(c) kinvs_42
(d) 4m_vsm4vv... 4m_vs_5m
33
(e) Iclsiz Icls3z
(f IsImIhIM 3z
(g) IsImIhIM
(h) IsImIhIM Sz
Figure 18: Q value curve in one episode on different scenarios.
29


Reviews:
[{"review_id": "p52VTz-mAP", "reviewer": "Reviewer_97k6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ The paper provides an interesting perspective (role-diversity) to measure the difference between MARL tasks. \n+ They analyze the impact of the role diversity on MARL methods both theoretical and experimental.  The experimental results, measured by the three role diversity metrics, answered a common concern question that why the MARL model performance varies across different tasks.  \n+   The discovered guidelines about the choice of training strategies will benefit the community.\n\nWeakness:\n- The generalization of the proposed metrics. The trajectory-based role relies on the observation overlap percentage. The observation overlap percentage can be easily defined and measured in SMAC, but how to measure it in other realistic scenarios, e.g., embodied vision robot? The contribution-based role highly depends on the output of the RL model. However, there are a great number of hyper-parameters that will impact the output, and different methods may be of different hyper-parameters. So, how to standardize these metrics for a fair comparison? \n- The experimental results are unclear. For example, in Table 1, only the policy-based role diversity is reported in MPE. Why not report the three metrics jointly for a more comprehensive analysis?  similar problems do also exist in Table. 2 and .3. These can help us better understand the difference between the three metrics.", "summary_of_the_paper": "This paper study the relation between role diversity of tasks and the MARL model performance. The role diversity (difference among agents) is described from three perspectives: policy-based, trajectory-based, and contribution-based. The authors analyze how the role diversity impacts the MARL performance in theory and experiments. The empirical results show a strong relation among the three metrics and the three main topics in MARL (parameter sharing, communication, and credit assignment).", "summary_of_the_review": "The paper is well-motivated and interesting. The provided guidelines are practical. But I am concerned about the generalization of the proposed metrics in other MARL environments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZqUkOYADLkl", "reviewer": "Reviewer_SqGu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The reviewer finds the paper is rather timely and interesting. With so many algorithms being proposed, how to choose one is quite challenging sometimes. This paper proposes to use role diversity to describe the MARL tasks and the experiments strongly show the relationship between algorithm choosing and role diversity. However, the reviewer also thinks the paper needs to be improved.\n\n(1) The paper does not provide enough discussion or experiment about the connections and differences between these three role diversity definitions. They might describe different diversity aspects. But there also might be an underlying diversity definition behind these three definitions. Another question is that is one role diversity definition (such as policy-based) for one algorithm design (such as parameter sharing)? Are there any connections?\n\n(2) Theoretical analysis is not clear enough. The reviewer appreciates that this paper provides formal analysis, but the reviewer is confused about how the empirical role diversity definitions are linked to theoretical analysis. Specifically, more explanations about Eq. (5-6) are expected.\n\n(3) The definitions of trajectory-based and contribution-based diversities should be explained in Eq. (3-4). For example, what are the definitions of $d_T$ in these two equations? How to compute them?\n\n(4) The discussion of how to obtain an accurate measurement of the role diversity before training in Sec. 6 could be moved to Sec. 3. \n\n(5)  What do the figures in Fig.2 (b)(c) want to say? \n\n(6) Some minor errors. It should be $T$ instead of $t_0$ when describing the notations of Eq. (2).", "summary_of_the_paper": "To address the problem of algorithm choosing in different MARL tasks, this paper proposes to use role diversity as a metric to describe MARL tasks. For choosing parameter sharing, communication mechanism, and credit assignment strategy, this paper defines three role diversity metrics, i.e. policy-based, trajectory-based, and contribution-based, respectively. They also find that the error bound in MARL can be decomposed into three parts that have a strong relation to the role diversity. To evaluate the proposed method, they further conduct some experiments on MPE and SMAC environments. ", "summary_of_the_review": "This paper is timely and interesting, but there are still some concerns, especially the the first one in the Main Review. The reviewer thinks this paper should be further improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BJ1M2x7jP5", "reviewer": "Reviewer_mf45", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n-- Positives --\n- Role diversity is important in multi-agent environments. Being able to quantify it might allow us to better understand how to improve multi-agent reinforcement learning. I also agree that such a metric can help us understand why some MARL algorithms perform so differently across a range of tasks.\n\n-- Negatives --\n\n1. - (3.1 - Policy-based role): The frequency of actions does not seem to be a reliable metric for determining the role of the agent for the following reasons: i) there is no guarantee that actions do the same thing (e.g. action '3' can mean different things for a \"marine\" and a \"zergling\" in the game of SMAC), ii) not considering the _order_ that the actions are selected loses important information (e.g. in a maze, going left and then right is significantly different to going right and then left). Therefore I believe focusing on the trajectory as a whole makes much more sense and includes the frequency of the actions of this paragraph.\n\n2. - (3.2 - trajectory-based role): The \"overlap percentage\" of the observations is not general enough and does not seem straightforward to calculate in all environments. The authors present an example for the SMAC task in the appendix, but SMAC happens to have a circular observation radius which is then easy to calculate. There are many environments where the observations cannot be easily compared (e.g. a simple image, or even features that describe the state of the environment without a specific physical representation). In addition, fully-observable environments are not uncommon and, in my understanding, the rule breaks down in that case (the distance between observations is always 0).\n\n3. - In general, all those metrics seems to require that training has already been completed. The contribution-based role specifically requires (trained, because I assume untrained aren't useful) q-values. As such, I cannot see how these metrics can help with the training of MARL agents, since they require those trained policies, to begin with. The same should apply to the policy-based rule. In my understanding, this is not in line with the motivations and promises of this work, that a diversity metric can help with a better training strategy and performance in MARL. \n\nOther/fixable:\n\n4. - (5.1 parameter sharing): I would expect different roles to require different parameters and roles that are similar to benefit from shared parameters (which is also shown in \"...selective parameter sharing...\"[7]). You could consider a similar parameter sharing scheme on top of the \"no/partly/fully\" shared parameters.\n\n- While the writing is clear, and conveys the author's intentions, it could certainly be further polished. There are some sentences that are vague or that could benefit from rewriting (e.g. first page, \"In other word, even adopting the ... performance\", but there are other sentences as well). The document could benefit from some proofreading.\n\n- The bibliography is untidy and needs to be further polished. Many of the references inaccurately state \"arxiv\" while they have been published in conferences. \n- Using citations as a noun is discouraged (e.g. \"In [52], the role]...\")\n- (page 5) acroess -> across", "summary_of_the_paper": "The authors propose a role diversity metric to quantify the difference between agents in multi-agent reinforcement learning. The metric is used to inform the use of communication, parameter sharing, and other MARL algorithmic decisions.", "summary_of_the_review": "For the stated reasons, the proposed diversity metric does not manage to convince me of its potential usefulness, although I still believe that the research direction is promising.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6ar1lZPRmZ4", "reviewer": "Reviewer_L6y7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths: \n\n* Interesting question of how to define roles in multiagent systems and apply them to analyzing policy behavior. \n\nWeaknesses:\n\n* The paper is not well written. The paper does not do a good job at motivating why we should care about analyzing multiagent systems from the perspective of roles, or the alternative role definitions they propose. \n\nI would recommend the authors re-frame the paper narrative and improve the writing quality; I’d made suggestions under “Summary of review”. \n\n* The paper does not do a convincing job at showing that the roles are core to “cooperative training strategies for multiagent RL”. \n\nAssuming we do think roles are an important way of understanding multiagent behavior, the role perspective only shines light on agent patterns, but don’t necessarily cause improved cooperative strategies. A convincing suite of experiments that would need to be conducted would be if agents were explicitly optimized for those particular metrics with minimal impact to the nominal performance. That would be the right test to show that the decomposition does in fact play an important role in policy optimization, which is what the authors claim in their abstract. \n\n* The theoretical analysis is not well-motivated and it’s not clear whether it provides any operationalizable insights. It adds more confusion to their role diversity framework. \n\nFor example, the authors try to show that their policy-based and contribution-based role can be tied to an approximation error between the optimal joint and independent policies. What’s confusing is that their analysis suggests that agents should minimize these role metrics in order to perform closer to the optimal joint distribution. This conclusion doesn’t make sense since the work’s experimental setup and narrative is that these metrics are important in improving the policy quality.\n\n", "summary_of_the_paper": "This work looks at defining role diversity as a means for analyzing multi agent dynamics. It proposes three different perspectives on analyzing roles in multiagent systems. One is from a policy perspective (based on KL divergences), an trajectory perspective (how much do the agents overlap in what they observe), and a team-contribution perspective (weighing the effects of the agent to the team reward). ", "summary_of_the_review": "\n--- motivation --- \nI’m not convinced that better understanding roles will lead to improvements in multiagent coordination. In fact, this is not really the investigation the paper conducts: The paper demonstrates there are certain correlations between their role definitions and coordination performance, but not that this understanding results in coordination improvements. What concrete challenges currently exist in multiagent optimization --- e.g. what do the authors mean by `roles can avoid the issue of “policy degradation”` in the abstract? --- and how can these challenges be addressed by understanding how groups/teams can be factorized into roles? \n\nI’m confused why the authors break down MARL cooperative algorithms by “parameter sharing”, “communication” and “credit assignment”. These are components that don’t seem directly comparable to each other, and it’s not clear how roles are related to the components. Parameter sharing is a technical means to learning policies, communication is an assumption by which we allow additional agent interaction / information passing, and credit assignment is a general challenge when optimizing in long-horizon settings for reinforcement learning. \n\nThe authors directly propose their three different role definitions in the second to last paragraph of the introduction, but these alternative definitions need more context and motivation. It’s not clear what challenges they address and why these definitions are ones we should use for characterizing multiagent behavior. Additionally, without context or grounding in prior work, it’s unclear what “policy-based,...” roles should even refer to. \n\n\n--- method --- \n\nHow does this proposal show that agents take on the same role? If I’m finding a good position just like all other agents, shouldn’t this indicate that we have the same role? This doesn’t seem reflected in this metric. The policy-based role also seems to have a lot of overlap with the trajectory-based role --- in some sense, the trajectory based role feels quite redundant: If I can infer what role the agent has, I should be able to have a good generative model on what the agents’ most likely trajectories should look like. \n\nIn Equation 1, a sum over the policy conditioned on an action is an unnormalized probability distribution. It’s not clear to me how this is being used in the KL metric. \n\nIn Equation 2, the normalization should happen over A^2, and not A. \n\nIt seems weird that the trajectory-based role is defined as the observation overlap. Observation overlap seems to be a measurement of … pairwise observability (i.e. how much information do I have about what you see, and you about what I see?), and less about roles. \n\nI disagree with the authors on their interpretation of Q values on “contribution-based roles”: in the setting where agents are performing individual policy optimization, I don’t see how Q values [optimized to reflect the agents’ expected return, and not a ratio between the outcomes of an agent’s actions and all other teammates] should not be interpreted as the agent’s contribution to the team. If there’s a misunderstanding or some related work that asserts this, the authors should elaborate. \n\n--- theoretical analysis --- \n\nThe notation is not self-contained in this section, e.g. “Please refer to Sec. B.1 for the detailed definitions”. \n\nThe second term in the error is the irreducible error after we find a best fit for relating the optimal joint Q values and the concatenation of independent Q values. How do we go from this line to line (5)? On a related note, how are we going from line (5) to (6)? \n\n“The first term on the RHS of (5)...” I don’t see how this difference is related to the contribution based role which is based on an agent’s Q values. How can this be related to the linear weights? \n\n“The first term on the RHS of (6)...” Minimizing the bias results in minimizing the policy-based metric, which intuitively leads to agents _not_ aligning their actions. Does it make sense to think about this minimization as reducing the error we incur when approximating a joint policy with independent policies? This seems contradictory to what the authors want to propose in their methods section. The bias term should also normalized over the number of samples too. \n\n\n--- claims --- \n“These definitions are intuitive and cannot accurately describe the role difference.” Why not? What is insufficient with their definitions? It would be nice if these definitions could also be used in the experiments to compare and contrast the success/failure modes.\n\n“As common sense would indicate, actions taken at the same time step can indicate different roles” - Do actions do this? This is arguable. Actions can be considered coordinated or uncoordinated, joint vs. independent. They can indicate different or shared roles.\n\n--- figure ---\nFigure 1 is quite confusing for the following reasons \n> What is the blue vs. green color schema supposed to refer to? \n> I don’t understand what MARL alg 1 and 2 should represent. I thought the agents are trained using the same algorithm? \n> Related to my comment on “why the authors break down MARL cooperative algorithms by ... ”, I don’t understand why these axes are being used as input parameters for the algorithm in Fig 1a, and then as outputs from Role Diversity.\nIf the point is that role diversity should inform our assumptions in developing algorithms, then I’d recommend rephrasing the figure caption, omitting the performance graphs, and improving the figure pipeline. The figure doesn’t elucidate the authors claim that “role diversity [avoids] possible bottleneck of a MARL algorithm … ” What is this bottleneck? \n\nFigure 2 onwards refer to the scenarios with hard-to-parse names, eg. “1s1m1h1M_vs_5z”. These figure names should be updated for clarity.\n\n\n\n--- writing nits ---\nIntroduction\n> “lack the study of the question of why” - too wordy and awk. \n> “In other word” → “In ”\n> “Current researches are more focusing”\n> “The reason maybe:” - informal. \n\nRelated works\n> “adopt q-learning” - note capitalization\n> “In this study, we contended that” - note tense \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a Role Diversity metric, meant to quantify how different roles are in a multi-agent RL setting. There's actually three versions of this metric, or three aspects (the distinction is not entirely clear to this area chair).

The reviewers are generally not very enthusiastic about the paper, with scores hovering at or just below the acceptance threshold. There has been extensive discussion between reviewers and authors, but there a sense that there is confusion about the exact purpose and contribution of the paper. This is reinforced by the authors' "letter to area chair", which outlines several ways the reviewers have not gotten the message. Reading the paper, it appears to me that the root cause is that the authors are indeed not communicating clearly what the paper contributes and why. It is, after all, the authors' responsibility that the reviewers understand the work. My own impression is that the text is dense and not particularly easy to get through. Perhaps the authors are simply trying to cram too many contributions into a single conference paper? This is a classic error which leads to hard-to-read papers. In addition to this, there is a lingering concern about the generalizability of the proposed methods.

I think the authors need to work more on their presentation, and perhaps reconsider which parts to include in their paper and exactly which measure they want to send, before they submit to another venue.
Decision: Reject