Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
Authors: Chenghao Li
Tonghan Wang
Chengjie Wu
Qianchuan Zhao
Jun Yang
Chongjie Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Multi-Agent Reinforcement Learning
Diverse Behaviors
Partial Parameter Sharing

Abstract: Recently, deep multi-agent reinforcement learning (MARL) has shown the promise to solve complex cooperative tasks. Its success is partly because of parameter sharing among agents. However, such sharing may lead agents to behave similarly and limit their coordination capacity. In this paper, we aim to introduce diversity in both optimization and representation of shared multi-agent reinforcement learning. Specifically, we propose an information-theoretical regularization to maximize the mutual information between agents' identities and their trajectories, encouraging extensive exploration and diverse individualized behaviors. In representation, we incorporate agent-specific modules in the shared neural network architecture, which are regularized by L1-norm to promote learning sharing among agents while keeping necessary diversity. Empirical results show that our method achieves state-of-the-art performance on Google Research Football and super hard StarCraft II micromanagement tasks.


Celebrating Diversity in Shared Multi-Agen Reinforcement Learning
Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang', Chongjie Zhang"
Tsinghua University
(lich18, wangth18. wucj19] 19] @mailsl manilusssingua..cdu.crr
(zhaoqe, yangjun603, chongjie red..ggua.edu.cn
Abstract
Recently, deep multi-agent reinforcement learning (MARL) has shown the promise to solve complex cooperative tasks. Its success is partly because of parameter sharing among agents. However, such sharing may lead agents to behave similarly and limit their coordination capacity. In this paper, we aim to introduce diversity both optimization and representation of shared multi-agen reinforcement learning. Specifically, we propose an information iifommatio-thhooetiica regularization to maximize the mutual information between agents' identities and their trajectories, encouraging extensive exploration and diverse individualized behaviors. In representation, incorporate agent-specific modules in the shared neural network architecture, which are regularized by L1-norm to promote learning sharing among agents while keeping necessary diversity. Empirical results show that method achieves state-of-the-art performance on Google Research Football and super hard StarCraft II micromanagement tasks".
1 Introduction
Cooperative multi-agent reinforcement learning (MARL) has drawn increasing interest in recent years, which provides promise for solving many real-world challenging problems, such as sensor networks [T] traffic management [2], and coordination of robot swarms []. However, learning effective policies for such complex multi-agent systems remains challenging. One central problem that the joint costen-observation space grows exponentially with the number of agents, which imposes high demand on the scalability of learning algorithms.
To address this scalability challenge, policy decentralization with shared parameters (PDSP) is widely used, where agents share their neural network weights. Parameter sharing significantly improves learning efficiency because dramatically reduces the total number of policy parameters, while experiences and gradients of one agent can used train others. Enjoying these advantages, many advanced deep MARL approaches adopt the PDSP paradigm. including value-based methods [4] policy gradients [3133 and communication learning algorithms 15). These approaches achieve state-of-thc-as performance on tasks such as StarCraft II micromanagement [16].
While parameter sharing has been proven accelerate training 177. its drawbacks also apparent in complex tasks. These tasks typically require substantial exploration and diversified strategies among agents. When parameters are shared, agents tend acquire homogeneous behaviors because they typically adopt similar actions under similar observations, preventing efficient exploration and the emergence of sophisticated cooperative policies. This tendency becomes particularly problematic
'Equll advising
Videas are available at http:://wwaagooog....//////aaaar with
codes.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

for many challenging multi-agent coordination tasks, hindering deep MARL from broader applica- tions. For example, the unsatisfactory performance of state-of-the-ar MARL algorithms on Google Research Football (Fig. 1 and (88) highlights urgent demand for diverse behaviors.
Notably, sacrificing the merits of parameter shar- ing for diversity is also unfavorable. Like hu- mans, sharing necessary experience or under- standing of tasks can broadly accelerate coop- eration learning. Without parameter sharing, agents search in much larger parameter space, which may be wasteful because they do need to behave differently all the time. Therefore, the question is how adaptively trade-off diversity and sharing. In this paper, we solve this dilemma by proposing several structural and learning nov- elties.
(a) Parameter sharing: similar behaviors (competing for ball).
(b) Our approach: each has its responsibility to score.
Agents' history location
The goal's passing,
Off-she-bh ball
dribbling and shooting
moving
Figure 1: Shared parameters induce behaviors (left) and can hardly learn successful policies the challenging Google Research Football task. Our method learns sophisticated cooperative strate- gies by trading off diversity and sharing (right).
To encourage diversity, we propose novel information-theorrtiin objective to maximize the mutual information between agents' identi- ties and trajectories. This objective enables each agent to distinguish themselves from others and thus involves the contribution of all agents. Accordingly, we derive an intrinsic reward for motivating diversity and optimize it with the global environmental reward by learning the total Q-function as combination of individual Q-functions. Structurally, we further decompose individual Q-functions as the sum of shared and non-shared local Q-functions for sharing experiences while maintaining representation diversity. We hope agents can use and expand shared knowledge whenever possible. Thus we introduce LI regularization on each non-shared Q-function, encouraging agents to share and be diverse when necessary on several critical actions. Combining these novelties achieves a dynamic balance between diversity and homogeneity, efficiently catalyzing adaptive and sophisticated cooperation.
We benchmark our approach on Google Research Football (GRF) [8]. and StarCraft II micro- management tasks (SMAC) [16]. The extraordinary performance of our approach on challenging benchmarking tasks shows that our approach achieve significantly higher coordination capacity than baselines while using diversity as catalyst for more robust and talent policies. To our best knowledge, approach achieves state-of-the-art performance SMAC super hard maps and challenging GRF multi-agent tasks like academy_3 Saatimy___vv___ f_vs____ii teepeer academy_counteraat _ard,, and full-field scenario 3_vs_ 1_ wi with keeper (full field).
2 Background
A fully cooperative multi-agent task can be formulated Dec-POMDE [19] which is defined as tuple G = (N, S, A, P. R, O,Ω, n. 7),, where is a finite set of n agents, E S is the true state of the environment, A is the set of actions, and agent i e N receives his own observation o E Ω according to the observation function O i), and selects an action a; E A. which results in joint action vector a. The environment then transitions to a new state based on the transition function (ss'| |s, a). and inducing a global reward r = R(s, shared by all the agents. Each agent has its to partial observability, each agent conditions its policy n; (a,, (T.) on T;. The joint policy æ induces the joint action-value function Q (s, a) E Exp... .0:oc
[0, 1) is discount factor. At cach time step, each
observation history т; E T, = (Ω, A)" Due
1 7 80 = 8, ag a, π..
2.1
Centralized Training with Decentralized Execution
Our method adopts the framework of centralized training with decentralized execution (CTDE) 20 [ 21 22 II]. This framework tackles the exponentially growing joint action space by decentralizing the control policies while adopting centralized training to learn cooperation. Agents learn in centralized manner with access to global information but execute based on their local action-observation history. One promising approach to implement the CTDE framework is value function factorization. The IGM findividual-goobal-o
principle 21 guarantees the consistency between the local and global greedy actions. When IGM is satisfied, agents can obtain the optimal

---

global action by simply choosing the local greedy action that maximizes each agent's individual utility function Q, Some algorithms have successfully used the IGM principle [5] 6 23 to push forward the progress of MARL.
3
Method
In this section, we present a novel diversity-driven MARL framework (Fig- 2 that balances each agent's individuality with group coordination, which is general approach that can be combined with existing CDTE value factorization methods.
3.1 Identity-Aware Diversity
We first introduce how encourage behavioral diversity by designing in- trinsic motivations. Intuitively, to
TD Loss
L1 Loss
Agent
encourage the specialty of individ- ual trajectories, agents need to be- have differently to highlight them- selves from others, taking different actions and visiting different local ob- servations. To achieve this goal, we use an information-theoreti objective for maximizing the mutual informa- tion between individual trajectory and agents' identity:
Quoe
Mixing Network
@
F)
Q
Diversity Loss
Agent
L1 Lass
[(ccid(t)
1 (d)
Figure 2: Schematics of our approach.
p(rr idd
I" (TT: id) H ((T) H(rrlid) = Ead, log
p(rr)
where Ty and id is the random variable for agent's local trajectory and identity, respectively. ı the joint policy. To optimize Eq. we expand p(Tr) as p(oo) IT p(axlTe)p(og 10411/7, at), and p(rr jid)
as (d) II p(a, id)(or4 id). Therefore, the mutual information can be written as:
p(oolid) p(op)
pluelTe, id) log p(a, TT)
plo (T, dze (t,id) p(octilTe, at) (2)
1a (m(rridd) Eid, log
log
Term ① is determined by the environment, and we can ignore it when optimizing the mutual information. The second term quantifies the information gain about agent's action selection when the identity is given, which measures action-aware diversity as [(a; id]r). However, p(a,) (/a:/7 Te, id) is typically the distribution induced by e-greedy, which only distinguishes the action with the highest possibility. Therefore, directly optimizing this term conceals most information about the local Q- functions. To solve this problem, we use the Boltzmann softmax distribution of local values to replace p(a: Fr. id), which forms lower bound of term @:
Eid,r log p(aelTe,dd) id) p(ac(te)
2 Exd, log S Q id)) p(ar(Te)
(3)
The inequity holds because the KL divergence DKL (p(-I Im, [[of]||[oaaaa 1 Q+IT, id))) is non- negative. We maximize this lower bound optimize Term @. Inspired by variational inference approaches (24), we derive and optimize tractable lower bound for Term 3 at each timestep by introducing variational posterior estimator 9s parameterized by Φ:
Esa, log p(o++1 Te, at id)
To (o 01, id)
2 Eid,r
log
(4)
p(o1+1lT,,
p(o142/T6, at)
Similar to the second term, the inequality holds because for any do. the KI. divergence DKL (p(- a, ae, id)) is non-negative. Intuitively, optimizing Eq. encourages agents

---

to have diverse observations that are distinguishable by agents' identification and thus measures observation-aware diversity as [o'; id|T, a). To tighten the this lower bound, we minimize the KL divergence with respect to the parameters ø. The gradient for updating @ is:
log p id) do (0 a, id)
VoL(Ø) V&,rr i [DEL (p (-I7, a, id) Ila. (-IT, a, id)) Valler,
-dd ( log do (o'|T, a, id)]
(5) Based on the lower bounds shown in Eq. and Eq we introduce intrinsic rewards optimise the information coommatio--thooetic objective (Eq. T for encouraging diverse behaviors:
r Edd
(6)
+31 log 9 Te, ar, id) - logp(o+11/7.
We introduce two scaling factors B1, B2 2 0 when calculating intrinsic rewards. When (1 is 0,
only optimize the entropy term H(rr) in the mutual information objective (Eq. [ 83 is used to adjust the importance of policy diversity compared with transition diversity. In Appendix A, discuss and compare two different approaches for estimating p (a,lT) and (0.+1/T,,,))
3.2 Action-Value Learning for Balancing Diversity and Sharing
In the previous section, we introduce an informationntheoreff objective for encouraging each agent to behave differently from general trajectories. However, the shared local Q-function does not have enough capacity to present different policies for each agent. For solving this problem, we additionally equip each agent with an individual local Q-function Q Defining experiences that need to be shared or exclusively learned is inefficient and usually can not generalize. Therefore, we let agents adaptively decide whether to share experiences by decomposing Q; as:
Qa(a:TT:) (a.lt)) (a,lT))
where Q$ is the shared Q-function among agents. In its current form, agents may learn decompose their local Q-function arbitrarily. On the contrary. we expect that agents can share as much knowledge as possible so that we apply LI regularization on individual local Q-function Q' shown Fig 2 Such regularization can also prevent agents from being too diverse and ignore cooperating to finish the task. In our experiments, we show that the L.1 regularization is critical to achieving a balance between diversity and cooperation.
3.3 Overall Learning Objective
In this section, discuss how use the diversity- encouraging reward train the proposed learning framework. Since the intrinsic rewards /' inevitably involves the influence from all agents, we add r environment rewards rf and use the following TD loss:
(T) = [ice + 3rd max Qiot (6,,''00) Qto a; 0)))
where 0 is the parameters in the whole framework, 0 is periodically frozen parameters copied from 0 for a stable update, and ,9 is a hyper-parameter adjusting the weight intrinsic rewards compared with environment rewards. We use QPLEX decompose Qtot mixing of local Q-functions Q; and train the framework end-to-end by minimizing the loss:
(()) = (0) + Σ Lr, (O))),
where 0 the parameters of Q/ Cr, (Q,) is the LI regularization term for independent Q-functions, and À is scaling factor.
4 Case study: outperforming by being diverse only when necessary
We design Pac-Men shown in Fig. 3 to demonstrate how our approach works. this task, four agents initialized at the center room and can only observe 5 x 5 grid around them. Three dots

---

are initialized randomly in each edge room. To make this environment more challenging, paths to different rooms have different lengths, which are down left up right = : 12 Three out
of four paths are outside agents' observation scope, which brings about the difficulty of exploration. Dots will refresh randomly after rooms are empty. An ineffective competition between agents occurs when they come together in one room. The total environmental reward is the number of dots eaten in one step or -0.1 if no one eats dots. The time limit of this environment set to 100 steps.
Pac-Men
10
10
100
CDS(ours)
D
80
QPLEX
AT
A2
Enter di different rooms
Af
A2
QMIX
Observation
10
10
based on independent
Episade Relum
60
MAVEN
S
Q Q-nelworks
EOI
40
A4
A4
20
0-
Search dots based the shared Q-nelwork
(mil)
(a) Comparison against baselines
(b) Visitation heatmap
(c) Behavior diversity heatmap
Figure 3: Why does our method work? The balance between identity aware diversity and experience sharing encourages sophisticated strategies.
Fig. middle demonstrates the learned strategies of our approach, with a heatmap showing the visitation number. Driven by the objective of mutual information between individual trajectory and identity, agents achieve diversity and scatter different rooms to eat dots. further analyze the role of independent and shared Q-functions during different stages in Fig. right. We visualize the value of (D() ())/SD ( (.)), where SD denotes the standard deviation (SD) of Q values for different actions. A higher SD ratio indicates the independent Q Q-functions play leading role, while lower SD ratio indicates the shared Q function's domination.
We notice that the SD ratio is considerably larger in the central room and four paths than in four edge rooms. This observation means that agents use independent Q networks to reach different rooms while use the shared Q network to search for dots in them. The result shows that method achieves a good balance between diversity and knowledge sharing. Taking this advantage, our approach outperforms baselines (Fig, 3 left, baselines are introduced in Sec. 6). Other methods, such as variational exploration (MAVEN and individuality emergence (EOI (26)), are slower to learn optimal strategies.
5 Related Work
Deep multi-agent reinforcement learning algorithms have witnessed significant advances in recent years. COMA 200) MADDPG (9), PR2 (27), and DOP [10) study the problem of policy- based multi-agent reinforcement learning. They use (decomposed) centralized critic to calculate gradients for decentralized actors. Value-based algorithms decompose the joint value function into individual utility functions in order to enable efficient optimization and decentralized execution. VDN 4 QMIX (5). and QTRAN (21) progressively expand the representation capabilities of the mixing network. QPLEX (6) implements the full IGM class 21 by encoding the IGM principle into duplex ducling network architecture. Weighted QMIX 23 proposes weighted projection to decompose any joint action- value functions. There are other works that investigate into MARL from the perspective
of coordination graphs (2430), communication 3 32 [5], and role-based learning 17 33
Knowledge sharing in MARL From IQL
QPLEX, many works focus on designing mixing network structures and have provided promising empirical and theoretical results. For these works, experience sharing among agents has been an important component. Learning from others one essential skill engraved in humans' genes to survive in society. Based on the relationship between teachers and students human society, series of research work hopes each agent can learn from others or selectively share its knowledge with others (35-37]. But it is challenging specify
5

---

knowledge in practice, let alone deciding what to share or learn. SEAC [38 partially solves this problem by sharing trajectories only for off-policy training. NCC [2] maintains cognition consistency by representation alignment between neighbors. Roy et al. 39 force each agent to predict others' local policies and adds a coach for group experience alignment. Christianos et al. 40 group agents during pre-training and force agents in the same group to use one policy. In this paper, we do not try to let agents choose whether to learn or share experiences. Our neural network structure shown in Fig. 2 can balance group coordination and diversity by gradient backpropagation.
Diversity In single-agen aggett settings, diversity emerges for exploration or solving sparse reward prob- lems. Existing methods such as curiosity- algorithms [41 44 or maximising mutual infor- mation 44 have shown great promise. When encouraging diversity in MARL settings, agents' coordination must be considered. Several recent works study this problem, such as MAVEN [25). EITT EDTI 488 and EOI 26 MAVEN learns a diverse ensemble of monotonic approximations with the help of latent space to explore. EITI and EDTI consider pairwise mutual influence to encourage the interdependence between agents. EOI combines the gradient from the intrinsic value function (IVF) and the total Q-function to train each agent's 's local Q-function. In this paper, encourage agents to explore unique trajectories by optimizing the mutual information between agent's identity and trajectory. Moreover, we propose a novel network structure to enable experience sharing or consensus, which combines all agents' rare ideas, while still maintain independent action- veauue functions for each agent to behave differently when necessary. Our approach considers the rrfde-off relationship between knowledge sharing and diversity, and learns to establish a balance and leverage their advantages for joint task solving.
6 Experiments
In Sec. 4 we use a toy game to illustrate how our approach adaptively balances experience sharing and identity-aware diversity. In this section, we use challenging tasks from GRF and SMAC benchmark to further demonstrate and illustrate the outperformance of our approach. We compare approach against multi-agent value-based methods (QMIX [5) QPLEX (), variational exploration (MAVEN (25).. ) and individuality emergence (EOI (6) methods. Different from baselines, do not include agents' identification in inputs when calculating local Q-functions. We show the average and variance of the performance for our method, baselines, and ablations tested with five random
seeds.
6.1
Performance on Google Research Football (GRF)
CDS(ours) academy vs with keeper
QPLEX academy counteratlack hard
QMIX
MAVEN
1 with keeper (full field)
40
60
40
30
Wes %
30
40
20
20
20
10
0-
0
2
4
5
15
(mil)
T (mil)
T (mil)
Figure 4: Comparison of our approach against baseline algorithms on Google Research Football.
We first benchmark our approach on three challenging Google Research Football (GRF) offensive sce- narios academy_3, ______________________________ 1__1___ 1_with_____ keeper,
caalleey_counteratte baartacckhhrdd,,,. and our own designed full-field scenario 3 vs 1 with keeper (full field). Agents' initial locations for each scenario are shown in Appendix B.3. In GRF tasks, agents need to coordinate timing and positions for orga- nizing offense to seize fleeting opportunities, and only scoring leads to rewards. In our experiments, we control left-side players (in yellow) except the goalkeeper. The right-side players are rule-based bots controlled by the game engine. Agents have a discrete action space of 19, including moving in eight directions, sliding, shooting, and passing. The observation contains the positions and moving directions of the ego-agent, other agents, and the ball. The z-coordinate of the ball is also included.
vs_

---

We make small and reasonable change to the half-court offensive scenarios: our players will lose if they or the ball returns to our half-court, All baselines and ablations are tested with this modification. Environmental reward only occurs at the end of the game. They will get +00 if they win, else get -1,
We show the performance comparison against baselines in Fig. 4 Our approach outperforms all the scenarios. MAVEN needs more time to explore sophisticated strategies, demonstrating that CDS incentives more efficient exploration. EOI lets each agent consider individuality and cooperation simultancously by setting local learning objectives but without exclusive Q networks, making cooperation and individuality hard to be persistently coordinated. In comparison, taking advantage of the partially shared network structure, CDS agents learn diverse but coordinated strategies. For example, as shown in Fig. [ three agents have different behaviors, with the first agent passing the ball, the second scoring, while the third running to threaten. These diverse behaviors closely coordinate, forming perfect scoring strategy and leading to significant outperformance against EOI.
6.2 Performance on StarCraft II
CDS
QPLEX
QMIX
MAVEN
EOI
Corridor
MMM2
6h vs 8z
90
90
90
Test Win %
60
60
30
30
30
0
0
0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
T (mil)
T (mil)
T (mil)
3s5z 3s6z
5m 6m
3s 5z
90
90
90-
Test Win %
60
60-
60
30
30
30
0
0
0
0
2
3
4
5
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
(mil)
(mil)
(mil)
Figure 5: Comparison of our approach against baseline algorithms on four super hard SMAC maps: corridor, MMM2, 6h_vs_ sa__v__eet vs_8z, and 3s5z_vs_aaa vs_ vi__sss__ and two hard SMAC maps: 5m_vs 6m and 3s_vs_5z.
In this section, we test our approach on the StarCraft II micromanagement (SMAC) benchmarks [6] This benchmark consists of various maps classified as easy, hard, and super hard. Here we test our method on four super hard maps: corridor, MMM2, 6h_vs_BB________ vs_8z, and 3s5z_vs vs ps__szz, and two hard SMAC maps: 5m_vs 6m and 3s ss__s__5_ For the four super hard maps, approach outperforms all baselines with acceptable variance across random seeds, as shown in Fig. 5 The baselines QPLEX and QMIX can achieve satisfactory performance as 3s5z_vs vs ___666622222 and MMM2. But on other maps, they need the proposed method to get better performance. Compared with MAVEN and EOI, our approach maintains its out perfrorforeence with the balance between diversity and homogeneity for learning sophisticated cooperation. Our approach performs similarly with baselines for the two hard maps, indicating our balancing process may not improve the learning efficiency in environments that require pure homogeneity. But for challenging environments, where sophisticated strategies laborious to explore, our approach can efficiently search for valuable strategies with stable updates.
some challenging benchmarks, such tneersity-celebratii
6.3
A Allaionn and Visualization
To understand the contribution of each component in the proposed CDS framework, we carry out ablation studies to test the contribution of its three main components: Identity-aware diversity (A) encouragement and partially shared (B) neural network structure with L. regularization (C) on
non-shared Q-functions. To test component A, we ablate our intrinsic rewards to four different levels. (L) -Dw-Raw ablates all intrinsic rewards by setting / in Eq. 8 to zero. (2) CDS-No-Identity ablates
7

---

H (Tr (Trlid) and only optimize H (Tr) in Eq. by setting 31 in Eq. to zero. (3) CDS-No-Action ablates item 2 Eq. by setting 32 in Eq. to zero. (4) CDS EDS-No-OU Obs ablates item 3 in Eq. 2 by ablating 31 log 45 (0t+1 dz. id) - log p (0++1/ in Eq. 6 To test component B, we design CDS-All-Shared, which ablates independent action-value functions together with the I.1 loss and, like baselines, adds agents' identification to the input. To test component C, we design CDS-No-LI,
which ablates LI regularization terms by setting À in Eq. to zero.
scedemy counteratiack hard
40
CDS(ours)
CDS-Raw
Teat %
30
CDS No-Idenlily
CDS AoS-io-Aclion
20
CDS-No-Obs
10
CDS-No-LT
CDS-AU-Shgred
0
2
Agents' history
The goal's dribhling, passing. crossing and shooting route
Off-the-ball moving
(mil
location
Figure 6: Left. Ablation studies on academy counterattack hard. Right. Visualization of trained policies, which achieve complex cooperation with impressive off-the-ball moving strategies.
We first carry out ablation studies on academy_courre wwwnterattackk hard to analyze which part of our novelties lead to the outstanding performance as shown on the left side of Fig. 6 The ablation of each part our intrinsic reward will bring a noticeable decrease performance. Among them, the least impact on performance is the ablation of action-aware diversity. CDS-No-LI performs similarly to MAVEN, which indicates that unlimited diversity is harmful to cooperation. CDS-All-Sharee performs even worse than QPLEX, demonstrating that identity-awar diversity is difficult to emerge without our specially designed network structure.
We further visualize the final trained strategies on the right side of Fig. 6 which shows complex cooperation between agents. Our players first attack down the wing by dribbling and passing the ball. Then one of them draws the attention of the enemy defenders and the goalkeeper, while the ball being passed across the penalty area. Another player catches the ball and completes the shot. The most impressive part of our sophisticated strategies is off-the-ball moving strategies. All agents without the ball try to use their unique and valuable moves to create more scoring opportunities, which shows behavior and position diversity for finishing the goal.
Corridor
80
CDS(ours)
CDS-Raw
60
CDS-No-Ideriity
CDS-No-Action
40
CDS-No-Obs
CDS-No-L1
20
CDS-AI-Sharod
0
0.0
0.5
1.0
1.5
2.0
Different offensive routes
Attracted enemies
(mil
Figure 7: Left. Ablation studies in super hard map corridor. Right. Visualization of the final trained strategies, which achieves a hard-earned victory brought by the sacrifice of a warrior.
We also carry out ablation studies on the super hard map corridor as shown in Fig. leff. Same as results academy_counteree counterattack hard, the ablation of action-aware diversity causes the least performance gap. Among all the ablations, CDS-No-LE and performance is similar to QPLEX. This phenomenon indicates excessive diversity is harmful to the emerge of complex cooperation. CDS-AII- All-Shared achieves acceptable performance, different from the GRF scenario, reflecting the different demand levels for the representation diversity of these two kinds of benchmarks.
perform worst, whose
To better explain why our approach performs well. On corridor, we also visualize the final strategies in Fig- 7 right. In this super hard map, six friendly Zealots are facing 24 enemy Zerglings. The disparity in quantity means our agents are doomed to lose if they attack together. One Zealot, whose
8

---

route is highlighted blue, becomes a warrior leaving the team to attract the attention of most enemies in the blue oval. Although doomed to sacrifice, he brings enough time for the team to eliminate small part of the enemies in the green oval. After that, another Zealot stands out to attract some enemies and enables teammates to eradicate them. These sophisticated strategies reflect the leverage between diversity and homogeneity by encouraging agents to be diverse only when necessary.
7 Closing Remarks
Observing that behavioral diversity among agents is essential for many challenging and complex multi-agent tasks, in this paper, we introduce a novel mechanism of being diverse when necessary into shared multi-agent reinforcement learning. The balance between individual diversity and group coordination induced by our CDS approach pushes forward state-of-the-art of deep MARL on challenging benchmark tasks while keeping parameter sharing benefits. We hope that method can shed light on future works to motivate agents to cooperate with diversity to further explore complex multi-agent coordination problems.
Acknowledgments
This work was funded by the National Key Research and Development Project of China under Grant 2017YFC0704100 and 2016YFB0901 900, in part by the National Natural Science Foundation of China under Grant 61425027 and U1813216, in part by Science and Technology Innovation 2030- "New Generation Artificial Intelligence" Major Project (No. A010000))))) grant from the Institute of Guo Q Tsinghua University, and a grant from Turing AI Institute of Nanjing.
References
[1] Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in net- worked distributed pomdps. In Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.
[2] Arambam James Singh, Akshat Kumar, and Hoong Chuin Lau. Hierarchical multiagent rein- forcement learning for maritime traffic management. In Proceedings of the 19th International Conference Autonomous Agents and MultiAgent Systems, pages 1278- 2020.
[3] Maximilian Hütteerauch, Adrian Šošić, and Gerhard Neumann. Guided deep reinforcement 2017.
learning for swarm systems. arXiv preprint
[4] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo. Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Pro- ceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, pages 2085-2087. International Foundation for Autonomous Agents and Multiagent Systems, 2018.
[5] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonie value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4292-4301, 2018.
[6] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang, Qplex: Duplex dueling multi-agent g-larnning International Conference on Learning Representations (ICLR), 2021.
[7] Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. arXiv preprint arXiv: 200::206.000 034003400, 2021.
[8] Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference Machine Learning, pages 4596-4606. PMLR, 2021.

---

[9] Ryan Lowe, Yi Wu, Aviv Tamar. Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competttve environments. In Advances in Neural Information Processing Systems, pages 6379-6390, 2017.
[10] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off-policy multi-agent decomposed policy gradients. In Proceedingg of the International Conference on Learning Representations (ICLR), 2021.
[11] Xiaoteng Ma, Yiqin Yang, Chenghao Li, Yiwen Lu, Qianchuan Zhao, and Jun Yang. Modeling the interaction between agents in cooperative multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 853 861, 2021.
[12] Kamal K Ndousse, Douglas Eck, Sergey Levine, and Natasha Jaques. Emergent social learning via multi-agent reinforcement learning. International Conference on Machine Learning. pages 7991-8004. PMLR, 2021.
[13] Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entropp multi-agent reinforcement learning. In International Conference on Machine Learning, pages 12491-12500. PMLR, 2021.
[14] Jakob Foerster, loannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pages 21 2137-2145, 2016.
[15] Tonghan Wang, Jianhao Wang. Chongyi Zheng. and Chongjie Zhang. Learning nearly decom- posable value functions with communication minimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[16] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nan- tas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: arXiv:1902.044 1902.04043. 2019.
[17] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforce- ment learning with emergent roles. In Proceedings of the 37th International Conference Machine Learning, 2020.
[18] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. Iaeennnnrrrrreeeeee Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4501-4510, 2020.
[19] Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
[20] Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[21] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896, 2019.
[22] Kyunghwan Son, Sungsoo Ahn, Roben Delos Reyes, Jinwoo Shin, and Yung Yi. Qtran++: Improved value transformation for cooperative multi-agent reinforcement learning, 2020.
[23] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
[24] Martin J Wainwright, Michael Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends@ in Machine Learning, 1(1-2):1-305, 2008.
10

---

[25] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi- agent variational exploration. In Advances in Neural Information Processing Systems, pages 7-11-7 2019.
[26] Jiechuan Jiang and Zongqing Lu. The emergence of individuality in multi-agen reinforcement learning. arXiv preprint arXiv:2006. 05842, 2020.
[27] Ying Wen, Yaodong Yang, Rui Luo, Jun Wang. and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[28] Carlos Guestrin, Michail Lagoudakis, and Ronald Parr. Coordinated reinforcement learning. In ICML., volume 2, pages 227-234. Citeseer, 2002.
[29] Carlos Guestrin, Daphne Koller, and Ronald Parr. Multiagent planning with factored mdps. In Advances in neural information processing systems, pages 1523-1530, 2002.
[30] Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In Proceed- ings of the 37th International Conference on Machine Learning, 2020.
[31] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[32] Hangyu Mao, Wulong Liu, Jianye Hao, Jun Luo, Dong Li, Zhengchao Zhang, Jun Wang, and Zhen Xiao. Neighborhood cognition consistent multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7219-7226, 2020.
[33] Tonghan Wang. Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. In Proceedings of the International Conference Learning Representations (ICLR), 2021.
[34] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceed- ings of the tenth international conference on machine learning, pages 330-337, 1993.
[35] Changxi Zhu, Ho-fung Leung, Shuyue Hu, and Yi Cai. A v-values sharing framework for multiple independent g-learners. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages
2019.
[36] Yongyuan Liang and Bangwei Li. Parallel knowledge transfer in multi-agent reinforcement learning. arXiv preprint urXiv:2003.13055 2020.
[37] Yonggan Fu, Zhongzhi Yu, Yongan Zhang, and Yingyan Lin. autt-agent-distiller: Towards efficient deep reinforcement learning agents via neural architecture search. arXiv preprint UrXiv:2012.13091, 2020.
[38] Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
[39] Julien Roy, Paul Barde, Félix Harvey, Derek Nowrouzezahrai, and Chris Pal. Promoting coordination through policy regularization in multi- agent deep reinforcement learning, Advances in Neural Information Processing Systems, 33, 2020.
[40] Filippos Christianos, Georgios Papoudakis, Arrasy Rahman, and Stefano V Albrecht. Scal- ing multi reinforcement learning with selective parameter sharing. arXiv preprint arXiv:2102.07475, 2021.
[41] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. -uriosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pages 2778- 2787, 2017.
[42] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint
2018.

---

[43] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi. Daniel Guo, Bilal Piot, Steven Kapturowski. Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andew Bolt, al. Never give up: Learning directed exploration strategies. arXiv preprint (rxXi:2002.06088 2020.
[44] Adria Puigdomênech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvit- skyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507-517. PMLR, 2020.
[45] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Representations, 2018.
[46] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. In International Conference on Learning Representa- tions, 2020.
[47] Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills, In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
[48] Tonghan Wang, Jianhao Wang. Wu Yi, and Chongjic Zhang. Influence-based multi-agent exploration. In Proceedings of the International Conference Learning Representations (ICLR), 2020.
12


Reviews:
[{"review_id": "uVdqPh19VVy", "reviewer": "Reviewer_u9nV", "summary": "Incentivizes agents to be diverse from each other in cooperative multi-agent tasks by getting them to maximize the mutual information between their trajectories and their ID. Breaks this down into action-diversity (maximizing the KL divergence between each agent’s action distribution & the group average), as well as observation-diversity (using a variational approximation of I(o_t;ID|tau,a_t)). Also propose a new architecture that allows sharing of information between agents, while retaining some agent-specific parameters. Thorough evaluation with 6 state-of-the-art multi-agent baselines show the method gives consistently much higher performance than prior techniques at both Google Research Football and Starcraft. Great analysis of the learned behaviors and why diversity helps in both environments, cool ablation studies.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Quality:**\n- The paper proposes a well-motivated and effective technique for improving cooperation and learning in multi-agent RL (MARL). It combines a (somewhat) novel diversity objective with a novel architecture that enables sharing information between agents while allowing for agent-specific policies. \n-  The experimental evaluation is extremely thorough, and includes 6 baseline methods spanning state-of-the-art multi-agent techniques. The performance gains over these methods are convincing, across 2 challenging MARL environments.\n- The paper also provides experiments in a 3rd environment which provide insight into the method.\n- Thorough ablation studies analyze exactly which components of the algorithm contribute to performance. Interestingly, this differs between the multi-agent environments under consideration. \n\n**Significance:**\n- Given the magnitude of the performance between this technique and several SOTA baselines, this paper is definitely of interest to the MARL community\n- The progress on GRF and the analysis of how diversity contributes to this is also compelling. \n\n**Originality/Novelty:**\n- The biggest issue with the paper is novelty. Similar diversity objectives have been proposed in DIAYN (https://arxiv.org/abs/1802.06070), and in the multi-agent case with EOI (https://arxiv.org/pdf/2006.05842.pdf) and CMRL (https://arxiv.org/abs/1903.02710). However, the paper benchmarks against EOI and shows convincing performance improvements above this method. DIAYN is appropriately cited. \n\n**Clarity:**\n- The paper is well motivated, and the introduction is clear and compelling. \n- Intro figure clearly sets up the issue with learning shared policies and why diversity is important. \n- The derivation of the method in section 3.1 is very clear and crisp, especially highlighting each of the three terms in equation 2. \n- However, it would improve clarity to briefly describe (even in 1-2 sentences) how p(a_t|\\tau_t) and p(o_{t+1}|\\tau_t,a_t) are estimated in the main text. \n- The L1 regularization could be more clearly motivated. Why L1 not L2? \n- Figure 3 would be much more compelling if it included a heatmap for one of the baseline methods (as in the EOI paper). This helps make the case for the importance of explicitly incentivizing diversity, rather than just optimizing shared reward. \n- Lines 255-260 talk about how during phase 2 of learning in GRF, the agents \"lose their individuality\" to focus on scoring. And yet even in this phase, the performance of CDS is significantly higher than the baselines (~25% vs~10%). If the agents have lost their individuality, what explains this performance gap?\n\n**Detailed feedback/suggestions:**\n- Lines 54-57 in the intro could be made more clear/crisp. \n- Although the results of the paper are convincing, calling your own results \"extraordinary\" (line 62) might be considered a bit grandiose.\n- \"agents need to behave differently to /highlight/ themselves from others\" -> not proper grammar. \"distinguish themselves from others\" would be more appropriate \n- When referring to QPLEX in line 153 you should include the citation of the QPLEX paper.\n- The idea of rewarding agents for mutual influence was first proposed in https://arxiv.org/abs/1810.08647. It would be appropriate to cite this work in lines 208-209 of the related work when introducing EITI and EDTI.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The introduction clearly identifies real-world problems for which this technique could provide social benefit, but the paper does not include a discussion of limitations and potential negative impact in the main text. It might be worth mentioning the limitation that because the method relies on centralized training, it is not applicable to multi-agent problems in which some of the other agents are humans, and thus are outside of the control of the learning algorithm. E.g. it is not applicable to autonomous driving. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "ib2wtoQoTHH", "reviewer": "Reviewer_zJ1w", "summary": "This paper proposes a CDS method that adaptively trades off diversity and sharing in multi-agent reinforcement learning. The authors introduce an information-theoretical objective to encourage diversity and decompose individual Q-functions as the sum of shared and non-shared local Q-functions to boost shared knowledge usage whenever possible. Visualizations show that the balance between identity-aware diversity and homogeneity promotes sophisticated strategies.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: It is novel and interesting to introduce diversity into the common-used parameter sharing technique. The method is new and differs from previous works in this area. The related works are adequately cited and reasonably compared.\n\nQuality: I have not checked all details, but the paper appears to be technically sound. The experimental results illustrate the method’s superiority to some extent, but some visualizations may not support some of the ideas in the paper very well.\n\nClarity: This paper is clearly written and well organized. It provides the reader with some essential information to reproduce its results.\n\nSignificance: This submission introduces diversity into the parameter sharing technique, and to some extent, advances the field of multi-agent systems. Other researchers are likely to use the ideas in this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not address the limitations of their work, but I have some questions and suggestions about this paper: \n\n1. The essence of exploration should be for the team as a whole to interact thoroughly with the environment, so why should the agents within a team have to behave differently? How does the algorithm perform in scenarios that require multiple agents to work together on a single task (for instance, a door that requires two agents to open together)? \n\n2. Figure 3 is attractive and illustrates some of the intrinsic mechanics and advancements of the method, but I did not understand the role of independent Q very well with this example. Is the passage of agents through different paths a reflection of different strategies? Passing diverse paths can also be summarized macroscopically as the same strategy. I suggest designing more complex scenarios, such as some agents passing through paths while others need to stay in the central room to accomplish the task together. These may be more like manifestations of different strategies. \n\n3. Why were the agents' identifications not included when calculating the local Q-functions in the experiments? \n\n4. Figure 5 shows why the CDS method works, illustrating the process of training in which the agents first personalize, then assimilate, and finally find a balance between diversity and homogeneity. This visualization, although interesting, does not seem to be a convincing illustration of this method. Because this process can also be explained by the fact that the agents first focus on self-exploration and then learn to cooperate, which is the same with the vast majority of MARL approaches. An ablation experiment is a more proper way to prove this. I suggest removing modules that encourage individualization and assimilation, respectively, and visualize how these cases perform at the same number of training steps. These experiments can show that the CDS is effective because of its balance between diversity and homogeneity.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns with this paper. ", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "XSNFLc64lNV", "reviewer": "Reviewer_yUaF", "summary": "This paper focuses on the centralized training with decentralized execution (CTDE) setting within cooperative multi agent reinforcement learning (MARL). The authors observe that in some cases training with a joint Q function can lead to myopic agents that do not effectively coordinate with one another. The authors introduce CDS, a new method that factorizes the Q function into shared and individual components, with an additional loss term maximizing the mutual information between the individual Q-function and the agent ID. They demonstrate the effectiveness of the approach on a toy problem and then outperform a set of strong baselines on two challenging benchmarks. Overall the work seems like a solid contribution to an active area of research. \n\nI vote to weak accept the paper, with a possible increase if my questions below are answered in a satisfactory manner. The only reason why this paper is not an accept is a general impression that the authors prioritized SoTA over transparency. There is no discussion of how hyperparameters were tuned, very limited discussion of limitations and no discussion of how environments were chosen. Overall it leaves the impression that maybe the method doesn't generalize to other environments, and is brittle to hyperparameters.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper:\n\n- It is well-written and motivated. The ideas fit well with the current literature.\n\n- The idea of optimizing mutual information between agent ID and the trajectory is intuitive, and the breakdown into action and observation diversity is well done.\n\n- The toy example is well presented and explained. In particular, the result that the agents follow their local Q-functions along the paths then use the shared one for the rooms is interesting and certainly demonstrates the method effectively.\n\n- The experimental results are strong, outperforming recent (and relevant) baselines, and on both GRF and SMAC (vs. usually just SMAC for related works).\n\nWeaknesses/Questions: \n\n1) How many seeds were used for the pacman task? What does the plot actually show? Mean/median? What are the error bars?\n\n2) The paper introduces four new hyperparameters, all of which were tuned for different environments. There are no ablations showing the sensitivity of the performance to these hyperparameters. How were they chosen? How brittle is the algorithm to these changes? \n\n3) It seems for SMAC the parameters were individually tuned per task, was there not a single parameter configuration that could perform well on all three tasks? This is a separate question from above because it also highlights another issue - how does CDS perform on other tasks? The authors mention that it is not expensive to run their algorithm, yet only use four StarCraft maps, which is fewer than other works. Then for GRF, one of the three environments chosen was hand designed, presumably to favor this specific approach. \n\n4) For GRF, how were the number of timesteps chosen? It looks to be arbitrary, and makes it appear that the other methods may outperform later on. \n\n5) The language is, at times, a little overzealous. In particular, referring to the experimental results as \"extraordinary\" (l.62). I don't think this is needed, *the results are strong as they are without sensationalizing them*.\n\n6) The limitations of the method are barely mentioned. There is one paragraph at the end of the appendix, and 50% of it is reiterating the strength of the approach. It would be useful to understand when this fails. I think this gives the general impression that the authors are trying to only show strengths, maybe through fear of reviewers criticising weaknesses. In my opinion it would actually be better for the paper to show when it does not work, so that future research can be done to improve upon it.\n\n7)  The Appendix shows the objective is actually using the mean over IDs, which the authors say themselves should not be accurate, so it is possible they tuned hyperparameters to a noisy objective and picked the best result. \n\nMinor issues and typos (did not impact score):\n\n- l.16 \"provides a promise\" → \"provides promise\"\n\n- l.63 \"achieve\" → \"achieves\"\n\n- l. 64 \"talent\" → \"talented\"\n\n- l.105 \"between individual\" → \"between an individual\"\n\n- l. 112 \"about agent's\" → \"about an agent's\"\n\n- l.335 \"into\" → \"in\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations should be discussed more adequately in the main body of the paper. Societal impact is likely minimal. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "BDXnQUTpsMe", "reviewer": "Reviewer_Rpsf", "summary": "The paper deals with the problem of imposing policy diversity along the different agents of a MARL problem. To do so, the paper proposes to introduce a novel objective to maximize the mutual information between the agents identities and trajectories. Additionally, the paper proposes a decoupled Q function architecture to represent shared and non-shared goals.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and presents both quantitative and qualitative metrics to show the performance of the proposed method. Additionally, the paper introduces an insightful ablation study to evaluate the effect of the proposed elements in the learning performance.\n\nHowever, I think there are few points that should be clarified:\n\n- In Line 120, the paper claims the intractability of the Term 3, given an integral term, but the Term 3 is not explicitly showing any integral term in Equation 2.\n\n- Equation 5, presents the gradient over a Loss function, but this loss function has not been previously introduced properly.\n\n- Even if the paper proposes a decoupled architecture between shared and non-shared Q functions, the learning is done jointly. The paper claims that L1 regularization over the independent Q function, allows the learning to be balanced between the shared and non-shared terms. Can the authors provide any theoretical guarantee of that claim or only experimental?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a novel approach to cooperative multi-agent reinforcement learning that focuses on balancing diverse behaviors with shared information. The approach incorporates several new insights (mutual information losses to encourage action and observation diversity, shared and local architecture components and a regularization term) and conceptually and empirically demonstrate how each contributes to the algorithms performance. Empirical validation is extensive and includes challenging environments, often showing dramatic performance improvements compared to baselines. Detailed ablations and analysis provide further insights into how the proposed approach works.

Initial reviews were positive, highlighting the novel insights and strong empirical results achieved in the paper. The paper was assessed as well written and largely clear. Open questions included experimental details (e.g., the procedure for hyper-parameter tuning and impact of hyperparameters on performance), positioning of the paper relative to related work (e.g., DIAYN, and the general question of how this work relates to previous approaches that encourage diversity), and conceptual questions (e.g., potential conflicts between diverse behavior and the need for coordinated exploration, the role of L1 regularization over the independent Q functions).

The authors addressed reviewer questions and suggestions in detail, providing clarifications as well as additional empirical results. As a result, several reviewers increased their scores and all indicated that their concerns and questions had been addressed. The AC agrees with this consensus. 
Decision: Accept (Poster)