Attention-Guided Contrastive Role Representations for Multi-agent Reinforcement Learning
Authors: Zican Hu
Zongzhang Zhang
Huaxiong Li
Chunlin Chen
Hongyu Ding
Zhi Wang
Venue: ICLR 2024 poster (2024)
Keywords: Multi-agent reinforcement learning
contrastive learning
attention mechanism
Primary area: reinforcement learning
TL;DR: We propose an attention-guided contrastive role representation learning framework to promote behavior heterogeneity, knowledge transfer, and skillful coordination across agents for MARL.

Abstract: Real-world multi-agent tasks usually involve dynamic team composition with the emergence of roles, which should also be a key to efficient cooperation in multi-agent reinforcement learning (MARL). Drawing inspiration from the correlation between roles and agent's behavior patterns, we propose a novel framework of **A**ttention-guided **CO**ntrastive **R**ole representation learning for **M**ARL (**ACORM**) to promote behavior heterogeneity, knowledge transfer, and skillful coordination across agents. First, we introduce mutual information maximization to formalize role representation learning, derive a contrastive learning objective, and concisely approximate the distribution of negative pairs. Second, we leverage an attention mechanism to prompt the global state to attend to learned role representations in value decomposition, implicitly guiding agent coordination in a skillful role space to yield more expressive credit assignment. Experiments on challenging StarCraft II micromanagement and Google research football tasks demonstrate the state-of-the-art performance of our method and its advantages over existing approaches. Our code is available at [https://github.com/NJU-RL/ACORM](https://github.com/NJU-RL/ACORM).

Published as a conference paper at ICLR 2024
ATTENTION-GUIDER CONTRASTIVE ROLE REPRE- SENTATIONS FOR MULTI-AGENT REINFORCEMENT LEARNING
Zican Hu', Zongzhang Zhang", Huaxiong Li', Chunlin Chen', Hongyu Ding', Zhi Wang
1 Department of Control Science and Intelligent Engineering, Nanjing University
2 School of Artificial Intelligence, Nanjing University
canhu, hongyuding @mmail..... nnju iej..e.......... edu. cn
2 zhang, huaxiongli clchen, zhiwang) enju.edu.co edu.en
ABSTRACT
Real-world multi-agent tasks usually involve dynamic team composition with the emergence of roles, which should also be a key efficient cooperation multi-agent reinforcement learning (MARL). Drawing inspiration from the cor- relation between roles and agent's ' behavior patterns, we propose novel frame- work of ceaetioon-guiedd COntrastive Role representation learning for MARL (ACORM) to promote behavior heterogeneity, knowledge transfer, and skillful coordination across agents. First, introduce mutual information maximization to formalize role representation learning, derive a contrastive learning objective, and concisely approximate the distribution of negative pairs. Second, leverage an attention mechanism to prompt the global state to attend to learned role repre- sentations in value decomposition, implicitly guiding agent coordination in skill- ful role space to yield more expressive credit assignment. Experiments on chal- lenging StarCraft II micromanagement and Google research football tasks demon- strate the state-of of-- of-the-art performance of our method and its advantages over ex- isting approaches. Our code is available https:/ggthub.com///--III
INTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) aims coordinate system of agents wards optimizing global returns (Vinyals et al.) 2019). and has witnessed significant prospects in various domains, such autonomous vehicles (Zhou al. 2020), smart grid (Chen et al. 2021a), robotics (Yu et al., 2023), and social science (Leibo et al. 2017). Training reliable control policies for coordinating such systems remains major challenge. The centralized training with decentral- ized execution (CTDE) (Foerster et al.) (2016) hybrids the merits of independent Q-learning (Foerster al., (017) and joint action learning (Sukhbaatar et al., (2016), and becomes compelling paradigm that exploits the centralized training opportunity for training fully decentralized policies (Wang et al., 2023 Subsequently, numerous popular algorithms are proposed, including VDN (Sunehag et al,
2018
QMIX (Rashid et al. 2020), MAAC (Iqbal & Sha 2019), and MAPPO (Yu et al. 2022).
Sharing policy parameters is crucial for scaling these algorithms to massive agents with acceler- ated cooperation learning (Fu et al. (022). However, it is widely observed that agents tend to acquire homogeneous behaviors, which might hinder diversified exploration and sophisticated co- ordination (Christianos al., 202T). Some methods (Li al. 2021 Jiang & Lu 2021 Liu al.) 2023) attempt to promote individualized behaviors by distinguishing each agent from the others, while they often neglect the prospect of effective team composition with implicit task allocation. Real-world multi-agent tasks usually involve dynamic team composition with the emergence of
roles (Shao et al. 2022 Hu et al. 2022 Early works introduce the role concept into multi-agent
*Correspondence Zhi Wang <zhiwang wwraaann@njjjnuu..eeee
'Taking the football game (Kurach et al. 2020) as an example, the midfielders are primarily responsible for delivering the hall to the forwards to coordinate shots goal in the offensive phase, while they need to drop back and join the defenders to block passing lanes the defensive.

---

Published as a conference paper at ICLR 2024
systems (Dastani al. 2003 Sims al. 2008 Lhaksmana et al. 2018), while they usually require
prior domain knowledge to pre-define role responsibilities. Recently. ROMA (Wang et al. (020)
learns emergent roles conditioned solely on current observations, and RODE (Wang al.
2021
associates each role with a fixed subset of the joint action space. COPA (Liu et al. 2021) allows dynamic role allocation via distributing global view of team composition each agent during exe-
cution. Some works decompose the task into a set of skills (Liu et al., (222)) or subtasks (Yang et al.)
2022 Iqbal et al. 2022) with hierarchical structure for control. Overall, existing role-based meth- ods still suffer from several deficiencies, such as insufficient characterization of complex behaviors for role emergence, neglect of evolving team dynamics, or relaxation of the CTDE constraint.
To better leverage dynamic role assignment, we propose novel framework of Attention iienttio-guided COntrastive Role representation learning for MARL (ACORM). Our main insight is to learn a compact role representation that can capture complex behavior patterns of agents, and use that role representation to promote behavior heterogeneity, knowledge transfer, and skillful coordina- tion across agents. First, we formalize the learning objective as mutual information maximization between the role and its representation, to maximally reduce role uncertainty given agent's behav- iors while minimally preserving role-irrelevant information. We introduce contrastive learning method to optimize the infoNCE loss, mutual information lower bound. To concisely approximate the distribution of negative pairs, we extract agent behaviors by encoding its trajectory into latent space, and periodically partition all agents into several clusters according to their latent embeddings where points from different clusters are paired as negative. Second, during centralized training, we employ an attention mechanism to prompt the global state to attend to learned role representations in value decomposition. The attention mechanism implicitly guides agent coordination in skillful role space, thus yielding more expressive credit assignment with the emergence of roles.
ACORM is fully compatible with CTDE methods, and we realize ACORM on top of two popu- lar MARL algorithms, QMIX (Rashid et al. 2020) and MAPPO (Yu et al. 2022), benchmarked on challenging StarCraft multi-agent challenge (SMAC) (Samvelyan et al. 2019) and Google re- search football (GRF) (Kurach et al.) 2020) environments. Experiments demonstrate that ACORM achieves statc-of-the- performance on most scenarios. Visualizations of learned role representa- tions, heterogencous behavior patterns, and attentional value decomposition shed further light on advantages. Ablation studies confirm that ACORM promotes higher coordination capacity by virtue of contrastive role representation learning and inrentioo-guider credit assignment, respectively, even agents have the same innate characteristics. In summary, our contributions are threefold:
We propose a general role representation learning framework based on contrastive learning, which effectively tackles agent homogenization and facilitates efficient knowledge transfer.
We leverage role representations to realize more expressive credit assignment via attention mechanism, promoting strategical coordination in a sophisticated role space.
We build our method on top of popular QMIX and MAPPO, and conduct extensive experiments on SMAC and GRF to demonstrate our state-of-the-arr performance and advantages.
2 METHOD
In this section, we present the ACORM framework. We consider cooperative multi-agent tasks formulated as Dec-POMDP Olliehoek & Amato 2016). G = S,
O, n,n),, where I is a finite set of n agents, s E Iis the global state, and 7 E [0, 1) is the discount factor. At each time step, each agent draws observation 0; E O from (ss, and selects local action a, A. After executing the joint action a = [aI,..., an E A", the system transitions to a next state s' according P(s'|s, sss a) and receives reward r R(s, a) shared by all agents.
Our idea is to learn a compact role representation that can characterize complex behavior patterns of agents, and use the role information to facilitate individual policy learning and guide agent coordina- tion. Agents with similar roles can enjoy higher learning efficiency via more aggressive knowledge transfer, and agent heterogeneity is also guaranteed with the discrimination of diverse roles. For- mally, we propose the following definition of the role and its representation.
Taking StarCraft Il as an example, the acquired roles represent diverse strategies in a team team-based based manner, such as focusing fire, sneaking attack, and drawing fire. Agents with similar role representations learn a special- ized strategy more efficiently by more positive information sharing, and the attention mechanism is responsible for coordinating these heterogeneous behaviors more strategically with clearer role extraction in the team.
2

---

Published as a conference paper at ICLR 2024
(e,
(f
Contrastive Loss
g+
Mixing Network
MLP
Similarity
(1(o..a))
Agent
Agent n
Contrastive Representation
(0f, o
>
(of dn
Encoder
Momentumn Encoder
- GRU
ef Ei cp
of (1' E o (1'
Tinba
Dot Product
MLP
(of. af +
K Clusters
PP
Attention
Softmax
+ Scaled
(b)
Clustering
Product
2
Concatemation
ML.P
W
W
W
z Role Represcntation
Agent Embedding
(d)
Figure 1: The ACORM framework based QMIX. (a) The overall architecture. (b) The structure of shared individual Q-network. (c) The detail of contrastive role representation learning, where ž; is the query 9, and 24' /2- positive/negative keys /-/K... (d) The attention module that incorpo- rates learned role representations into the mixing network's ' input for better value decomposition.
Definition 1. Given a cooperative multi-agen task = (I....SS.PPPR....... n. Y), each agent i is associated with a role M; E M that describes behavior pattern. Each role M; quantified by role representation zς 2, which is obtained by training a complex function as = J (Pi), where P, E (O, A) the local trajectory of agent i, and I is the number of observation-action pairs. TT, O X A x Z [0, 1] is the individual policy for agent i.
ACORM consists of individual Q-networks in Fig. [b) and a mixing network in Fig. Ia). We introduce mutual information maximization formalize role representation learning, and derive contrastive learning objective that optimizes agent embeddings [ in a self-supervised way to acquire contrastive role representations {=²)r=: which is shown Fig. [c) and will be introduced in detail in Section 2.1 In value decomposition, we employ multi-head attention (MHA) to prompt the global state to attend learned role patterns, guiding skillful agent coordination in the high-level role space for facilitating expressive credit assignment, described in Fig. dd) and in Section 2.2 Appendix B presents the pscudocode, and Appendix C gives the extension to MAPPO.
2.1 CONTRASTIVE ROLE REPRESENTATIONS
Our objective is ensure that agents with similar behavior patterns exhibit closer role representa- tions, while those with notably different strategies are pushed away from each other. This stands in contrast to using one-hot ID preserve the agent's individuality, which lacks adequate discrimi- nation under the paradigm of parameter sharing. Hence, the primary issues we aim to tackle are: i) how to define a feasible metric to quantify the degree of similarity between agent's 's behaviors, and ii) how to develop an efficient method to optimize the discrimination of role representations.
Agent Embedding. To tackle the first issue, we learn an agent embedding et from each agent's trajectory to extract complex agent behaviors with contextual knowledge as e = (o q ef-1),
where & is a shared gated recurrent unit (GRU) encoder, of is the current observation, a is the last action, and e: the hidden state of the GRU. Naturally, the distance between the obtained serve the metric to measure the behavior dissimilarity between agents.
agent embeddings
Contrastive Learning, An ideally discriminative role representation should be dependent on roles associated with agent's behavior patterns, while remaining invariant across agent identities. We introduce mutual information to measure the mutual dependency between the role and its repre- sentation. Formally, mutual information aims to quantify the uncertainty reduction of one random variable when the other one is observed. To tackle the second issue, we propose to maximize the mutual information between the role and its representation, and learn role encoder that maximally
3

---

Published as a conference paper at ICLR 2024
reduces role uncertainty while minimally preserving role-irrelevant information. Mathematically., we formalize the role encoder 0 as a probabilistic encoder 2² z'~ f= where 2' denotes the role representation at time t, and e¹ = for (o', 1)) denotes the agent embedding obtained from the history trajectory. 3 Role M follows the role distribution P(M), and the distribution of agent embedding is determined by its role. The learning objective for the role encoder is:
p(M 1z)
max M) = E,, log
(1)
P(M)
In practice, directly optimizing mutual information is intractable. Inspired by the noise contrastive estimation (InfoNCE) (Oord et al. (0118) in the literature of contrastive learning (Laskin et al. 2020, Yuan & Lu 2022), we derive lower bound of Eq. (I with the following theorem.
Theorem 1. Let M denote set of roles following the role distribution P(M), and |M K. p(ele), where M E EMEMM is a given role. Let e = fo "), 2 ~ fo(zle), and h(e, z) = p(2)
E,(o',a'-1) is the agent's local trajectory following given policy. For any role M E EM, let e* denote the agent embedding generated by the role M then we have
h(e,z) (2: M) 2 log K + EM,2, log EMremhler,.
The proof of this theorem is given in Appendix A. Since we cannot evaluate p(z) or p(zle) directly, we turn to techniques of NCE and importance sampling based on comparing the target value with randomly sampled negative values. Hence, we approximate h with the exponential of score func- tion Szz,=*) =²) that a similarity metric between latent codes of two examples. We derive a sampling version of the tractable lower bound to be the role encoder's learning objective as
log exp(S(2,21)) + EMrEMM exp(S(z, ]
exp(S(2,2'))
min Ck = M...M.....M..MM........
(3)
where M is the set of training roles, e, are two instances of agent embeddings sampled from the dataset of role M,, and 2, z' are latent representations of e'. For any role M MM, 2" is the representation of agent embedding e* sampled by role M Following the literature, we denote (e, c'M, as a positive pair and denote {(e,e") M.MMMMMM E.M., as negative pairs. The objective Eq. ! optimizes for K-way classification loss to classify the positive pair out of all pairs. Minimizing the InfoNCE loss LK maximizes lower bound on mutual information in Eq. 2 and this bound becomes tighter K becomes larger. The role encoder ought to extract shared features in agent embeddings of the same role to maximize the score of positive pairs, while capturing essential distinctions across various roles to decrease the score of negative pairs.
Negative Pairs Generation. Periodically, we partition all n agents into K clusters {C₂),= accord- ing to agent embeddings F Naturally, we encourage role representations from the same cluster to stay close to each other, while differing from agents in other clusters. For agent i, we denote its role repre- sentation z; the query q. and role representations of other agents as the keys K = (=I,... Fn } Points from the same cluster as the query, i E Cj, are set as positive keys { and those from differ- KY( J. In practice, we use bilinear products (Laskin et al. 2020) for the score function in Eq. 3.. and similarities between the query and keys are computed as q W k. where W is learnable parameter matrix. The InfoNCE loss in Eq. (3) is rearranged as
ent clusters are set as negative {k=
exp(q W)4)
Evec, exp(z, Wzx)
Lx = log exp(q WKk) + exp(a Wk
log
Σ exp(z, W2;)) (4)
Σ exp(2 W FEC,
*CC,
Following the MoCo method (He et al.
2020 we maintain a query encoder 0,, and a key encoder gg. and use a momentum update to facilitate the key representations' consistency as
0₂ + 30 + (1 3)0,,
(5)
where B [0, 1) is momentum coefficient, and only parameters 8g are updated by backpropagation.
'Here, we use the superscript i for highlighting the time-evolving property of role representations and relevant variables, and we will partially omit it for simplicity in the below.
"In this paper, we simply use K-means (Hartigan & Wong 1979) based on Euclidean distances between agent embeddings. Moreover, it can be easily extended to more complex clustering methods such as Gaussian mixture models (Bishop) 2006). In Appendix D, we conduct the hyperparameter analysis about the influence of different K values and how determine the number of clusters automatically.
4

---

Published as a conference paper at ICLR 2024
2.2 ATTENTION EITENTION-GUIDED ROLE COORDINATION
After acquiring agents' contrastive role representations from their local information, we introduce an attention mechanism in value decomposition to enhance agent coordination in the sophisticated role space with a global view. Popular CTDE algorithms, such as QMIX, realize behavior coordination across agents via mixing network that estimates joint action-values as a monotonic combination of per-agent values, and the mixing network weights are conditioned on the system's 's global state. Natu- rally, it is interesting to incorporate the learned role information into the mixing network to facilitate skillful coordination across roles. The simplest approach is to concatenate the global state and role representations for generating mixing network weights, while it fails to exploit the internal structure to effectively extract correlations in the role space. Fortunately, the attention mechanism (Vaswani jet al. 2017) aligns perfectly with our intention by prompting the global state to attend to learned role patterns, thus providing more expressive credit assignment in value decomposition.
The attention mechanism aims to draw global dependencies without regard to their distance in the input output sequences, and has gained substantial popularity as a fundamental building block of compelling sequence modeling and transduction models, such as GPT (Brown et al. 2020), vision transformers (Dosovitskiy et al. 2022T. and decision transformers (Chen et al. 2021b). An attention function can be described as mapping a query and a set of key-value pairs to weighted sum of the values, where the weight assigned to each value is computed by compatibility function of the query and corresponding key. As role representations are learned based on extracting agent behaviors from history trajectories, we also use a GRU to encode the history states (s", ss", ..... s²) into a state embedding T' for facilitatingg information matching between states and role representations. Then, [= =n] Raxd as both the key and value, where d is the dimension of role representation and d, is the length of state embedding. Formally, we calculate a weighted combination of role representations as
we set the state embedding T E RRd as the query, and the role representations
Tallen
Of122 i=1 2, W.
(6)
where the value V; is a linear transformation of z; by a shared parameter matrix W E [ddx.. The attention weight a; computes the relevance between the state embedding t and the i-th agent's role representation 2,, and we apply softmax function to obtain the weight as
TWQ (H,WK)T
exp
Val
12 プド= exp (J TWO (T)WK)T)²
where Wº, W E dd are shared parameter matrices for linear transformation of query-key pairs, and 1/ 1/vd Te is a factor that scales the dot-product attention. We use multi-head attention (MHA) for allowing the model jointly attend information from different representation sub- spaces at different positions, and obtain the aggregated output as
(8) where T (h E (1, H)) is the attention output using projections of W W and WY, and Wo E IR Hd xd is the parameter matrix for combining outputs of all heads. Finally, the MHA output is combined with the global state to be responsible for generating weights of the mixing network, as shown in Fig. d)). In this way, we flexibly leverage role representations to offer more comprehensive information for value decomposition. By allowing the global state to attend to the learned role patterns, the attention mechanism implicitly guides the agent coordination in skillful role space, thus yielding more expressive credit assignment with the emergence of roles.
Tmha
Concat (Talenn..., T... a W
3 EXPERIMENTS
We evaluate ACORM to answer the following questions: (i) Can ACORM facilitate learning effi- ciency and stability in complex multi-agent domains? If so, what are the respective contributions of different modules to the performance gains? (See S.I.B.I). (ii) Can ACORM learn meaningful role representations associated with agent's behavior patterns and achieve effective dynamic team composition? (See Sec. 3.2). (iii) Can ACORM successfully attend to learned role representations to realize skillful role coordination and more expressive credit assignment? (See Sec. 3.3
5

---

Published as a conference paper at ICLR 2024
ACORM (Ours)
ECI
RODE
MACC
CIA
QMIX
CDS
GoMARL
3s5z [easy)
5m 6m (hard)
642g (hard)
1.0
Win Rale
3.88
Rate
0.88
Win
0.6
0.6
0.6
0.4
0.4
0.4
165t
0.2
3.22
0.2
0.0
0.0
1.0
0.5
10
1.5
0.0
0.5
3.0
1.5
2.0
Timesteps
Timesteps
1cB
Timesteps
1c5
MMM2 (super hard)
3552 vs. 3562 (super hard)
corridor (super hard)
1.0
1.0
1.0
Win Rate
0.88
1est Win Hate
0.88
30.8
0.6
my
0.6
0.4
0.4
0.2
8.02-
0..2
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
1.5 Timesteps
2.5
3.0
0.0
0.5
1.0
1.5
2.0
3.0 1e6
Timestops
1e6
1e6
Timesteps
Figure 2: Performance comparison between ACORM and baselines on six representative maps.
Implementations. We choose SMAC (Sanvelyan et al. 2019) as the first testbed for its rich maps and convenient visualization tools, and realize ACORM top of the popular QMIX algorithm. For visualization, we render game scenes, and show agent embeddings and role representations using t-SNE. Appendix C gives evaluation results the GRF benchmark, and Appendix D shows the algorithm architecture, experimental settings and results of MAPPO-based ACORM.
Baselines. We compare ACORM to QMIX and six baselines: 1) RODE (Wang et al. 2021 with action space decomposition; 2) EOI (Jiang & Lu 2021) that encourages diversified individuality via training an observation-to-idennte to concentrate on most related subtasks; 4) CDS (Li et al.) (021) that introduces diversity in both
classífier; 3) MACC (Yuan et al. (022) that uses attention
optimization and representation; 5) CIA (Liu et al. 2023)
that boosts credit-level distinguishability 2023) with an automatic grouping mechanism.
via contrastive learning; and 6) GOMARL Zang et al.
3.1 PERFORMANCE AND ABLATION STUDY
For evaluation, all experiments are carried out with five different random seeds, and the mean of the test win rate is plotted as the bold line with 95% bootstrapped confidence intervals of the mean (shaded). Appendix B describes the detailed setting of hyperparameters.
Performance. SMAC contains three kinds of maps: easy, hard, and super hard. Super hard maps typically complex tasks that require deeper exploration of diversified behaviors and more skillful coordination. Since ACORM is designed to promote these properties, the performance on these maps is especially significant to validate our research motivation and advantages. Fig. presents the performance of ACORM on six representative tasks, and performance on more maps can be found Appendix B. ACORM obtains the best performance on all super-hard maps and most of the other maps. A noteworthy point is that ACORM outperforms all baselines by the largest margin on super hard maps that demand a significantly higher degree of behavior diversity and coordination: MMM2, $s52_vs_33s 62, and corridor: In these maps, ACORM gains an evidently inster-increasing trend regarding the test win rate in the first million training steps, which supposed benefit from the high efficiency of exploring cooperatively heterogeneous behaviors via our discriminative role assignment. Moreover, ACORM exhibits the lowest variance in learning curves, signifying not only superior learning efficiency but also enhanced training stability.
Ablations. We carry out ablation studies to test the respective contributions contrastive learn- ing and attention. We compare ACORM to four ablations: i) SCORM_W/O__S it only ex- cludes contrastive learning; ii) (Vanilla), it removes both the attention and state encoding, and directly feeds the current state into the mixing network like QMI:; and iv) QMIX, it removes all compo- nents. For ablations, all other structural modules kept consistent strictly with the full ACORM.
w/O_MHA, it only removes the attention module; iii)
ACORM

---

Published as a conference paper at ICLR 2024
ACORM
ACORM wio MHA((anllaa
ACORM, wwlo MHA
ACORM_WiS
QMIX
MMM2 (super hard)
3557
3562 (super hard)
corridor (super hard)
1.0
1.0
1.0
90.8
1851 Win Rate
Win Raleo
30.8
0.6
0.6
0.6
0.4
0.4
0.4
2.2
0.2
0..2
0.0
1.0 Timesteps
0.5
2.5
3.0
1.0
2.5 Timesteps
3.0
Timesteps
166
Figure 3: Ablation studies. ACORM w/o_CL, removes contrastive learning, ACORM MHA removes attention, and ACORM w/o MHA (Vanilla) removes attention and state encoding.
Figure shows ablation results on three super hard maps, and more ablations on other maps can be found in Appendix F. When either of the two components is removed, ACORM obtains decreased performance and still outperforms QMIX. It demonstrates that both components are essential for ACORM's capability and they are complementary each other. Especially, both ACORM w/o CL and ACORM w/o MHA achieve significant performance gains compared to QMIX. which further verifies the respective effectiveness in tackling complex tasks. Specifically, ACORM w/o MHA (Vanilla) obtains very similar performance compared to ACORM BiRMMM_wo__rP Wo_MHA, indicating that the effectiveness comes from the attention module other than encoding the state trajectory via GRU,
3.2 CONTRASTIVE ROLE REPRESENTATIONS
To answer the second question, we gain deep insights into learned role representations through visu- alization on the example MMM2 task, where the agent controls team of units (1 Medivac, 2 Maraud- ers, and Marines) to battle against an opposing army (1 Medivac, 3 Marauders, and 8 Marines). Fig. 4 presents example rendering scenes in evaluation trajectory of the trained ACORM policy. Initially 12), all agent embeddings tend to be crowded together with limited discrimination, and the K-means algorithm moderately separates them into several clusters. Via contrastive learning, the acquired role representations within the same cluster are pushed closer to each other, and those in different clusters are notably separated. tered widely throughout the space with good clustering effect so far. This phenomenon indicates that the system has learned effective role assignment with heterogeneous behavior patterns. Then, the role encoder transforms these agent embeddings into more discriminative role representations.
later stage (t=40), agent embeddings are already scat-
The team composition naturally evolves over time. = 1, Marauders {0, 1} form group and Marines 3,, 4, 5,6,77 7} form another due to their intrinsic agent heterogeneity. In the middle of the battle t = 12, Marauders {0, 1} join the same group of Marines {2, 4, 7, 6, 8) to focus fire on enemies, while Marines (3, separate from the offense team since they severely injured. Late in the battle at f = 40, Marines (2. 3, 6, 7} are still in the offense team, while Marauders {0, 1} and Marine 5 fall into the same dead group. In summary, it is clearly verified that ACROM learns meaningful role representations associated with agent's 'ss behavior patterns and achieves effective dynamic team composition. More insights and explanations be found in Appendix G.
3.3 EYTENTION-GUIDEA ROLE COORDINATION
To answer the last question, we visualize attention weights a Eq. (6) using heatmaps, as shown in Fig. 5 The number of agent clusters is K = 4. In most heads, roles in the same cluster have similar attention weights, while different clusters exhibit significantly varying weights (e-g., in all four heads at f == 10, the weight distribution over four clusters: {0, 1, 2, 3). (5), (4, 6, 7, 7,8). (9)). This phenomenon indicates that the global state has successfully attended to the learned role patterns.
The attention mechanism draws several interesting insights on the battle, such as: i) Head 2 evidently attends to the injury-rescus pattern, since the largest weights come from Medivac and low-health units (Marauder 1 at t 4, Marine 5 at t 10, and Marines {3, 4, 6, 7) at t 36). ii) In most heads, attention weights of Marauders (0, 1} are usually high at the beginning. and are significantly decreased over time. It corresponds the behavior pattern that Marauders play important role
7

---

Published as a conference paper at ICLR 2024
agent embedding
role representation
cluster
cluster
cluster 3
cluster 4
10
20
40-
+
0
20
-10-
-20
-40
-20
-30
20 -40
10
20
60
(a) t= |
(b) t 12
(c) [ 40
Figure 4: Example rendering seenes at three time steps in an evaluation trajectory generated by the trained ACORM policy MMM2. The upper row shows screenshots of combat scenarios that contain the information of positions, health points, shield points, states of ally and enemy units, etc. The lower row visualizes the corresponding agent embeddings (denoted with bullets 'o') and role representations (denoted with stars **') by projecting these vectors into 2D space via t-SNE for qualitative analysis, where agents within the same cluster are depicted using the same color.
10
222
6.5
12
0.2
622
5.
3.6
6.6
6.8
62
46
66
28
10
32
4.
311
75
28
28
11
3.
66
68
26
15
27
377
1.5
81
644
03
10
9.0
57
86
98
711
19
100
t=
4
(b) 10
(0) 36
Figure 5: Example rendering scenes in an evaluation trajectory generated by the trained ACORM policy on MMM2. The lower row visualizes attention weights (a in Eq. (6)) of all four heads that explain how the global state attends to each role to guide skillful coordination in the role space. A higher weight means a larger contribution made by the corresponding role for value decomposition.
at carly attacks and the offensive mission will gradually be handed over to Marines. ii) On the verge of victory at t= 36, the primary concern is using Marines (2,3, 5} make final attacks with low-health Marines {4, 6, 7} providing auxiliary support. Obviously, heads {0, intuitively reflect this strategy, as Marines {2, 3,5), 5) have the highest weights, followed by Marines {4, 6, 7} and all other units. Moreover, the capability of our attention module could be much more profound than these examples from the superficial visualization.
4 RELATED WORK
Agent Heterogeneity. As compelling paradigm, CTDE Foerster et al. 2016) has yielded numer- algorithms (Lowe et al. 2017: Son et al.] 2019 Wang al. 2023). Many of them share policy parameters to improve learning efficiency and seale to large-scale systems, which results in homo- geneous behaviors across agents (Liu et al. 2022) To promote diversity, SePS (Christianos al.
8

---

Published as a conference paper at ICLR 2024
2021 partitions agents into fixed set of groups and shares parameters within the same group only, while igmoring evolving dynamics of the team. GoMARL (Zang et al., 2023) generates dynamic groups with automatic grouping mechanism to possess diverse strategies. MAVEN (Mhajjan et al. 2019) learns diverse exploratory behaviors by introducing a latent space for hierarchical con-
trol. CDS (Li al. (02T)) equips each agent with an additional local Q-function decompose
the policy to the shared and non-shared part. EOI (Jiang & Luj (021)) promotes individuality by encouraging agents to visit their own familiar observations. CIA (Liu et al.] 2023) boosts agent
distinguishability in value decomposition via contrastive learning. While differentiating each agent from the rest, these methods neglect the development of effective team composition with implicit task allocation, and might hinder the discovery of sophisticated coordination patterns.
Role Emergence. Researchers have also introduced the role concept into multi-agent tasks (Sims) et al. 2008 [haksmana et al. [0118] Xia et al. 2023] Cao et al., (023), or similarly, the concept
of skills (Yang et al.] 2020a) or subtasks (Yuan et al. 2022). ROMA (Wang et al. 2020) condi- tions individual policies
roles and solely relies on the current observation to generate the role
embedding, which might be inadequate for capturing complex agent behaviors. RODE (Wang et a
2021) associates each role with a fixed subset of the full action space to reduce learning complexity. Following RODE, SIRD (Zeng et al. (003) transforms role discovery into hierarchical action space clustering. Nonetheless, they neglect the evolving dynamics of the team since roles kept fixed in the training stage. For dynamic role assignment, some works learn identity representations to group agents during training, and maintain selection strategy to realize the assignment from agents to skills (Liu et al., 2022) or subtasks (Yang et al. [022] Igbal et al., 2022). Nevertheless, they encode the identity solely from a one-hot vector, which might be insufficient distinguish complex agent characteristics. COPA (Liu et al. 2021) realizes dynamic role allocation via periodically dis- tributing a global view of team composition
each agent even in execution. However, relaxes the CTDE constraint by introducing communication during decentralized execution, and the global composition is simply sampled from a fixed set of teams.
In summary, our method differs from the above approaches involving the role concept, and exhibits several promising advantages. Our method strictly follows the CTDE paradigm, accommodates the dynamic nature of multi-agent systems, and learns more efficient role representations.
Contrastive Learning and Attention Mechanism. Contrastive learning is gaining widespread popularity for self supf-vupeee representation learning in various domains (He et al. 2020 Su al. 2022 Laskin et al., 2020). As a simple and effective technique, contrastive learning is also inves- tigated assist MARL tasks, such as boosting the credit-level distinguishability (Liu et al., 2023), facilitating the utilization of the agent-level contextual information (Song et al. (003)). and ground- ing agent communication (Lo & Sengupta| 2022). In this study, we apply contrastive learning to optimize role representations, facilitating sophisticated coordination with better role assignment.
Attention is the fundamental building block of famous transformer architectures (Vaswani et al., 2017) that exhibit growing dominance in advances of Al research (Brown et al. [020] Dosovitskiy ct al. 202T; Chen et al. 20216). Due its superiority in extracting dependencies between se- quences, attention has been widely applied to MARL domains for various utilities, such as learning a centralized critic (Iqbal & Sha, (2019), concentrating on relevant subtasks (Yuan et al. 20022) for- mulating MARL as sequence modeling problem (Wen et al., 2002)), addressing stochastic partial observability (Phan et al. 2023). etc (Yang et al.) 2020b/ Shao et al. [223 Zhai et al. 2003)). In this study, we use attention to guide skillful role coordination for more expressive credit assignment.
5 CONCLUSION AND DISCUSSION
In this paper, we propose a general framework that learns contrastive role representations to promote behavior heterogeneity and knowledge transfer across agents, and facilitates skillful coordination in a sophisticated role space via an attention mechanism. Experimental results and ablations verify the superiority of our method, and deep insights via visualization demonstrate the achievement of mean- ingful role representations and skillful role coordination. Though, our method does not consider the exploitation of history exploratory trajectories for extracting roles, and also needs an explicit clus- tering component with a pre-defined number of total behavior patterns. We leave these directions as future work. Moreover, extending our framework to offline settings is a promising line for practical scenarios where online interaction is expensive and even infeasible.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The work was supported by the National Natural Science Foundation of China under Grant 62376122, Grant 62073160, Grant 62276126, and Grant 62176116.
REFERENCES
Christopher M Bishop. Pattern Recognition and Machine Learning, volume 4. Springer, 2006.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, pp. 1877-1901, 2020.
Jiahan Cao, Lei Yuan, Jianhao Wang, Shaowei Zhang, Chongjie Zhang, Yang Yu, and De-Chuan Zhan. LINDA: Multi-agent local information decomposition for awareness of teammates. Science China Information Sciences, 66( (122101), 2023.
Dong Chen, Kaian Chen, Zhaojian Li, Tianshu Chu, Rui Yao, Feng Qiu, and Kaixiang Lin. Power- net: Multi-agent deep reinforcement learning for scalable powergrid control. IEEE Transactions on Power Systems, 37(2):1007-1017, 2021a.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, pp. 15084-15097, 2021b.
Filippos Christianos, Georgios Papoudakis, Muhammad Rahman, and Stefano Albrecht. Scal- ing multi-agent reinforcement learning with selective parameter sharing. In Proceedings of Inter- national Conference on Machine Learning, pp. 1989-1998. 2021.
Mehdi Dastani, Virginia Dignum, and Frank Dignum. Role-assignment in open agent societies. In Proceedings of International Conference on Autonomous Agents and Multi-Agent Systems, pp. 496, 2003.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. In Proceedings of International Conference Learning Representations, 2021.
Jakob Foerster, Joannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning, In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In Proceedings of International Conference Machine Learning, pp. 1 1146-1155, 2017.
Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in coopera- tive multi-agent reinforcement learning. Proceedings of International Conference on Machine Learning. pp. 6863-6877, 2022.
John A Hartigan and Manchek A Wong. Algorithm as 136: k-means clustering algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 1801)1000-108, 1979.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In Proceedings IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Siyi Hu, Chuanlong Xie, Xiaodan Liang, and Xiaojun Chang. Policy diagnosis via measuring role diversity in cooperative multi-agent RL. In Proceedings of International Conference on Machine Learning, pp. 9041-9071, 2022.
10

---

Published as a conference paper at ICLR 2024
Shariq Iqbal and Fei Sha. ings of Interaational Conference on Machine Learning, pp. 2961-299977 2019.
for multi-agent reinforcement learning. In Proceed-
Shariq lqbal, Robby Costales, and Fei Sha. ALMA: Hierarchical learning for composite multi-agent tasks. In Advances in Neural Information Processing Systems, pp. 7155-7166, 2022.
Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In Proceedings of International Conference Machine Learning, pp. 4992-5001, 2021.
Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michat Zajac, Olivier Bachem, Lasse Espeholt, Car- los Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. Proceedings of AAAI Conference on Artificial Intelligence, pp. 4501-4510, 2020.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions for reinforcement learning, In Proceedings of International Conference on Machine Learn- ing. pp. 3639-5650 2020.
Joel Z Leibo, Vinicius Zambaldi, Mare Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In Proceedings of International Conference on Autonomous Agents and Multi-Agent Systems, pp. 4644473, 2017.
Kemas M Lhaksmana, Yohei Murakami, and Toru Ishida. Role-based modeling for designing agent behavior in self-organizing multi-agent systems. International Journal of Software Engineering and Knowledge Engineering, 28(01):79-96, 2018.
Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Cele- brating diversity in shared multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3991 10002 2021.
Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandkumar. Coach-player multi-agent reinforcement learning for dynamic team composition. In Proceedings of Interna- tional Conference on Machine Learning, pp. 6860-6870, 2021.
Shunyu Liu, Yihe Zhou, Jie Song, Tongya Zheng, Kaixuan Chen, Tongtian Zhu, Zunlei Feng, and Mingli Song. Contrastive identity-aware learning for multi-agent value decomposition. In Pro- ceedings of AAAI Conference on Artificial Intelligence, pp. 11595-11603, 2023.
Yuntao Liu, Yuan Li, Xinhai Xu, Yong Dou, and Donghong Liu. Heterogeneous skill learning for multi-agent tasks. In Advances in Neural Information Processing Systems, pp. 37011-37023, 2022.
Yat Long Lo and Biswa Sengupta. Learning to ground decentralized multi-agent communication with contrastive learning. arXiv preprint arXiv:2203.03344 033304 2022.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competiive environments. In Advances in Neural Infor- mation Processing Systems, pp. 9379, 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, pp. 7611 1-7622, 2019.
Frans A Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs, volume 1. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.0377 00740, 2018
Thomy Phan, Fabian Ritz, Philipp Altmann, Maximilian Zom, Jonas Nüßlein, Michael Kölle, Thomas Gabor, and Claudia Linnhoff- Popiin.. Attention-based recurrence for multi-agent inforcement learning under stochastic partial observability. In Proceedings of International Con- ference Machine Learning, pp. 27840-27852 2023.

---

Published as a conference paper at ICLR 2024
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi aulti-agent reinforcement learning. Journal of Machine Learning Research, 21(1):7234-77 7284, 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt. Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft multi-agent challenge. arXiv preprint arXiv: (902:0490000443333 2019.
Jianzhun Shao, Zhiqiang Lou, Hongchang Zhang, Yuhang Jiang, Shuncheng He, and Xiangyang Ji. Self-organized group for cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 5711 1-5223,, 2022.
Jianzhun Shao, Hongchang Zhang. Yun Qu, Chang Liu, Shuncheng He, Yuhang Jiang, and Xi- angyang Ji. Complementary attention for multi- ageent reinforcement learning. In Proceedings of International Conference on Machine Learning, pp. 30776-30793, 2023.
Mark Sims, Daniel Corkill, and Victor Lesser. Automated organization design for multi-agent sys- tems. Autonomous Agents and Multi-Agent Systems, 16: 15:1511 2008.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. QTRAN:
Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Proceedings of International Conference Machine Learning, pp. 5887-5896, 2019.
Haolin Song, Mingxiao Feng, Wengang Zhou, and Houqiang Li. MA2CL: Masked attentive con- trastive learning for multi-agent reinforcement learning. In Proceedings of International Joint Conference on Artificial Intelligence, pp. 4226-4234 2023.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. In Advances in Neural Information Processing Systems, pp. 21548-21561, 2022.
Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems, pp. 2244-2252, 2016.
Peter Sunchag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Velue-decomposition networks for cooperative multi-agent learning based on team reward. Proceedings of Interna- tional Conference on Autonomous Agents and Multi-Agent Systems, pp. 2085-2087. 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 9998-6008, 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782):350-355 2019.
Jiangxing Wang, Deheng Ye, and Zongqing Lu. More centralized training, still decentralized exe- cution: Multi-agent conditional policy factorization. In Proceedings of International Conference on Learning Representations, 2023.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. ROMA: Multi-agent reinforcement learning with emergent roles. In Proceedings of International Conference on Machine Learning, pp. 9876-9886, 2020.
Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang- RODE: Learning roles to decompose multi-agent tasks. In Proceedings of International Confer- ence on Learning Representations, 2021.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, pp. 16509-16521, 2022.
12

---

Published as a conference paper at ICLR 2024
Yu Xia, Junwu Zhu, and Liucun Zhu. Dynamic role discovery and assignment in multi-agent task decomposition. Complex & Intelligent Systems, pp. 1-12, 2023.
Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforce- ment learning with skill discovery. In Proceedings of International Conference on Autonomous Agents and Multi-Agent Systems, pp. 1566-1574, 2020a.
Mingyu Yang. Jian Zhao, Xunhan Hu, Wengang Zhou, Jiangcheng Zhu, and Houqiang Li. LDSA: Learning dynamic subtask assignment in cooperative multi-agent reinforcement learning. Ad- vances in Neural Information Processing Systems, pp. 1698-1710, 2022.
Yaodong Yang. Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv 2020b.
preprint ar arXiv:2002.0392
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Advances in Neural Information Processing Systems, pp. 14611-24622 2022.
Chao Yu, Xinyi Yang. Jiaxuan Gao, Jiayu Chen, Yunfei Li, Jijia Liu, Yunfei Xiang, Ruixin Huang, Huazhong Yang. Yi Wu, and Yu Wang. Asynchronous multi-agent reinforcement learning for ef- ficient real-time multi-robot cooperative exploration. In Proceedings of International Conference Autonomous Agents and Mulri-Agent Systems, pp. 1107-1115, 2023.
Haoqi Yuan and Zongqing Lu. Robust task representations for offline meta-reinnorcemmett learning via contrastive learning. In Proceedings of International Conference on Machine Learning, pp.
25747-25759, 2022.
Lei Yuan, Chenghe Wang, Jianhao Wang, Fuxiang Zhang, Feng Chen, Cong Guan, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. Multi-agent concentrative coordination with decentralized task representation. In Proceedings of International Joint Conference on Artificial Intelligence, pp. 599-605, 2022.
Yifan Zang, Jinmin He, Kai Li, Haobo Fu, Qiang Fu, Junliang Xing, and Jian Cheng. Automatic grouping for efficient cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023.
Xianghua Zeng, Hao Peng. and Angsheng Li. Effective and stable role-based multi-agent collab- oration by structural information principles. In Proceedings of AAAI Conference on Artificial Intelligence, pp. 1777--1778, 2023.
Yunpeng Zhai, Peixi Peng, Chen Su, and Yonghong Tian. Dynamic belief for decentralized multi- agent cooperative learning. In Proceedings of International Joint Conference on Artificial Intelli- gence, pp. 344-352, 2023.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Mont- gomery Alban, Iman Fadakar, Zheng Chen, et SMARTS: Scalable multi-agent reinforcement learning training school for autonomous driving. In Proceedings of Conference on Robot Leam- ing, 2020.
13

---

Published as a conference paper at ICLR 2024
APPENDIX A. CONTRASTIVE ROLE REPRESENTATION LEARNING
In this section, we give the proof of Theorem I in the text based on introducing lemma as follows. Lemma 1. Given role from the distribution M P(M), lete fo(E, (o'aa') as the agent embedding generated by role M, and ~ fo(zle), where Er aa= the agent's local trajectory following a given policy. Then, we have
p(M[z) (MM)
p(zle)
Ee
(9)
p(2)
Proof.
p(M
p(21AM)
(MM)
p(=)
J. plel p(eeMD((zee) de
(10)
p(2)
[(2le)] p(z)
= Ee
The proof is completed.
Theorem 1. Let M denote sel of roles following the role distribution P(M), and |M| K. M M M given role. Let e == fo( a 2 fo(zle), and hie, 2) p(z(e)
where
p(z)
Eo'o',a'-1) is the agent's 's local trajectory following given policy. For any role M E M, let e* denote the agent embedding generated by the role M then we have
h(e,2)
[zz M) 2 log K + EM. log EMEEMh hile,
(2)
Proof.
Using Lemma A. and Jensen's 's inequality, we have
p(=lc) p(=)
h(e,z) 2)
E EM log
E M2
log p(aee)
EM-EM h(e²,2)
-EMEMM
p(aee
P(2)
P(2)
p(2)
p(zlet)
EMM
log
1
Σ
p(z(e)
M.EMiM p(2)
p(2)
p(2le")
2 E.M.2.c
log
(K 1)BMEMM
p(zle)
p(2)
1 p(2)
-E,
log
(K - 1)
p(z|e)
E.M.
log
1 p(e)
(K 1)
p(=|c)
< EMM..
log
p(=) K p(zle)
p(((ee)
< E,M
log EE
log K
p(=)
log p(M (z)
E.M=
log K
p(M)
[(2; M) - log K.
(1I)
Thus, complete the proof.
14

---

Published as a conference paper at ICLR 2024
APPENDIX B. IMPLEMENTATION DETAILS AND EXTENDED EXPERIMENTS OF QMIX-BASED ACORM
Based the implementations in Section 2, summarize the brief procedure of ACORM based
QMIX in Algorithm D
Algorithm 1: ACORM based on QMIX
Input: Φ; agent's 's trajectory encoder
0: role encoder
K: number of clusters
Ta: time interval for updating contrastive loss
number of agents
replay buffer
T: time horizon of learning episode
Initialize all network parameters
Initialize the replay buffer B for storing agent trajectories for episode = 1, 2..... do
Initialize history agent embedding el, and action vector af for each agent for = 1, .... T do
Obtain each agent's partial observation the {of]== and global state for agent 1, .2. do
Calculate the agent embedding e = [o a; ,
Calculate the role representation 2 = fo(e))
Select the local action a; according to individual Q-function Qi(e, at)
11
end
Execute joint action a' = [ai, ab...., T and obtain global reward yd end
12
13
Store the trajectory to B Sample batch of trajectories from B
15
16
if episode mod Tel
then Partition agent embeddings {ef) into K clusters {C using K-means for agent 2, n do
Construct positive keys (z |ve, and negative keys (z+- Jiyc, for query z,, iE end
20
21
Update contrastive learning loss according to Eq. 44h
22
Update momentum role encoder according to Eq.
23
Calculate attention output Timaa via prompting the global state to attend to role representations {= in Eqs. (6--8
24
Concatenate Tmha with state embedding T to form the input to the mixing network
25
Update the parameters of individual Q-network and the mixing network
end
15

---

Published as a conference paper at ICLR 2024
In this paper, the attention mechanism. Specifically, the trajectory encoder contains fully-connected multi-layer perceptron (MLP) and GRU network with ReLU as the activation function, and encodes agent's 's trajectory into a 228-dimensional embedding vector. The role encoder is fully-connected MLP that transforms the 128- 1o-dimensional agent embedding into 64-dimensional role representation. The setting of the mixing network is kept as the same as that of QMIX (Rashid et al. 20200. where the architecture contains two 32-dimensional hidden layers with ReLU activation. Table shows the details of network structures.
simple network structures for the trajectory encoder, the role encoder, and
For all tested algorithms, we use the Adam optimizer with learning rate of 6e-4. For exploration, we use the e-greedy strategy with annealed linearly from 1.0 to 0.02 over 80k time steps and kept constant for the rest of the training. Every time an entire episode from online interaction is collected and stored in the buffer, the Q-networks are updated using a batch of 32 episodes sampled from the replay buffer with capacity 5000 state transitions. The target Q-network is updated using soft update strategy with momentum coefficient 0.005. The contrastive learning loss is jointly trained every 100 steps of the Q-network updates. The decentralized policy is evaluated every 5k: update steps with 32 episodes generated. For all domains, the number of clusters in ACORM is set to K 3. Appendix D provides an analysis on this hyperparameter, and the experiments show that the performance of ACORM is not significantly affected by the value of K. The details of hyperparameters can be found in Table
Table 1: The network configurations used for ACORM based on QMIX.
Network Configurations
Value
Network Configurations
Value
role representation dim
64
hypernetwork hidden dim
32
agent embedding dim state embedding dim attention output dim attention head num attention embedding dim
128
hypemneworr layers num
2
64 type of optimizer
Adam
64
activation function
ReLU
4
add last action
True
128
Table 2: Hyperparameters used for ACORM based QMIX.
Hyperparameter
Value
Hyperparameter
Value
buffer size
5000
start epsilon 6,
1.0
batch size
32
finish epsilon cf
0.02
lcarning rate
x 10-4
e decay steps evaluate interval
80000
use learning rate decay
True
5000
contrastive learning rate
8 x 10-4
evaluate times
32
momentum coefficient /
0.005
target update interval
200
update contrastive loss interval Tat cluster num
100
discount factor 7
0.99
3
16

---

Published as a conference paper at ICLR 2024
ACORM (Ouns)
EDI
RODE
MACC
CIA
QMIX
CDS
GOMARL
leasy!
1c3s5z (easy)
3s5z leasy)
1.0
1.0
189t Win Rat3
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.0
0.6
0.8
1.0
0.2
0.4
0.8
1.0
0.0
0.5
3.0
1.5
2.0
Timesteps
1e6
Timesteps
1e6
banc bane (hard)
2c vs. 642g (hard)
5mn vs. 6m (hard)
1.0
80.8
80.0
90.8
0.6
0.4
0.2
0.2
P0.2
0.0
0.0
n.0
0.0
0.2
0.6
0.5
1.5
1.5
Timesteps
Timesteps
1e6
Timesteps
(super
hard)
1.0
Win Rate
3.88
0.0.
Rate
90.8
0.6
0.6
0.4
0.4
20.2
0.2
0.0
0.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
3.0
1.5
2.0
Timesteps
1e6
Timesteps
1a6
Timesteps
1e5
3s5z vs 3s6z (super hard)
6h vs 8z (super hard)
corridor (super hard)
1.0
1.0
3Nnn Rate
0..8
19st Win Rate
0.8
Win daea
m my
0.6
0.6
0.6
0.4
0.4
0.4
0..
30.2
0.2
0.0
1.
Timesteps
Timesteps
Timesteps
Figure 6: Extended performance comparison between ACORM and baselines on 12 SMAC maps.
Fig. 6 presents the extended performance of ACORM 12 SMAC maps. Obviously, the observa- tions and conclusions from the extended performance are kept consistent with those in Sec. 3.1 of the main paper.
17

---

Published as a conference paper at ICLR 2024
APPENDIX C. EXTENDED EXPERIMENTS ON GOOGLE RESEARCH FOOTBALL
In addition to SMAC environments, we also benchmark our approach on three challenging Google research football (GRF) offensive scenarios
academy_ 3 vs __wiie th th_keeeerr Three of our players try to score from the edge of the box, one on each side, and the other at the center. Initially, the player at the center has the ball and is facing the defender. There an opponent keeper.
4 versus counter-attack with keeper; all the remaining players of both teams run back towards the ball.
a
academy_run_toe tason_tt__so______ee www www.wwhhteeeereeee Our player starts in the middle of the field with the ball, and needs to score against a keeper. Five opponent players chase ours from behind.
In GRF tasks, agents need to coordinate timing and positions for organizing offense seize fleeting opportunities, and only scoring leads to rewards. In our experiments, we control left-side players except the goalkeeper. The right-side players are rule-based bots controlled by the game engine. Agents have discrete action space of 19, including moving in eight directions, sliding, shooting, and passing. The observation contains the positions and moving directions of the ego-agent, other agents, and the ball. The z-coordinate of the ball is also included. Environmental reward only occurs at the end of the game. They will get +100 if they win, else get -1.
Fig. presents the performance comparison between ACORM and baselines on three challenging GRF scenarios. It can be observed that QMIX obtains poor performance, since the tasks in GRF more challenging than in the SMAC benchmark. In contrast, ACORM gains significantly im- proved increase in the test win rate, especially in the first 1M training timesteps, which successfully demonstrates the effectiveness of our method evaluated on GRF benchmarks. Together with evalua- tions on SMAC domains, the same conclusion can still be drawn that ACORM outperforms baseline methods by larger margin on harder tasks that demand significantly higher degree of behavior diversity and coordination. In summary, experimental results on extended GRF environments are generally consistent with those on the SMAC benchmark.
Due to the very limited time for rebuttal revision, we only compare ACORM to QMIX and CDS, as other baselines (RODE, EOI, MACC, CIA) are not evaluated on GRF in their original papers. Also, we just show the performance within 2M training steps. We hope that the extended experimental evaluation could demonstrate adequate persuasiveness of our method. We are rushing to conduct the evaluation of baseline methods GRF scenarios with more training timesteps, and trying to update them before the rebuttal deadline.
ACORM (Ours) academy run, to score
RODE
MACC
QMIX
GOMARL
academy 3 vs. with keeper
academy run, pass pa___ppssddd shoot, with keeper
Vin Ca40.
0.88
Rate
0.8
0.6
0.6
0.6
M
0.44
Ø0.2
0.02
0.0
0.0
0.0
0.5
2.0
1.5
Timesteps
Timestops academy corner
1e6
Timestops
academy, counterattack
stochastic
1.0
1.0
1.0
30.8
300.8
go.8
0.6
0.6
20.4
0.4
0.4
20.2
20.2
1o.2
0.0
0.0
0.5
1.0 Timesteps
1.5
2.0
0.0
0.5
1.0
1.5 Timesteps
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
1c6
1c6
Timesteps
165
Figure 7: Performance comparison between ACORM and baselines three GRF scenarios.
18

---

Published as a conference paper at ICLR 2024
APPENDIX D. ACORM BASED ON MAPPO
In addition, we realize ACORM on top the MAPPO algorithm, as show Fig. g Most of the network structure is kept the same QMIX-based ACORM. including the agent embedding, the role encoder, and the attention mechanism. To align with MAPPO that uses an actor-critic architecture, input both the agent's observation and the augmented global state & (obtained from the attention mechanism in Fig. [(e)) into the critic, as shown in Fig. B().. Tables and present the detailed network structure and experimental hyperparameters, respectively.
training execubin
Contrnstiv Loss
Similaity
Actor
Actor
Controstive Representation
of
Encader
Momentum Encoder
GRU
of (1E G
et (t' c))
ef (C' $ C.)
Critie
Critic
MLP
of
Clustering
Product
MI.
Attention
e
Softmax
(c)
h GRU
A/
Scaled Dot
Product
MI.P
MLP
W
W
W:
Role Representation
Agent Embedding
Figure 8: The ACORM framework based on MAPPO. (a) The overall architecture. (b) The shared actor network structure for each agent, where the role representation is extracted from agent's tra- jectory. (c) The detail learning role representations via contrasting learning. (d) The shared critic network structure for each agent. (e) The attention module that incorporates learned role represen- tations into value decomposition.
Table 3: The network configurations used for ACORM based on MAPPO.
Network Configurations
Value
Network Configurations
Value
role representation dim
64
attention output dim
64
agent embedding dim state embedding dim critic RNN hidden dim
128
attention head num
4
64
attention embedding dim
128
64
type of optimizer
Adam
add agent ID
False
activation
ReLU
19

---

Published as a conference paper at ICLR 2024
Table 4: Hyperparameters used for ACORM based MAPPO.
Hyperparameter
Value
Hyperparameter
Value
batch size
32
entropy coefficient
0.02
mini batch size
32
cluster num
actor learning rate
10
discount factor 7
0.99
critic learning rate
8 X 10
momentum coefficient 3
0.005
contrastive learning rate
x 10-4
evaluate interval
5000
update contrastive loss interval Tel clip
32
evaluate times
32
0.2
use advantage normalization
True
GAE lambda À K epochs
0.95
use learning rate decay
False
5
ACORM (Ours)
MAPPO
2532 Jeasy]
3s52 (easy)
vs 6m (hard)
1.0
1.0
80.8
185t Win Rate
0.0
9/..8
0.6
0.6
0..4
0.4
0.2
3..2
0.2
0.0
0.0
0.6
0.2
0.8
2.0
Timesteps
Timesteps
166
Timestops
MMM2 (super hard)
corridor [super hard)
3r52 vs 3552 (super hard)
1.0
Win Rate
30.8
Hate
0.8
3..8
0.6
0.6
0.6
0.4
0.4
20.2
10.2
0.0
0.0
0.0
0.5
1.0
15
2.0
0.0
0.5
1.0
1.5
2.0
3.0
Timesteps
1e5
Timesteps
16B
Timesteps
1e6
Figure 9: Performance comparison between MAPPO-based ACORM and the MAPPO baseline representative maps, including two easy levels (2s3z, 3s5z), one hard level (5m_vs_6m)) and three super hard levels (MMM2, corridor. 3s5z_vs__3e 3362)
Figure presents the performance of MAPPO-based ACORM on six representative tasks: 2s32, 3s5z, 5m_vs_____ 6m, MMM2, corridor, and 3s5z_v ____v__________ _s66.. It can be observed that ACORM achieves a significant performance improvement over MAPPO. Akin to the QMIX-based version, ACORM outperforms MAPPO by the largest margin super hard maps that demand signifi- cantly higher degree of behavior diversity and coordination. Again, experimental results demon- strate ACORM's superiority of learning efficiency in complex multi-agent domains.
20

---

Published as a conference paper at ICLR 2024
APPENDIX E. HYPERPARAMETER ANALYSIS ON THE NUMBER OF CLUSTERS
We test the influence of the number of clusters K on ACROM's performance. Fig. 10 shows the performance of ACORM with varying values of K 2, 3, 4,5. Generally speaking, ACORM obtains similar performance across different values of K. It demonstrates that ACORM achieves good learning stability and robustness as the performance is insensitive to the pre-defined number of role clusters. An outlier case is observed in the 5m_ 6m map where the ACORM's performance drops little when K 5. This is likely because there are only agents in 5m_vs_ ________________ _6.. When K =5 5, each agent represents distinct role cluster. It forces the strategies of each agent to diverge, which might not be conductive to realize effective team composition across agents.
K=2
k=
k=
K
3552 (easy)
5m vs 6m (hard)
MMM2 (super hard)
1.0
2.0
1.0
8.08-
00.8
1est Win Rate
0.6
0.6
0.6
0.4
0.44
0.4
0.2
2
0.2
0.2
0.0
0.0
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Timesteps
146
Timesteps
1e6
Timestops
1e6
Figure 10: Hyperparameter analysis on the number of clusters K in negative pairs generation.
To illustrate why ACORM performs well across different values of K, we show visualizations from different learned ACORM policies with K 3, 4, 5 in Fig. II When the clustering granularity is coarse as K = 3 in Fig. Maa,, even within the same cluster, ACORM can still learn meaningful role representations with distinguishable patterns. Agent embeddings of Marines {2, 3, 4, 5, 6, 7.8} crowded together with limited discrimination. Via contrastive learning, the obtained role rep- resentations exhibit an interesting self-organized structure where Marines (2.5, 6} and (3. 4, 8] implicitly form two distinctive sub-groups. It can be observed from the rendering scene that Marines {2, 5, and {3, 4, 8, are all at an attacking stage, while the former sub-group is in lower health than the latter. On the other hand, with fine granularity of K 5 in Fig. ITc., the contrastive learning module transforms clustered agent embeddings into more discriminative role representa- tions. For example, while Marines 3, 5, 6, 8, and 4, 7 form three clusters, their role representations are still closer to each other and farther from Marauders {0, 1} and Medivac {9), since they are the same type of agents with similar behavior patterns. In summary, it again demonstrates that ACORM learns meaningful role representations and achieves effective and robust team composition.
agent embedding role representation
cluster
cluster 2
cluster cluster 4
cluster
20
20
64
10
9
-20
O+
20-
-10
-40-
10
-20
7
1
-20
20
40
-20
20
-10
0
(8) K=
(b)
(c)
Figure 11: Visualization with different number of clusters in negative pairs generation.
21

---

Published as a conference paper at ICLR 2024
APPENDIX F. MORE ABLATION RESULTS
Figure 12 presents the full ablation results of ACORM on all six representative maps. It again demonstrates the significance of both the contrastive learning module and the attention mech- anism for ACORM's performance. A noteworthy point is that both ACORM_w/S W/M___WW_LLLLL and ACORM w/o Liw_w/ooHHH_ MHA gain remarkable performance improvement by the largest margin on super hard maps, which further validate ACORM's advantages of promoting diversified behaviors and skillful coordination
complex multi-agent tasks. Another interesting observation is that when omitting the contrastive learning module, there is notable increase in the variance of learning curves. can be interpreted as an evidence that contrastive learning helps training more robust role representations and enhances learning stability.
ACORM
ACORM_WA w/o MHA(Vanilaa)
ACORM wio MHA
WHO
ACORM_ wil_ CL
QMIX
3s52 [easy]
5m vs 6m (hard)
25. vs 642g (hard)
1.0
1.0
185 Win Hate
0.8
80.8
0.6
0.6
0.6
0.4
0.4
0.4
10.2
0.02-
0.0
0.0
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Timesteps
1c6
Timesteps
leb
Timesteps
1e6
(super
(super
corridor (super
1.0
1.0
1.0
Rate
Win deee
9..8
0.6
0.6
0.44
00.4
0.4
0.22
0..
0.0.2
0.0
1.0 Timestops
1.0
2.5
3.0
1.0 1.5
3.0
Timesteps
Timesteps
Figure 12: Full ablation results on ACORM. ACORM_ w/o CL means removing contrastive learn- ing, ACORM w/o MHA represents excluding attention, ACORM MHA (Vanilla) represents excluding attention and state encoding, and QMIX corresponds to removing all components.
22

---

Published as a conference paper at ICLR 2024
APPENDIX G. MORE INSIGHTS ON CONTRASTIVE REPRESENTATION LEARNING
Based Section 3.2, here we provide more insights the effectiveness of learned role represen- tations. Indeed, one reason for ACORM's significant improvement on learning efficiency comes from its capability of facilitating implicit knowledge transfer across similar agents throughout the entire learning process. For example, Marines {2. 3, 4, 5, 6, 7 form a group at = 1, Marauders {0. and Marines {2,4, 6, 7) form a group t 12, and Marines {2, 3, 4, 6, 7} form a group at = 40. It can be observed from the rendering scenes that these three groups are all responsible attacking enemies. At different time steps, agents in the attacking group can share similar role representations to promote knowledge transfer, even if they belong to heterogeneous agent types. This implicit transfer across agents and across timesteps can significantly increase the exploration efficiency of agents.
Another highlight of ACORM is the promotion of behavior heterogeneity, even agents have the same innate characteristics. For example, while Marines {2, 4, 5, 6, 7, 8) belong the same agent type, they are distributed to different groups with heterogeneous roles as: 1) at = 12, Marines {2, 4. 6, 7 with role of attacking, and Marines {3} and (3) with role of the wounded; and 2) at t= 40, Marines {2, 3, 4, 6, 7} with role of attacking, and Marines {5} and (8) with role of the dead. In general,
information is lost during dimension reduction using t-SNE, is evident that our role representation still manages to exhibit such remarkable results.
though
agent embedding
role representation
cluster
cluster
cluster
cluster 4
20
0-
-20
-20
-10
10
-20
20
40
(=)==
(b) =
(c) 1
Figure 13: Example rendering scenes at three time steps in evaluation trajectory generated by the trained ACORM policy on MMM2. The upper row shows screenshots of combat scenarios that contain the information of positions, health points, shield points, states of ally and enemy units, etc. The lower row visualizes the corresponding agent embeddings (denoted with bullets 'o') and role representations (denoted with stars 'x') by projecting these vectors into 2D space via t-SNE for qualitative analysis, where agents within the same cluster are depicted using the same color.
23


Reviews:
[{"review_id": "H3Tz2jhgYt", "reviewer": "Reviewer_53j5", "summary": "The paper introduces a method that combines a multi-head attention structure and the InfoNCE contrastive learning framework to enhance learning efficiency in MARL tasks by learning and utilizing role representations.", "questions": "1.\tIn Fig 4. (c), both Agent 5 and 8 are 'Dead Marines'. Why are they clustered into different classes?\n2.\tWhy was a new map, 2s3z, introduced for the experiments in MAPPO? Why not directly use the previously employed 3s5z_vs_3s6z map? I would like to see an additional MAPPO experiment on 3s5z_vs_3s6z.\n3.\tHow is the setting cluster_num = 3 applied on the map 2c_vs_64zg when there are only 2 agents available for control on this map? Besides, why didn't the performance decline since it forces the strategies of each agent to diverge as same as the experiment about cluster_num = 5 applied on the map 5m_vs_6m?\n4.\tIn the derivation, both the approximation and logK indicate that a larger value of K yields better results. The experiments conducted in the selected maps have a limited number of agents. It is suggested to have more agents and experiment with larger values of K. For example, experiments with larger K values, such as K=2, 4, 8, 16, can be run in scenarios like 30m and bane_vs_bane.\n5.\tThe paper doesn't explicitly clarify the difference between clustering directly on agent embeddings and on role representations. As it considers role representations to be more discriminative, it's important to further elucidate the necessity of obtaining discriminative representations through contrastive learning.\n6.\tThe model has been added with a global state GRU and a MHA structure. Therefore it increase the number of parameters of the networks. It is recommended to conduct ablation studies with the same network size.\n\nMinor: Bigger size of networks and additional contrastive learning procedure may limit the application. The theoretical derivation of Theorem 1. is very similar to previous work.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe description of the methodology is clear and accurate.\n2.\tThe performance of the experiments' results is really promising and impressive.\n3.\tThe chapter of related works is rich and comprehensive.", "weaknesses": "1.\tExtremely Lack of Experiments. There is a lack of experiments as the author only conducted experiments on 6 maps in SMAC. The same applies to the ablation experiments. Hope there will be additional experiments in a wider range of environments and on more maps within SMAC.\n2.\tThe author did not provide the source code to verify.\n3.\tThere might be some errors in the analysis. Such as the analysis in Appendix D, there is room for debate regarding the phenomenon of sub-groups. It is incorrect to measure the distance between 2 points within 1 cluster in the original space based on their proximity in the t-SNE space. Evaluating the emergence of sub-groups should start from the original space rather than the two-dimensional space after t-SNE reduction. Similarly, after t-SNE reduction, the distance between clusters still does not reflect the real distance. Therefore, the conclusion of 'their role representations are still closer to each other' in the later part still requires the author's reconsideration.\n4.\tThe relationship between the two parts, MHA and CL, in the article is not particularly close; they seem more like two relatively independent components.\n5.\tThere are some typos and the writing of the paper needs some improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hCG1utJpHq", "reviewer": "Reviewer_WWCs", "summary": "This paper introduces the ACORM framework, which utilizes mutual information maximization to formalize role representation learning through a contrastive learning objective. It also incorporates an attention mechanism to encourage the global state to attend to learned role representations.\nEmpirical evaluations carried out on SMAC scenarios demonstrated that ACORM surpasses the performance of baseline methods. Additionally, visualizations and ablation studies show the pivotal roles played by the contrastive role representation and attention mechanism in this task.", "questions": "* As previously mentioned, ACORM does not impose constraints on the attention mechanism, and it even utilizes the learned latent variable $z$. How can we ensure its correct execution under the CTDE paradigm?\n* Regarding the analysis of Contrastive role representation, Figure 4 is not particularly convincing:\n    * In subfigure (b), *(0) is even further from other points in its cluster compared to *(5). While I understand that clustering is done in higher dimensions, this example can be confusing.\n    * While it's claimed that the role representation better forms coordination teams, in actuality, in subfigure (b) and (c), it seems just the agent embedding alone might suffice.\n* Additional experiments are needed to bolster the paper's persuasiveness.\n* The supplementary materials do not include the source code, making reproducibility uncertain.\n\nI would like to raise my score if my concerns are addressed.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed ACORM framework integrates representation learning, encoding the trajectory history from the traditional framework into a latent variable z. This representation associated with the role is learned through clustering and using positive-negative samples.\n* Compared to the traditional framework, the global state used incorporates role-related representations through an attention mechanism.", "weaknesses": "* **Novelty and Reliability**: ACORM is not the first work to introduce the attention mechanism in the mixing network. For instance, works like [Qatten](https://arxiv.org/pdf/2002.03939.pdf) have introduced certain constraints in the network to satisfy the IGM principle. However, this paper does not provide evidence of complying with the IGM principle or any explanations.\n* **Experimental Evaluation**: The experiments in the article are conducted solely on SMAC. To my knowledge, various versions of SMAC exist, and different algorithm implementations often involve custom modifications to this environment. Relying solely on SMAC for experiments may not be sufficiently persuasive. It might be beneficial to include experiments from other environments such as GRF and Ma-MuJoCo.\n* **Reproducibility**: The supplementary materials do not include the source code, making reproducibility uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zrV9ByiiF6", "reviewer": "Reviewer_1yPm", "summary": "This paper proposes a novel framework of attention-guided contrastive role representation learning for multi-agent reinforcement learning, ACORM. On the one hand, the role representation of each agent is inferred from the agent embedding through contrastive learning and clustering algorithms. On the other hand, the paper introduces an attention mechanism in value decomposition to enhance agent coordination in the role space. By introducing the above two contributions, ACORM performs better than other role-based multi-agent reinforcement learning algorithms in the SMAC environment. The paper also has intuitive visualizations to illustrate the role of the corresponding modules.", "questions": "Please see the questions in the weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to understand.\n2. The experimental part has detailed case studies, which is very important for understanding the role of the two modules proposed in the paper. The figures about the t-sne embedding or weights corresponding to each snapshot have reasonable analysis.\n3. The proposed framework is suitable for reinforcement learning algorithms based on value functions and those based on policy gradient. The relevant algorithms have been tested on SMAC and show that the ACORM variant is much better than the vanilla algorithm.\n4. The proof of the ELBO is given in the appendix, which is correct to me and improves the soundness of the submission.", "weaknesses": "1. In cooperative multi-agent reinforcement learning, inferring the role of an agent based on its trajectory is not a novel method and has been proposed in many previous works [1, 2]. Moreover, none of the above-mentioned important papers are cited in the paper.\n2. The number of baselines used for comparison with ACORM is relatively tiny. Why not use CDS [3] as a baseline, since you mentioned it in the paper?\n3. SMAC is a relatively old multi-agent testbed. Recently, it has been pointed out that it has a series of problems [4]. I am not against the author evaluating the performance of the algorithm on SMAC, but I feel that the performance of the algorithm should be tested in multiple different domains. Many environments, such as the Google Research Football [5] mentioned in the paper, can be used to enhance the credibility of experimental results.\n4. Ablation experiments are insufficient. Compared with the vanilla QMIX, ACORM_w/o_MHA still has an additional MLP and GRU for the global state. One wonders whether what really works is just the representation learning of the state trajectory before input to the Mixing Network.\n5. It is not possible to reproduce the results from the description given in the paper. Some key details (such like $T_{cl}$) are unclear, and some key resources (code) are not furnished.\n\n\n\n**Reference**\n\n[1] Cao, Jiahang et al. LINDA: multi-agent local information decomposition for awareness of teammates. 2021.\n\n[2] Yang, Jiachen et al. Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery. 2019.\n\n[3] Li, Chenghao et al. Celebrating Diversity in Shared Multi-Agent Reinforcement Learning. 2021.\n\n[4] Ellis, Benjamin et al. SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning. 2022.\n\n[5] Kurach, Karol et al. Google Research Football: A Novel Reinforcement Learning Environment. 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zgpnltNd0H", "reviewer": "Reviewer_jtxX", "summary": "The paper proposes a method of attention-guided promotion (incorporating attention mechanisms in the global state to participate in value decomposition) to maximize mutual information to formalize role representation and derive a contrastive learning objective function. ACORM choose the StarCraft multi-agent challenge (SMAC) benchmark and achieves state-of-the-art performance on most hard and superhard maps.", "questions": "1. Does GRU encoding S play a more significant role in the effect ?\n\n2. How do you know the state encoding after this attention, input to the mix network can have an impact on credit assignment ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Using mutual information to distinguish roles\n\nThe experiment was conducted on a difficult map in StarCraft\n\nThe experimental diagram is very detailed", "weaknesses": "There is no reasonable explanation or formula for the promotion of credit assignment  by attention, and the paper only demonstrates the effectiveness of the method through experiments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)