Learning Distinguishable Trajectory Representation with Contrastive Loss
Authors: Tianxu Li
Kun Zhu
Juan Li
Yang Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-Agent Reinforcement Learning
multi-agent exploration
trajectory representation
contrastive learning
multi-agent diversity
Primary area: reinforcement_learning

Abstract: Policy network parameter sharing is a commonly used technique in advanced deep multi-agent reinforcement learning (MARL) algorithms to improve learning efficiency by reducing the number of policy parameters and sharing experiences among agents. Nevertheless, agents that share the policy parameters tend to learn similar behaviors. To encourage multi-agent diversity, prior works typically maximize the mutual information between trajectories and agent identities using variational inference. However, this category of methods easily leads to inefficient exploration due to limited trajectory visitations. To resolve this limitation, inspired by the learning of pre-trained models, in this paper, we propose a novel Contrastive Trajectory Representation (CTR) method based on learning distinguishable trajectory representations to encourage multi-agent diversity. Specifically, CTR maps the trajectory of an agent into a latent trajectory representation space by an encoder and an autoregressive model. To achieve the distinguishability among trajectory representations of different agents, we introduce contrastive learning to maximize the mutual information between the trajectory representations and learnable identity representations of different agents. We implement CTR on top of QMIX and evaluate its performance in various cooperative multi-agent tasks. The empirical results demonstrate that our proposed CTR yields significant performance improvement over the state-of-the-art methods.

Learning Distinguishable Trajectory Representation with Contrastive Loss
Tianxu Li
Kun Zhu 122
Juan Li
Yang Zhang!
*College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics,
China
?Collaborative Innovation Center of Novel Software Technology and Industrialization {tianxuli, zhukun, yangzhang, juanli)@nuas edu. cn
Abstract
Policy network parameter sharing is commonly used technique in advanced deep multi-agent reinforcement learning (MARL) algorithms to improve learning effi- ciency by reducing the number of policy parameters and sharing experiences among agents. Nevertheless, agents that share the policy parameters tend to learn similar behaviors. To encourage multi-agent diversity, prior works typically maximize the mutual information between trajectories and agent identities using variational inference. However, this category of methods easily leads to inefficient exploration due to limited trajectory visitations. To resolve this limitation, inspired by the learning of pre-trained models, in this paper, we propose a novel Contrastive Tra- jectory Representation (CTR) method based on learning distinguishable trajectory representations to encourage multi-agent diversity. Specifically, CTR maps the trajectory of an agent into latent trajectory representation space by an encoder and an autoregressive model. To achieve the distinguishability among trajectory representations of different agents, we introduce contrastive learning to maximize the mutual information between the trajectory representations and learnable identity representations of different agents. We implement CTR top of QMIX and evalu- ate its performance various cooperative multi aalli- tasks. The empirical results demonstrate that proposed CTR yields significant performance improvement over the state-of- the-art methods.
Introduction
Cooperative multi-agent reinforcement learning (MARL) can provide effective collaboration among agents and has shown promise for solving real-world multi-agent tasks, such robot swarms [Hüttenrauch et al. 2017), autonomous driving [hhllla et al. 2020 Dinneweth et al. 2022] and wireless communications [Li et al.] (022b). However, effective collaboration in complex multi-agent tasks still remains a challenge for MARL. One of the key issues is that the joint action-observation space grows exponentially in size with the number of agents, which highlights urgent demand for the scalability of MARL algorithms.
To address the scalability issue, learning decentralized policies for agents has been widely adopted. This allows agents to make action decisions based on their partial observations. However, learning a private decentralized policy network for each agent in MARL may require training a large amount of policy network parameters, resulting in inefficient learning. To enhance learning efficiency, many advanced MARL algorithms adopt the parameter sharing technique, including policy gradients (Lowe)
et al., 2017 Ma et al. 2021, Wang et al.) 2020d] Ndousse et al. 2021 Zhang et al.] 2021 and
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

value-based algorithms [Igbal et al. 2021 Yang et al. 2021 Wang et al. 2020a Sunehag et al. 2018 Rashid al. 2018]. Incorporating parameter sharing enables all agents to make action decisions using a shared policy network. This significantly reduces the number of policy network parameters. Additionally, training a shared policy network facilitates the sharing of experiences among all agents, alleviating the unstable learning problem arising from partial observability. These advantages of parameter sharing dramatically improve the learning efficiency and accelerate the training speed of MARL algorithms [Wang et al. 2020b].
However, although parameter sharing has many advantages, agents sharing the policy network parameters tend to become homogeneous since they typically learn similar behaviors under similar observations [Hu et al. [0222] Mahajan et al.) 2019). resulting in inefficient exploration and poor diversity. Challenging multi-agen tasks typically require extensive exploration and diverse policies among agents. For example, in a foothall game, agents in team require to play different roles such as goalkeeper, defender, midfielder, and forward, taking diverse tactics to achieve more credits. If they behave similarly to compete for a ball, they may not achieve satisfactory results.
One of the most common methods to encourage multi-agent diversity is to maximize the mutual information between trajectories and agent identities by using variational inference methods (Jiang and Lu 202 Li et al. 202T] that learn parameterized trajectory discriminators to distinguish the trajectories of different agents given agent identities. However, due to the high mutual dependence between agent identities and trajectories, the agents tend to frequently visit known trajectories that contain more identity information, where they can achieve larger rewards than discovering new trajectories, leading to serious overfitting of trajectories to agent identities. Consequently, despite the emergence of diversity among agents, the agents unfortunately suffer from inefficient exploration.
To encourage multi-agent diversity while guaranteeing efficient exploration, we propose novel Contrastive Trajectory Representation (CTR) method based on learning distinguishable trajectory representations, which encourages multi-agent diversity in an abstract contrastive representation space. Our motivation that, although the shared policy network may receive similar inputs, it can still learn diverse representations, leading to varied behaviors. Unlike previous mutual information- based methods using variational inference, our method adopts a novel contrastive learning lower bound for the mutual information between trajectory representations and learnable identity representations. Notably, the learnable identity representation introduced in our method differs entirely from the fixed agent identity used in prior works. It is trained by minimizing the contrastive learning loss in order to constrain the trajectory representations of different agents to be linearly classified. As a result, the distinguishability among trajectory representations can be achieved and does not depend any fixed agent identity. The learned distinguishable trajectory representations can then be used in the downstream action-decision tasks to learn more diverse and exploratory policies.
Our contributions can be summarized as follows: first, we propose a novel method for encouraging multi-agent diversity through learning distinguishable trajectory representations, which minimizes the contrastive learning loss between trajectory representations and identity representations of different agents. The distinguishable trajectory representations do not rely on fixed agent identities and thus lead to more efficient exploration; second, to reduce the gap between the contrastive learning lower bound and the mutual information objective caused by the small size of the dataset storing trajectory representations, we further extend the contrastive learning loss by increasing the number of negative samples; third, we provide practical learning framework for CTR and apply our approach to QMIX; forth, we evaluate CTR in both grid world environments and the StarCraft Multi-Agent Challenge (SMAC) benchmark. The empirical results demonstrate that CTR significantly outperforms the existing state-of-the-ar methods and yields more exploratory and diverse policies.
2 Backgrounds
2.1 Multi-Agent System
We consider learning in the fully cooperative multi-agent Decentralized Partially Observable Markov Decision Process (Dec-POMDP) [Olichoek and Amato) [2055] described as tuple (A, U, P, R, O, 22 7), where A represents set of A agents, 8 E S is the global state of the environment, and U is a set of agents' actions. At the beginning of each time step, each agent receives an observation o E l2 according to the function O(s,a) a) and then selects an action u E U. All the agents' actions compose a joint action u, and the environment then transitions to the next
2

---

((
Trajettry Reperghere
Trajectery Reprocatoiion
umx ((
prostin
Reprevstttivn
Enceder
(ct.k.)
Agent
Contradtivn Learaing
Figure I: Architecture of CTR model.
state s' with the probability drawn from the transition function P (s'' u). At the same time, the environment feeds back to the agents a shared team reward r = R(s, u). E 10 is a reward discount factor. The observation-action pairs (o", u²) of agent make up its trajectory 7" E T. Each agent a learns its individual policy T7"h (u* ra), forming a joint policy π,, to maximize the joint
action-value function Q" (s, u) Eso000. L r're So 8, Ug u. T).
000
Contrastive Trajectory Representation
3.1 Motivation and Intuitions
To promote multi-agent diversity, the agents need to learn diverse policies. In order to achieve this purpose, prior works [Jiang and Lu 2021 Li 2021] have devoted to maximizing the mutual information between trajectories and agent identities of different agents via variational inference methods. However, this category of methods forces the agent visit known trajectories where they can achieve larger rewards than discovering new ones, making the learned policy prone to overfitting. The theoretical analysis of this limitation is provided in Appendix A To address this issue, the intuition behind our proposed CTR is that we can instead learn diverse policies from the trajectory representations distributed on contrastive representation hypersphere. In this section, we show how to learn trajectory representations and how the learned distinguishable trajectory representations can be used learn diverse policies practical learning algorithms.
3.2 Contrastive Trajectory Representation
In this section, we introduce the details of our CTR model that encourages multi-agent diversity by learning distinguishable trajectory representations.
The architecture of the CTR model is shown in Figure First, a non-linear encoder Cene maps the input assembled by the observation of agent a at time step of as well the last step action uf to latent representation 2/ = (of, uf-1). Next, to encode the previous action-observatioo sequences of the trajectory, an autoregressive model Cyar is used to summarize all the latent representations = and generate a trajectory representation ot Cgar (2g,). The trajectory representation of can empirically achieve better performance compared with the latent representation of when used to make action decisions in the Dec- Dec-POMDP setting since it stores additional information from the historical trajectory that can alleviate the non- staaionaton issue caused by the partially observable constraints (Sunchag al. 2017]. In practice, for simplicity, we adopt standard network structures such as resnet blocks for the encoder and GRUs for the autoregressive model. is notable that other types of encoder and autoregressive models also be employed in the CTR model.
Next, we introduce how to train the CTR model to learn distinguishable trajectory representations, encouraging multi-agent diversity. It can be difficult to enlarge the distance between the trajectory representations of different agents directly. Instead, we can introduce an additional learnable identity representation for each agent to represent the agent identity. Unlike previous works [Jiang and Lu

---

2021 Li et al. 2021] that use fixed one-ho hot vectors to represent the agent identities, in this paper. the identity representation adopted in our method is learnable vector trained linearly classify the trajectory representations of different agents.
Concretely, at the beginning of the training process, randomly initialize learnable vector d" E R for each agent as its identity representation that has the same dimensions as the trajectory representation. To achieve the distinguishability between trajectory representations of different agents, we maximize the mutual information between the trajectory representations and identity representations of agents:
p(c, I d)
[e;;d) = H(ce) - H(c. d) = Be,d log p(c))
(I)
where H. is the entropy. Estimating the mutual information directly is typically intractable. In this paper, we present novel method to solve the objective of mutual information between the trajectory representations and identity representations, unlike previous variational inference methods. Concretely, inspired by contrastive learning [Chen et al. 2020], a popular self-supervised learning method for learning representations,
a contrastive learning loss, or the InfoNCE loss (Oord et al. 2018), to derive and optimize a tractable lower bound for the mutual information:
[(c d) 2 log([A)) - CN.
(2)
where [A] is the number of agents, and CN is the InfoNCE loss. Note that log((A)) is a constant. Therefore, by minimizing the LN. maximize the mutual information 1 [(e;; d). We next design a practical contrastive learning loss to learn distinguishable trajectory representations. Given a set
of trajectory representations of all agents at time step C = {<44] a'=! and agent a's identity representation d", the goal of contrastive learning is make sure that the identity representation of agent a d' is close with its corresponding trajectory representation of while being distant from other trajectory representations in C \\ {. To achieve this goal, minimize the contrastive learning loss:
f (ct.d)) Eee' Ec (ef (cf'do)
LN
E log (d=, (D/D
where f' (c,d)= = exp dd) E R. ced measures the similarity between the trajectory represen- tation e and the identity representation d. Minimizing the contrastive learning loss trains both the CTR model and the identity representations. Here, the identity representation each agent serves as linear classifier, linearly classifying the trajectory representations output by the CTR model for minimal contrastive learning loss. As result, the distinguishability among trajectory representations can be achieved.
3.3 Multi-Agent Contrastive Learning Loss
One limitation of applying the contrastive learning loss given by Equation 3 the multi-agent setting that the small size of dataset C, which is equal to the number of agents, induces a larger gap between the true mutual information objective and the contrastive learning lower bound, which can hurt the performance. The contrastive learning lower bound requires a larger number samples to tighten its value to the true mutual information [Oord et al. 2018). To resolve this problem, we consider extending the Equation the contrastive learning loss with |A] positive samples:
A
|A| (ct, dd)
CT
E
log
(4)
A
ENA
,dd) + LLAL oa" "CC."" fa'
Σ
(C"" dad)
dat")
The contrastive leaning loss L shown in Equation 4 calculates the expectation over |A] positive pairs (cf, dd 1141 and |A[[|A] 1) negative pairs ((c dd) Jeg' "C |A Notably, LT actively

---

increases the number of negative samples in the denominator from O((c)) to O ([C]3) that can help the contrastive learning loss with a smaller dataset to lead to more stable and robust results in challenging multi-agent tasks. By minimizing the LN, the trajectory representations of all agents stay close to their corresponding identity representations while being far away from other identity representations, leading to the distinguishability among trajectory representations. We refer the reader to Appendix C for the Pytorch-style pseudocode of our proposed CTR.
Differences to contrastive learning We note that the contrastive learning employed in our method is quite different from its common usage in self supervise pre- gre-training that learns representations by contrasting the positive and negative pairs of instances. In our method, we apply contrastive learning to learn representations in a fully supervised manner by introducing an identity representation for each agent. This allows the trajectory representations belonging to the same agent to be pulled together on the trajectory representation hypersphere, while simultaneously pushing apart trajectory representations from different agents. The fully supervised manner ensures that the minimization of the contrastive learning loss entails increased distances between trajectory representations of different agents.
3.4 Learning Algorithm
In this section, we discuss how to integrate CTR with existing MARL algorithms adopting the decentralized policy, to encourage multi-agent diversity. As illustrated in Figure [ we implement CTR in individual policy networks of agents to learn distinguishable trajectory representations. The overall learning framework of CTR should consist of two parts: (i) the RL loss function of the MARL algorithm to train the decentralized policy towards maximizing the environment returns; (ii) the contrastive learning loss of CTR an auxiliary loss function to train the decentralized policy in order to learn distinguishable trajectory representations. Thus, we can formulate the total loss function as follows:
Liotal LRL + aLi.
(5)
where CR. is the RL loss function and 2 is the contrastive learning loss function. &r is a hyperpa- rameter adjusting the weight of contrastive learning loss LN compared with the RL loss LRL- The overall CTR framework is trained end-to-end in centralized manner. In this paper, we consider integrating CTR with the value-decomposition framework where each agent learns policy through optimizing an approximation of the joint action-valus function, denoted by Qtot (s, u), as follows:
[ max Quot (s',u';8-) Qroe (s,u: u) 2
(Tn(0)=>
(6)
11
where 6 is the batch size of transition samples, and 0 represent the parameters of Qtor and target Qtot, respectively, as in DQN (Mnih al. 2013). Qtor is a combination of per-agent utilities Qu where the decentralized policies are derived. We consider using QMIX [Rashid et al. 2018 to
decompose the combination Qtor, which factors the Qtot into monotonic nonlinear combination of agent utilities. Thus, we can train the overall CTR framework end-to-end by minimizing:
Liotal = LTD(f) + aC
(7)
The TD loss (TD(O) trains both the mixing network and agent utility networks. The contrastive learning loss C trains both the identity representations and agent utility networks (including the autoregressive model as well as the encoder of CTR). It is notable that CTR is a component of the agent utility network to learn distinguishable trajectory representations via contrastive learning. Thus, CTR won't break the IGM rule [Rashid et al. 2018 of QMIX. Moreover, our method only adds one linear layer (identity representation), resulting in a small overhead that would not decrease the learning efficiency of the integrated learning algorithm. This is crucial for the parameter sharing mechanism since we do not need to train an additional neural network at the expense of learning efficiency (the advantage of parameter sharing) for promoting multi-agent diversity. We refer the reader to Appendix B for the implementation of CTR top of the policy gradient method.
5

---

Traestop
(a) Illustration of Mhe-Men
(b) Learning curves
(c) Visitation heatmap
of(d) CTR
Visitation heatmap of
QMIX
Figure 2: The performance comparison between our proposed CTR and baselines in Pac-Men.
CTR QMIX
OTRAN
100
100
70.
60-
We foneht
Tex: Wi- Roleeh
Test WM Rate
40
40
20-
20-
20
20-
Timestpps
1e6
Timesteps
Timestepe
(a) 3s5z (easy)
(b) 2c vs ____v___2222 642g (hard)
(c)7sz (hard)
60
Tel Wi Roterh))
Test Rore/N
Timesteps
Timestops
3ce
Timestops
(d) 6h VS 8z (super hard)
(e) corridor (super hard)
(f)3s5z 3s6z (super hard)
Figure 3: Performance comparison between our proposed CTR and baselines in SMAC scenarios. Without loss of generality, all results are presented with the mean and standard deviation of perfor- mance tested with five random seeds.
4 Experiments
In this section, to demonstrate the outperformance of our proposed CTR, we evaluate CTR in Pac- Men, SMAC, and SMACv2 benchmarks. We compare our method with various state-of state-of-the-ar algorithms: value-decomposstion algorithms (QMIX [Rashid et al. [2018). QTRAN [Son et al. 2019] and mutual information- based exploration methods (MAVEN [Mahajan al., 2019 FOI (Jang and
Lu ], SCDS [Li et al., 2021], PMIC [Li et al. 2022a LIPO [Chaakkrrn et al. 2023 and FoX Jo al. 2024)). To ensure fair comparison, we set the same values for common hyperparameters across different methods. The hyperparameters used in our experiments and training details are provided in Appendix G Moreover, CTR does not include the agent identity in the input of the shared policy network to take action decisions.
4.1 Pac-Men
We first design a grid-world environment called Pac-Men, as shown in Figure 2a to demonstrate the effectiveness of CTR in encouraging multi-agen diversity. In Pac-Men, four agents are initialized at the center of the maze. Each agent has a partial observation and can only observe 4x4 grid around them. The goal of each agent is to eat the dots randomly initialized in edge rooms. We set different lengths for the paths towards edge rooms to improve the task difficulty and only the downward path is within the agent's 's observation scope, highlighting urgent demand for efficient exploration. The

---

CTR OMIX
MAVEN
C01
SCDS
PMIC
LIPO
60
70
70
60
60-
Tost Fate:))
Test 150 Fatells
5.-
Wed
40-
30-
20
10
(a) lerran_S __raa____vs_3 5
(b) protoss_5 5_vs_5
(c)zerg_3e 1arg______ 5_vs_5
Figure 4: Performance comparison between proposed CTR and baselines in SMACv2 scenarios.
(a) MAVEN (b) BOI
(c)SCDS
(d) LIPO
(e) FoX
(f) CTR+QMIX
Figure 5: Visitation heatmaps of different algorithms in the terran 5 scenario.
We note that QMIX falls into the local optimum and does not yield satisfactory performance in Pac-Men since some agents learned similar policies and went to the same edge room, resulting in ineffective competition for dots among agents. This can be verified by the visitation heatmap of QMIX shown in Figure 2c where three agents go to the bottom edge room. In contrast, CTR achieves significantly superior to all baselines in Pac-Men, avoiding falling into local optimum through optimizing the objective of mutual information between the identity representations and trajectory representations using contrastive learning loss. As shown in Figure 2dl the diverse policies learned by CTR enable the agents to go different edge rooms, leading efficient cooperation among agents. The mutual information-base based baselines achieve similar performance since they fail to find the edge room with the longest path due to inefficient exploration.
performance of CTR and baselines is shown in Figure 2b
4.2 SMAC and SMACv2
The StarCraft Multi-Agent Challenge (SMAC) (Samvelyan al. 2019] is common-used benchmark for evaluating cooperative MARL algorithms. To demonstrate the effectiveness of proposed CTR. we conduct experiments in 6 SMAC scenarios: 3s5z (casy), 2c vs 64zg (hard), 7sz (hard), 6h, vs__sz _s_8 (super hard), corridor (super hard), and 3s5z vs 3s6z (super hard), Note that the performance is not comparable between different versions SMAC. We use the SC2.4. 10 version SMAC.
The test win rates achieved by our method and baselines in different SMAC scenarios are shown in Figure demonstrating the outperformance of our proposed CTR in the SMAC scenarios compared to baselines. In the super hard scenarios: 6h_vs_88 8z, corridor, and are more powerful than agents, CTR significantly outperforms all baselines. Qmix fails learn optimal policies in these scenarios since these scenarios typically require agents to learn to distribute the enemies' attack due to a large strength gap between agents and enemies, which necessitates the emergence of diverse policies. CTR dramatically improves the final performance of QMIX by learning distinguishable trajectory representations. Compared to other mutual information- based baselines, CTR is more robust in promoting multi-agent diversity and achieves impressive final performance in super hard scenarios. EOI performs poorly because probabilistic trajectory classifier overfits to the agent identities, hindering efficient exploration.
where enemies
Moreover, homogeneous behaviors such as 'focus fire' are desired the easy scenario 385z to quickly defeat enemies. Note that CTR would not hinder such homogeneous behaviors that can lead to more environmental rewards and conversely achieves satisfactory performance. We refer the reader to Appendix B for further evaluations of our method in the scenarios requiring homogencous behaviors.

---

Mén
10
Ohtrrr---etttt CTR tetterrteNee
OMXX
So Scemerion in SIAAC
Figure 6: Performance comparison between CTR and ablation variants in SMAC scenarios.
(b)
(d)
Figure 7: T-SNE plots of trajectory representations of different agents learned by different variants of CTR ((a) QMIX (b)CTR- CCCTR-Wth--- With-Ons Sath-One-HHot (c) CTR-With-TCC With-------sss (d) CTR-With With-InfoNCEE (e) CTR), emerging in the corridor scenario of SMAC.
Stochasticity and Exploration Due to the fixed team compositions and initial positions in the scenarios of SMAC, the agents can easily win the game when they master the particular action sequences, such 'kiting', demonstrating that the SMAC scenarios lack efficient stochasticity to test the exploration of MARL algorithms. We further conduct experiments on a more challenging benchmark called SMACv2 (Ellis al. [002) that enables stochasticity in SMAC scenarios via introducing random team compositions and random start positions.
We conduct experiments in three SMACv2 scenarios: terran_ vs protoss $__is_______ 5_vs_5, and vrr_5_vs__5 Figure 4 shows the win rates for CTR and baselines. CTR yields more robust performance than baselines and successfully adapts to the stochasticity from environments. We attribute this to the efficient exploration strategies derived from distinguishable trajectory representa- tions. The mutual information-baser baselines do not achieve satisfactory performance in SMACv2 scenarios. These algorithms are prone to overfitting since they prefer to visit known trajectories that contain more identity information than exploring trajectories, resulting in poor exploration. This can be verified by the visitation heatmaps provided in Figure [ where the movements of agents trained by the mutual information-basse methods located in the fixed partial areas of the map. In contrast, CTR incentivizes more efficient exploration since CTR prevents overfitting by mapping the trajectories onto a contrastive representation hypersphere, where the representations only need to satisfy the distinguishability constraint. As result, CTR yields more exploratory policies. The movements of CTR agents uniformly distributed on the map.
4.3 Ablation Study
In this section, we conduct ablation studies on the proposed CTR framework to investigate the contributions of the main components in the CTR framework, including (A) autoregressive model, (B) identity representation, and (C) contrastive learning loss with |A| positive samples. To test component A, we design CTR-With With-MI.P that ablates the autoregressive model employed in CTR by replacing with multiple layer perceptions (MLPs). To test component B, we design CTR- With-Onth Oie--One--oo which ablates the learnable identity representation by using pre-defined onc-hot vector to represent the identity of an agent. To test component C, we design two variants: CTR- With-TC-Loss and CTR-With- With-InfoNCE. CTR- WTR--Wih-TCC-LsSS optimizes trajectory classification (TC) loss that directly predicts the corresponding agent identities given the trajectory representations regardless of negative performs the vanilla contrastive learning loss given by Equation for [A] times to distinguish the trajectory representations of all agents.
samples. CTR- CTR-With-
We test these variants in three SMAC scenarios: 3s5z (easy), 2o_vs_6472 (hard), and corridor (super hard). The results are shown in Figure CTR-With-One with similar performance to QMIX. This performance decline indicates that the pre-defined one-hot
performs worst among all ablations,

---

vector fails to enable the agents to learn diverse behaviors. We further provide the t-SNE plots, as shown in Figure 7 for learned trajectory representations. It becomes apparent that in the case of the trajectory representations of different agents are mixed together as in QMIX. Similarly, we notice the performance decrease in WTT--WithTTC--oss This degradation arises from the TC Loss's primary focus on agent identity prediction, essentially learning maximum likelihood function over the collected trajectories given agent identities. However, it does not incorporate constraints aimed at ensuring the separation between trajectory representations of different agents. stay close
FTR--Wth-One--
Consequently, trajectory representations of different agents learned by ETR-With-TC--an to each other without distinct gap as demonstrated in Figure 7 making it challenging distinguish them efficiently. In contrast, CTR robustly constrains the trajectory representations of the same agent to be close and those of different agents to be far apart the trajectory representation hypersphere.
consistently achieves better performance than CTR-With-TT Wos across all scenarios, demonstrating the benefits brought by contrasting negative samples when learning distin- guishable trajectory representations. However, CTR-With- With-InfoNCE: results in obvious performance degradation in the super hard corridor scenario. This phenomenon indicates that the contrastive learning loss given by Equation 4 adooted in our CTR method that involves more negative samples when learning trajectory representations leads to more stable and robust results. As illustrated in Figure 7 the trajectory representations of different agents learned by CTR entail an increase in distances compared to those learned by CTR- With-Info. obvious performance drop in the 3s5z (casy) and 2c s__v "s_64zg (hard) scenarios. However, in the super hard corridor scenario, CTR-With-Me that learning the context of the trajectory using an autoregressive model is beneficial for agents to improve their performance.
Se--With-Inffon
CTR- With-MLP does not lead to an
yields dramatic decrease in performance, demonstrating
5 Related Works
The diversity in MARL encourages the differences between the policies of agents. SVO [ccKee et al. 2020) studies multi-agent diversity by drawing on the social value orientation, a theory from social psychology, to solve multi-agent social dilemmas. It verifies the utility of population heterogeneity in cooperative MARL. SVO realizes the social value orientation via forming an intrinsic reward to encourage diverse policies. RODE [Wang et al. 2020c] achieves multi-agent diversity by assigning various actions to restricted roles. RODE is efficient when the agent has a small action space that can be decomposed. It can be inefficient for RODE to be applied in continuous action with huge action space.
MAVEN Mahajan et al. 2019] proposes to enable the value-based agents to condition on a shared latent variable controlled by a hierarchical policy. To learn diverse joint behaviors, MAVEN maxi- mizes the mutual information between the latent variable and the trajectories. Another method EOI (Tang and Lu, 202T) proposes to encourage the individuality of agents by a supervised learning method that trains a probabilistic classifier to learn probability distribution over agents with regard to their observations. CDS [Li et al., 2021] adopts the objective of mutual information to encourage multi-agent diversity. CDS optimizes the mutual information by formulating lower bounds derived by the Boltzmann softmax distribution and the variational inference, respectively. PMIC (Li et al.. 2022a] maximizes the mutual information associated with superior cooperative behaviors while minimizing the mutual information related to inferior ones. CIA [Liu et al.) 2023) realizes the credit-level distinguishability of vulue-decomposition based methods by identifying the temporal credits of different agents. LIPO [Charakorn et al. 2023) considers policy compatibility as a means to learn diverse behaviors, identifying the unique behaviors of each policy by optimizing the mutual information objective. FoX [Jo et al. 2024) presents formation- geraation-based exploration, which encourages the exploration of various formations by guiding agents to thoroughly comprehend their current formations. These works have shown promise in encouraging multi-agent diversity. However, they overemphasized learning the dependence between the agent identity and trajectory, forcing the agent to frequently visit similar observations, preventing the agents from further exploration.
6 Limitations and Future works
As our method needs to contrast all agents' trajectories, this necessitates that the training process should be centralized so that the trajectories of all agents can be collected. Thus, method cannot

---

be applied to fully decentralized MARL methods. Moreover, in contrastive learning, by collecting many negative samples, the model is challenged to distinguish the positive pair from larger pool of negatives. This helps the model learn more robust and discriminative features. Although we developed multi-agent contrastive learning loss to increase the number of negative samples, however, the number of negative samples is still limited if the total number of agents in multi-agent environments is very small. For our future work, we may develop efficient methods to augment existing trajectory samples to increase the number of samples.
Despite the emergence of multi-agent diversity, we also note the need for homogeneous behaviors. Although our method would not impede the learning of homogeneous behaviors that can lead to more environmental rewards, how control diversity automatically can be an interesting direction for our future work.
7 Conclusion
In this paper, we consider learning distinguishable trajectory representations over raw trajectories to encourage multi-agent diversity. Our method achieves distinguishability among trajectory repre- sentations by maximizing the mutual information between trajectory representations and identity representations of agents through the minimization of the contrastive learning loss. We evaluate our method in different challenging cooperative tasks, and it demonstrates significant performance improvement over existing state-of-the-ard methods. Our simple yet effective method reveals the importance of representation learning in promoting efficient exploration, leading optimal policies.
Acknowledgments and Disclosure of Funding
This work was supported in part by National Natural Science Foundation of China (46011, and in part by Natural Science Foundation of Jiangsu Province (Grant No. 3K20211567, BK20222012).
References
S. Bhalla, S. Ganapathi Subramanian, and M. Crowley, Deep multi agent reinforcement learning for autonomous driving. Advances in Artificial Intelligence: 33rd Canadian Conference on Artificial Intelligence, Canadian Al 2020, Ottawa, ON, Canada, May 13-15, 2020, Proceedings, pages 67-78. Springer, 2020.
R. Charakorn, P. Manoonpong, and N. Dilokthanakul. Generating diverse cooperative agents by learn- ing incompatible policies. In The Eleventh International Conference on Learning Representations, 2023.
mattfoormm?in 1i---UUUGUUHT
Chen, Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
J. Dinneweth, A. Boubezoul, R. Mandiau, and S. Espié. Multi-agent reinforcement learning for autonomous vehicles: a survey, Autonomous Intelligent Systems, 2(1):27, 2022.
B. Ellis, S. Moalla, M. Samvelyan, M. Sun, A. Mahajan, J. N. Foerster, and S. Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2212.07488, 2022.
Hu, C. Xie, X. Liang, and X. Chang. Policy diagnosis via measuring role diversity cooperative multi-agent rl. In International Conference Machine Learning, pages 9041-9071. PMLR, 2022.
M. Hüttenrauch, A. Šošié, and G. Neumann. Guided deep reinforcement learning for swarm systems. arXiv preprint arXiv: 100910700111111 2017.
S. lqbal, C. A. S. De Witt, B. Peng. W. Böhmer, S. Whiteson, and F. Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4596-4606. PMLR, 2021.
10

---

J. Jiang and Z. Lu. The emergence of individuality. In International Conference on Machine Learning. pages 4992-5001. PMLR, 2021.
Y. Jo, Lee, J. Yeom, and S. Han. Fox: Formation-aware exploration multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12985-12994, 2024.
C. Li, T. Wang, C. Wu, Q. Zhao, J. Yang, and C. Zhang. Celebrating diversity in shared multi-agent reinforcement learning. Advances Neural Information Processing Systems, 34:3991-40022 2021.
Li, H. Tang, T. Yang, X. Hao, T. Sang, Y. Zheng, J. Hao, M. E. Taylor, W. Tao, and Z. Wang. Pmic: Improving multi-agent reinforcement learning with progressive mutual information collaboration. In International Conference on Machine Learning, pages 19979-12999 PMLR, 2022a.
Li, K. Zhu, Luong, D. Niyato, Q. Wu, Zhang, and Chen. Applications multi-agent reinforcement learning future internet: A comprehensive survey. IEEE Communications Surveys & Tutorials, 2022b.
Liu, Y. Zhou, J. Song. T. Zheng, K. Chen, Zhu, Z. Feng, and M. Song. Contrastive seentity-aware learning for multi-agent value decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11 11595-11603, 2023.
R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-compettivy environments. Advances in neural information processing systems, 30, 2017.
X. Ma. Y. Yang, C. Li, Y. Lu, Q. Zhao, and J. Yang. Modeling the interaction between agents in cooperative multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Aulonomous Agents and MultiAgent Systems, pages 853-861, 2021.
A. Mahajan, T. Rashid, M. Samvelyan, and S. Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32, 2019.
K. R. McKee, I. Gemp, B. McWilliams, E. A. Duefiez-Guzmán, E. Hughes, and J. Z. Leibo. So- cial diversity and social preferences in mixed-motive reinforcement learning. arXiv preprint arXiv:2002.023255, 2020.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 10t.:::3//26003 2013.
K. K. Ndousse, D. Eck, S. Levine, and N. Jaques. Emergent social learning via multi-agen rein- forcement learning. In International conference on machine learning, pages 7991-8004. PMLR, 2021.
F. A. Oliehoek and C. Amato. A concise introduction to decentralized pomdps, 2015.
A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018
Rashid, C. De Witt, G. Farquhar, Foerster, S. Whiteson, and M. Samvelyan. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In 35th International Conference on Machine Learning, ICML 2018, pages 6846-6859, 2018
M. Samvelyan, T. Rashid, C. S. De Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr, J. Foerster, and S. Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: /90.000400 2019.
K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pages 5887-5896. PMLR, 2019.
P. Sunehag, G. Lever, A. Gruslys, W. M. Czameeki,, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Son- nerat, J. Leibo, K. Tuyls, et al. Value-decompossiion networks for cooperative multi-agent learning, arXiv preprint arXiv: 1 706.0000....0. 05296, 2017.

---

P. Sunehag, G. Lever, A. Gruslys, W. M. Czamnekk, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Son- nerat. J. Leibo, K. Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, pages 2085-2087 2018.
Wang. Z. Ren, T. Liu, Y. Yu, and C. Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint ar arXiv:2008.0 XXh::2008.01062,, 2020a.
T. Wang, H. Dong, V. Lesser, and C. Zhang. Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprini arXiv:2003.088 08039, 2020b.
T. Wang, T. Gupta, A. Mahajan, B. Peng, S. Whiteson, and C. Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020c.
Y. Wang. B. Han, Wang, H. Dong, and C. Zhang. Dop: Off-policy multi-agent decomposed policy gradients. In International conference on learning representations, 2020d.
Y. Yang, X. Ma. C. Li, Z. Zheng, Q. Zhang, G. Huang, J. Yang, and Q. Zhao. Believe what you Implicit constraint approach for offline multi- reinforcement learning. Advances in Neural Information Processing Systems, 34: 14:10099--0000 2021.
T. Zhang, Y. Li, C. Wang, G. Xie, and Z. Lu. Fop: Factorizing optimal joint policy of maximum- entropy multi-agent reinforcement learning. In International Conference on Machine Learning, pages 12491-12500. PMLR, 2021.
12

---

A Theoretical analysis of limitations of existing methods
This section analyzes the limitation of existing methods that the agents tend to visit known trajectories rather than exploring new ones from a theoretical perspective. To achieve this purpose, we calculate the reward functions achieved by visiting known trajectories and new ones, respectively. The theoretical results demonstrate that the agents achieve more rewards for visiting known trajectories than exploring new ones.
Consider an objective of mutual information between the agent identity and trajectory т
[(i; r) E., log p(i r)] E, (loggg()])
(8)
E., [log go [)] - [E,log
where the p(i 7) is an unknown posterior distribution that is approximated by a variational
distribution ( T) derived by the variational inference approach. ge 7) can be trained via maximum likelihood on (itules induced by the policy π of each agent, To maximize the mutual information, the variational lower bound can be deployed in MARL as intrinsic reward
T (T, log qo (i' 7) log (?')
llo go (i' 7) log |||
where /' ~ p(i), a fixed uniform distribution. Here, - log p (v') = log |A| since we have a total num- ber of |A| agents. We assume perfect variational distribution go r) where Σ!A (ia | () 1.
Reward for known trajectories The reward function encourages the agents visit identity trajectories to highlight themselves from others where qo (i' r) + 1, thus
T 7max = log + log ||| log |||
(10)
Reward for new trajectories For unseen trajectories, the value of ge (i' () is unknown. Here, we add background class to the model in order to assign null probability to unseen trajectories. Therefore, the agent receives a penalization for visiting unseen trajectories:
lim
log qo (i' () + log |A|
(11)
new
qo (
We note that the intrinsic reward enforces the agent to visit known trajectories that contain more identity information, making the agents prone to overfitting and leading to static trajectories.
B The CTR implementation on top of MAPPO
In addition QMIX, MAPPO is another state-of-the-an art policy-based MARL algorithm SMAC. learns a shared actor network updated towards maximizing expected returns. To integrate with MAPPO, we deploy the CTR model in the actor network and introduce auxiliary gradients derived from the contrastive learning loss to the policy gradients used to update parameters the actor network. Thus, we can achieve the overall objective that updates the actor network toward maximizing expected returns while minimizing the contrastive learning loss to learn distinguishable trajectory representations
(12) where Jactor is the objective of MAPPO train the actor network. Since CTR only needs to learn distinguishable trajectory representations via training the actor network, we don't need to change the other components of MAPPO. The experimental results of CTR integrated with MAPPO are listed in Table I demonstrating the outperformance of our method compared to baselines.
Troal Jactor aLd
C Pseudocode for CTR
The PyTorch-style pseudocode for CTR is given in Algorithm I
13

---

Algorithm 1: PyTorch-style pseudocode for CTR
# batch: collected trajectories
dimension identity representation
# IAI: number of agents
Edentitsy_represnttaiiio def ctr ctr_loss (batcch):
linear(H, IAI)
ctr_out =
for t in range bbatch.seg.lenge.
input_t concat "aatch""bs"]]] t|, batch]["actionssoneebbo t-1]
# Assemble the inputs
z_ embedding encoder (input_t)
c_ embedding, suaden__stated autoregreaive.mddellleeeeeddd
hidden hidden_states) states)
ctr ctr_out. append embedding))
ctr_out = th. stattckkcttr_ou dim=1) # Concat over time
wtdeetityy.eprrseeetet ctr_out, ctr_ ctr_loss contrastive_learringggo
[Al) # Calculate the contrastive learning loss.
return ctr_loss
D Environmental details and Additional experimental results
This section gives the details of the experimental environments including Pac-Men, SMAC, and SMACv2. We design a grid-world environment Pac-Men to demonstrate the effectiveness of CTR encouraging multi-agent diversity. In the Pac-Men environment, four agents are initialized the center of the maze. Each agent has a partial observation and can only observe a 4 x4 grid around them. Each edge room has some randomly initialized dots. During each episode, the agent moves to one of the four edge rooms to eat dots. To improve the difficulty of the task, we set different path lengths for the paths to edge rooms. The path lengths for the downward, left, right, and upward paths are 3, 6, 6, and 10, respectively. Only one path is within the observation scope of the agent, which further challenges the agent's ss ability to explore the environment. Dots will refresh after all of them are eaten by agents. The reward received by the agent is the number of dots eaten in each step.
SMAC and its upgraded version SMACv2 are the benchmarks for cooperative multi-agent rein- forcement learning research. Both of them consist of variety of fully cooperative tasks that are implemented on top of Blizzard's is real-time strategy game StarCraft II to evaluate the effectiveness of various MARL algorithms, SMAC realizes the agent level control using the Machine Learning APIs provided by StarCraft II and DeepMind's PySC2. In each task, there is a combat scenario that involves two armies: one of the armies is controlled by the allied RL agents and the other is controlled by the built-in non-learned game AI. The game ends when all units of any army have died or pre-defined timestep limit is reached. The goal of the allied agents is to learn policy that can maximize the win rate the game. Therefore, the agents need to learn a sequence of actions to collaborate with other allies to defeat the enemy forces. One example of such collaboration involves mastering kiting skills, where enemy units pursue and keep enough distance from the enemies to reduce damage. We use the SC2.4.10 version of SC2. Note that the performance is not comparable between versions.
agents form formations according to their armor types, forcing
SMACv2 is an upgraded version of SMAC that uses the same APIs as SMAC to control the game units. Unlike SMAC, SMACv2 introduces stochasticity in the StarCraft environment. Firstly, the initial positions of units are random, challenging the agent to learn to defeat the enemies from different angles. Secondly, the allied agents of each scenario have random unit types instead of pre-specified types. These beneficial changes provide additional challenges for MARL algorithms, as they must learn generalizable and robust policies to improve win rates. The average returns of compared algorithms in Pac-Men, SMAC, and SMACv2 are listed in Table
E Evaluations of CTR in scenarios requiring homogeneous behavior
Although behavioral diversity is crucial in the multi-agent environment, agents may sometimes find beneficial to act similarly in straightforward situations. For instance, allied agents might employ the
14

---

Table Average returns of compared algorithms in Pac-Men, SMAC, and SMACv2. + denotes the standard deviation over five random seeds.
SMACV2
Method
Pac-Men
365c
2e vv 64gg
75x
6b. vs.
corridor
365, w. Jehe
terran 5.v.
protoss 5. vv.
zerg 5 v
QMIX
021=0004
0.72=11.13
0.85:0.88
0,17=0.02
0.234:003
0.57:00077
0.36+00122
0.68:00.0
0.53+:0.5
0.41:1.044
MAPPO
0.49=0.33
0.81=0.05
0.8300044
0,52-0.06
0.53+0003
0.6240.05
0,57+0008
0.52+0004
0:4700033
0.37=0.08
MAVEN
0.32+0.05
0.51-0.21
0.72+0.06
0.00-0.00
0.17+004
0.36+0.08
0.18+0.15
0.58+0004
0.31+00.5
0.29-0.01
FOI
041:0.05
0.87=0.07
0.8300022
0,37=0.03
0.06+0033
025510.11
0.4200.13
0.65+0005
0:42+0003
0.47:0.44
QTRAN
0.28:0.08
0.21:0019
0.75:0005
0.00:0000
0.001:0022
0.08:0007
0.00100011
0,42:0.02
0,40100.4
0.25
SCIS
0
0.76:000
0.57:0009
0.21=10033
0.030:011
0.56200066
110000000
0,52-000
0.47:0055
0.38=00.4
LIPO
043+0012
0.71=0.0
0.76+0.22
0,59=0.04
0,36+:006
0.27+0:33
0.21+0003
0,4+0000
0/4++003
0.57-000
FoX
0.39-0:3
0.74-0.0
0.440055
0.56-0.03
044+0005
0.52+0:04
0A340004
0.54+0.01
0.56+0022
0.49-0.02
CIR-QMIX
0.78.0004
0,95-003
0.57.0005
0.82.0.05
0,7910005
082:0.07
0,8510.06
0,8310003
0,7510005
0.73-0083
CIR-MAPPO
0,75:003
0,92:0.04
0.95
0,7820000
0.712:00
0.76:0005
0.311004
0.7940.04
0.091003
0.65-000
Table 2: Performance of our method and QMIX in homogeneous scenarios.
Method
vs 6m
8m vs 9m
10m vs 11m
CTR+QMIX 0.95+0.03
0.93+00.0
0,94+0.02
0.91 0.04
QMIX 0.87+00.3
0.65
0,58+00.5
0.43++0.04
same tactic to simultaneously fire at an enemy to quickly eliminate it. To demonstrate the effectiveness of our method in learning such behaviors, we test it in four homogeneous SMAC scenarios that benefit from the focus fire trick. The results, shown in Table indicate that our method consistently outperforms QMIX across all scenarios. The outperformance of our method demonstrates that supports, rather than prevents, homogeneous behaviors when these result in higher environmental rewards. Moreover, our method is more efficient to search these optimal cooperative behaviors due to sufficient exploration.
F
Scalability
In MARL, as the number of agents increases, the state-action space grows exponentially, challenging agents search optimal collaborative policies. Many algorithms lack efficient exploration and may not scale well with a large number of agents. To demonstrate the scalability of our method, we test in four SMACv2 scenarios with an increasing number of agents: terran 5 vs 5, terran 10 vs_ 10. terran 15_ 1___v_______________________11 vs_1,, and terran_ 20 vs 2s_ The results are shown in Table 3 The performance of QMIX decreases significantly as the number of agents increases. We believe this is because QMIX suffers from poor exploration of the state-action space. With the help of our method, QMIX achieves essentially better performance and exhibits robust scalability, demonstrating that learning distinguishable trajectory representations for agents to make action decisions leads to efficient exploration.
G Training Details and Hyperparameters
Our proposed CTR model consists of an encoder and autoregressive model. We adopt two stacked resnet blocks with a hidden size of 64 followed by batch normalization for the encoder and a GRU unit for the autoregressive model. In the agent utility network of QMIX, the CTR model encodes a trajectory to a latent representation space, which is then input to the fully connected output layer to calculate the per-agent utilities. Similar to QMIX, in the actor network of MAPPO, the learned trajectory representations are input to a fully connected output layer followed by a softmax function
Table Performance of our method and QMIX in scenarios of SMACv2 with different number of agents
Method
terran_5_ 5__
terran 10_ vs 10
terran vs_
terran terran_20 20_____________________ vi__vs_200 20
CTR QMIX
0.804
0.87 10.02
0.83 10.03
0.82 10.03
QMIX
0.68-10.03
0.39
0.24 -0.06
0.11.10.05
15

---

to output the distributions over actions. Moreover, we introduce learnable vector for the identity representation that has the same dimensions as trajectory representations.
We use the same policy network architecture for other baselines as in our method to guarantee a fair comparison. In both SMAC and SMACv2, the target networks are updated via hard updates every 200 episodes. In Pac-Men, the target networks use soft updates at a momentum rate of 0.01. We set the evaluation interval 10K steps followed by 32 test episodes. We run all methods for 5 million steps. The hyperparameters of CTR and baseline algorithms in Pac-Men, SMAC, and SMACv2 are listed in Table To ensure a fair comparison, we set the same values for common hyperparameters across different methods in each multi-agent environment. Additionally, all methods adopt the parameter-sharing technique accelerate training speed. For generality, we report the mean and standard deviation of the experimental results tested with five random seeds. We set the replay buffer size to 5K. We implement our method with NumPy and Py PyTorch. All experiments are performed using NVIDIA GeForce RTX 4090 GPUs.
Table 4: Hyperparameters
Puc-Men 64
SMAC
SMACv2
hidden dimension leurning rate
128
0.0003
0.005
optimizer
Adam
target update
0.01(soft)
200(hard)
batch size
32
64
0.1 for 3s5z, 2c, vs. 642g. 8m. 5m, 6m, 8m, vs. 9m, and to__vs__llll llm.
for CTR
0.02
0.02 for 7sz, 6h_ 87, corridor, and 3s52_vs, v________622
0.004
0.1 for 3s52, 2e vs. 642g, 8m, 5m, ys. _s__ssn__ 6m, 8m, ys. 9m, and 10m_vs, 1mm_
CTR-MAPPO
0.05 for 7sz, 6h_ vs_ 8z, corridor, and 3s52 vs_ 3s63
0.02
200,000 3s5z, 2c_ vy 642g, 8m, 5m_ 6m, 8m, 9m, and 10m_vs_
epsilon anneal time
200,000
500,000
500,000 for 7s7, 6h, ys. 82, corridor, and 3s57_ys vs. _8_3s6
H
Visualization
We additionally provide visualized t-SNE plots in Figure g Figure 9 and Figure 10 intuitively compare the trajectory representations learned by QMIX, EOI, SCDS, and CTR in the super hard corridor, 6h_vs_88 sz_ and $s52_vs_3só6 scenarios. The trajectory representations of different agents learned by CTR ultimately stay away from each other and become distinguishable while those learned by QMIX are mixed. Moreover, although EOI and SCDS also encourage multi-agent diversity, the distinguishability between trajectory representations of different agents learned by them less pronounced compared to CTR.
To intuitively demonstrate the diverse policies learned by CTR+QMIX, present some visualization examples of the diverse policies emerging in 6h_ 6h__v_______________________ee _s__s_________ corridor, and 3s5z_vs vs sa___v_sss__ __s62 from initial to final in Figure [1 Green and red shadows represent agents and enemies, respectively. Green and red arrows represent the moving direction of agents and enemies, respectively. For example, in the 6h_vs. fh__s_8_ _z scenario, one agent first leaves the team separately incur the attention of most enemies. The agent keeps kiting the enemies and draws the fire to cover other agents. Then other agents move in different directions and attack the few remaining enemies. If all the agents take similar policies and consistently move toward the enemies, they will be killed immediately. These diverse policies can also be observed in the other two scenarios: corridor and 3s5z_ vs 3s6z. These examples suggest that learning diverse policies by distinguishing trajectory representations of different agents finally enables the agents to cooperatively defeat the enemies.
16

---

phase
phase 2
phase 3
phase
QMI
BOI
SCDS
CTR
Figure 8: T-SNE plots of trajectory representations of different agents learned by CTR and baselines, respectively, that emerge the corridor scenario, initial (left) to final (right). Each color represents the trajectory representations of agent.
phuse
phase 2
phuse
phase
QMIX
EOI
SCDS
CTR
Figure 9: T-SNE plots trajectory representations of different agents learned by CTR+QMIX and baselines, respectively, that emerge in the 6h_ v___s__z s2_88 scenario, initial (left) to final (right). Each color represents the trajectory representations of an agent.

---

phase 1
phase 2
phase
phase
QMIX
BO1
SCDS
Figure 10: T-SNE plots of trajectory representations of different agents learned by CTR+QMIX and baselines, respectively, that emerge in the 3s57_v8_3nt scenario, initial (left) to final (right). Each color represents the trajectory representations of an agent.
18

---

6h vs vs_82 8z
corridor
3s5z vs 3s6z
initial
final
Figure 11: Visualization examples of diverse policies emerging in 6h, vs 8z (top), corridor (medium), and 3s5z_wi _s_36z (bottom) from initial (left) final (right). Green and red shadows represent agents and enemies, respectively. Green and red arrows represent the moving directions of agents and enemies, respectively.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Our code can be found in the uploaded supplemental material. We also
provide the network architectures and hyperparameters Appendix and the pseudocode
in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Please refer to the uploaded supplemental material for our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.c./ cttpsscccpbbbcc//cceecceeccccccc hoaaa//gguudeee/eeeeebiiiisssss /ac////odebiissseeeeeeee www.sssiiiiiiiiiocooooooiliiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Please refer to Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the mean and standard deviation of the experimental results tested with five random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yss]
Justification: Please refer to Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpg://eerrris.cc/pbbb////iiii
Answer: [Yes].
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work provides novel exploration method for MARL methods and has no social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "eqxiRzekb0", "reviewer": "Reviewer_Urvi", "summary": "The authors propose a method to maximize mutual information between trajectory representations of different agents in the multi-agent reinforcement learning setting. Rather than comparing policies on a state-by-state basis, they instead learn a representation of the entire trajectory using sequence models and contrastive learning. They evaluate their method in a range of challenging multi-agent RL games and show that their method accelerates learning and achieves higher final rewards than baselines.", "questions": "N/A", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and it is easy to understand the main concepts. The experiments show strong advantages of the method in multiple settings, providing strong evidence that the method is making a significant difference. In addition to the strong comparisons to baselines, the authors also provide insightful visualizations and ablate over their design choices.", "weaknesses": "In theory, this method can be applied to other algorithms besides QMIX. It could be used to learn distinguishable skills in unsupervised skill learning, or in continuous control settings as well. There is no need to limit the thinking and future work to this setting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to review. We agree with your comments. Our method can be integrated with a variety of MARL methods based on the CTDE framework including value-based and policy-based methods. Since our method simply incorporates the trajectory encoder with the decentralized policy network to learn distinguishable trajectory representations, it is irrelevant to other network structures of MARL methods. Moreover, our method can be extended to learn diverse skills in the field of unsupervised skill learning. Prior works focus on maximizing the mutual information between states and latent variables to learn diverse skills, which may lead to overfitting as we discussed in our paper. Similar to our work, learning distinguishable representations for different states may be more efficient in learning diverse skills. We reserve this for our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XFvEQ8DBDh", "reviewer": "Reviewer_ncsG", "summary": "This paper introduces a novel approach to learning in multi-agent reinforcement learning (MARL) environments by focusing on distinguishable trajectory representations to encourage agent diversity. The proposed method, termed Contrastive Trajectory Representation (CTR), leverages a contrastive learning loss to effectively differentiate between agents' trajectory representations without necessitating fixed agent identities. This approach is designed to combat the common issue in MARL where agents converge to similar behaviors, thus limiting the overall system's adaptability and efficiency.\n\nThe paper provides a comprehensive evaluation of CTR through a series of experiments conducted in both grid-world environments like Pac-Men and more complex settings such as the StarCraft Multi-Agent Challenge (SMAC). The results demonstrate that CTR significantly outperforms existing state-of-the-art methods by facilitating more robust and diverse agent behaviors, leading to improved exploration and performance across various scenarios. This is achieved by mapping trajectories onto a contrastive representation hypersphere, which encourages more efficient exploration and prevents the overfitting associated with mutual information-based methods. The study underscores the effectiveness of CTR in promoting diverse and adaptive strategies in MARL settings, offering a scalable solution that enhances both the learning efficiency and strategic capabilities of multi-agent systems.", "questions": "Q1: I'm wondering whether environments like football games can get benefit as you mentioned. Is it possible to make such experiments?\n[1]TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations\n\nQ2: Could you elaborate on the mathematical or theoretical foundations that justify the use of contrastive learning specifically in MARL environments? How does this approach theoretically ensure improved agent diversity compared to traditional methods?\n\nQ3: How does the model perform over extended periods of interaction? Is there evidence of long-term stability in the agents' behaviors, or do they exhibit significant variance over time?", "limitations": "Probably no obvious limitations", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces an innovative approach within the field of multi-agent reinforcement learning (MARL) by focusing on the creation of distinguishable trajectory representations using contrastive learning. This method diverges from traditional reliance on mutual information maximization between agents' identities and trajectories, presenting a significant shift in how agent diversity is cultivated in MARL environments. The originality of this work is evident as it creatively combines the principles of contrastive learning, commonly employed in single-agent domains or supervised learning tasks, with the complexities of MARL, thereby addressing the challenge of agent homogenization without fixed identity assignments.\n\nThe quality of the research is high, evidenced by rigorous experimental design and thorough validation across several benchmarks, including grid-world scenarios and the StarCraft Multi-Agent Challenge (SMAC). The experiments are well-structured, with clear comparisons to baseline models and detailed discussions of the results. Furthermore, the authors have provided a robust statistical analysis to back their claims, reinforcing the reliability of their findings. The use of well-recognized MARL environments for testing also supports the methodological rigor of the study.\n\nThe paper is well-written, with a clear exposition of the concepts and methodology. The authors have successfully communicated complex ideas in an accessible manner, making the paper understandable to both experts in the field and readers with a more general background in machine learning.", "weaknesses": "W1: Dependency on Hyperparameters: The performance of the CTR model, as with many learning models, appears to be  dependent on the fine-tuning of hyperparameters, particularly the weight of the contrastive loss component. This dependency can introduce challenges in scenarios where the optimal hyperparameter settings are not apparent or vary significantly between environments. The paper could enhance its contribution by providing a more detailed analysis or guidelines on how to select or adapt these parameters effectively across different settings.\n\nW2 Scalability to Larger Agent Pools: The experiments are somewhat limited in scale, primarily focusing on scenarios with a small to moderate number of agents. The scalability of the approach to environments with large numbers of agents remains untested. It would be beneficial to investigate how the method performs as the number of agents increases significantly, which is a common challenge in real-world applications of MARL.\n\nW3 Handling Non-Stationarity: In MARL environments, the non-stationarity issue arises when the policies of other agents change, which can affect the learning and stability of a given agent’s policy. The paper perhaps does not address how the proposed method copes with the non-stationarity of the environment, which is crucial for ensuring robustness and reliability in dynamically changing scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and the valuable insights you provided during the review process. We clarify your concerns and problems below:\n\nW1: Dependency ... different settings.\n\nThe values for the weight of the contrastive loss in different scenarios are listed in Table 4 in our paper. To investigate the effect of the weight of the contrastive loss component in different scenarios, we use different weight values and test them in the easy scenario 3s5z and the super hard scenario corridor. The results are shown in Table 1 in the attachment of the global response. We note that smaller values for the weight of the contrastive loss can typically lead to better empirical results in the super hard scenario, while larger values are welcome in the easy scenario. This may be attributed to the environmental rewards achieved by agents in the super hard scenario being less than those achieved in the easy scenario. Therefore, less intrinsic rewards are desirable in the super hard scenario. Moreover, our method is not very sensitive to the values of the weight. Sub-optimal weights do not result in a significant performance drop even in the super hard scenario. \n\nW2: Scalability ... of MARL.\n\nWe evaluated the scalability of our method in different scenarios of SMACv2 with an increasing number of agents from 5 to 20. Our method scales well as the number of agents increases. We further test the scalability of our method in a large-scale multi-agent benchmark IMP-MARL [1]. Specifically, we use the environment uncorrelated k-out-of-n system from IMP-MARL with the number of agents varying from 10 to 100. We present normalised relative rewards achieved by our method and QMIX, respectively, in Table 2 in the attachment of the global response. Our method substantially achieves better performance than QMIX and scales well with a varying number of agents. A larger number of agents results in more negative samples that are desired to improve the discrimination of contrastive representation learning. \n\nW3: Handling ... changing scenarios.\n\nIn our method, we use an autoregressive model to encode the trajectory representations for action decision-making, which compresses the historical trajectory information. In multi-agent settings, the historical trajectory information can typically alleviate the non-stationary issue. Moreover, the InfoNCE loss learns trajectory representations by considering the trajectory representations of all agents, thus leading to more stable and robust policy learning.\n\nQ1: I'm ... Demonstrations\n\nYes. We further test our method on two challenging Google Research Football (GRF) offensive scenarios academy\\_3\\_vs\\_1\\_with\\_keeper and academy\\_counter\\_attack\\_hard. We present the results in Table 3 in the attachment of the global response. We note that our method achieves significant performance improvement over QMIX, demonstrating the effectiveness of our method in encouraging efficient exploration.\n\nQ2: Could ... methods?\n\nThe learning rule of the MARL method theoretically guarantees the learning of optimal policies. Our method does not break the learning rule of the integrated MARL method since its implementation is within the decentralized policy network. For example, our method does not break the monotonicity constraint of QMIX imposed on the mixing network. This constraint ensures that the gradient of $Q_{tot}$ with respect to any individual $Q_a$ is non-negative: $\\frac{\\partial Q_{\\text{tot}}}{\\partial Q_i} \\geq 0$. This guarantees that improvements in an agent’s policy that increase its own Q-value will not decrease the total Q-value. The monotonicity constraint is enforced by architecting the mixing network so that it uses non-negative weights. However, our method only introduces an auxiliary gradient derived by the contrastive learning loss to the individual agent utility network to learn distinguishable trajectory representations, that would not affect the value decomposition controlled by the mixing network. \n\nThe diversity measures the distance between the trajectories of different agents. We may associate such distances with mutual dependence from the perspective of information theory. We first model a density ratio as in CPC[2]: $\\frac{p(c_{t}^a | d^a)}{p(c_{t}^a)}$ that preserves the mutual information between the trajectory representation $c_{t}^a$ and identity representation $d^a$. Then, we let the similarity $f\\left(c_t, d\\right)= \\exp \\left({c_t^a}^T d^a \\right) \\in \\mathbb{R}$ be proportional to the ratio: $ f\\left(c_t, d\\right) \\propto \\frac{p(c_{t}^a | d^a)}{p(c_{t}^a)}$, which can be achieved by minimizing the InfoNCE loss shown in Equation 3 as theoretically proved by CPC. This demonstrates that the InfoNCE loss models $\\frac{p(c_{t}^a | d^a)}{p(c_{t}^a)}$ instead of $p(c_{t}^a | d^a)$. Previous methods typically aim to maximize the probability $p(c_{t}^a | d^a)$ to achieve the maximum of the mutual information. We notice that $\\frac{p(c_{t}^a | d^a)}{p(c_{t}^a)} > p(c_{t}^a | d^a)$. Thus, our method can be more efficient in distinguishing the trajectory representations of different agents.\n\nQ3: How ... over time?\n\nWe typically set a large time-step limit for each episode in test environments. For example, we run 100 steps per episode in Pac-Men. We provide visitation heatmaps of our method in Pac-Men in Figure 2d. We also present the visualization examples of policies learned by our method from initial to final in three super hard scenarios of SMAC in Figure 11. These results demonstrate that our method can continuously encourage agents to explore and exhibit efficient diverse policies over time.\n\nWe await your further suggestions.\n\n[1]Leroy, Pascal, et al. \"IMP-MARL: a suite of environments for large-scale infrastructure management planning via MARL.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2]Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. \"Representation learning with contrastive predictive coding.\" arXiv preprint arXiv:1807.03748 (2018).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mDZj3bjEDF", "reviewer": "Reviewer_GQbC", "summary": "The paper proposes using a contrastive trajectory representation to improve diversity and exploration in decentralized multi-agent reinforcement learning. Experimental evaluations show the positive impact in a small-scale environment and improved performance in various SMAC scenarios.", "questions": "If agent identities are randomly chosen, what effect does the distance or mutual information between the trajectory representation and this random vector have? \n\nHow does your approach improve limited exploration?", "limitations": "Limitations are briefly discussed but can be extended (see above).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors provide a varied and insightful evaluation, showing the positive impact of their approach, which aims to improve MARL diversity and explorative capabilities, which constitutes a relevant problem in MARL by integrating constrastive learning.", "weaknesses": "- The authors should state more clearly their use of parameter sharing in a decentralized learning scenario to avoid the confusion of only a single policy with shared parameters being learned.\n- The proposed method and its intentions could be explained in greater detail (e.g., the consequences and effect of Eq. (4) or the difference between the contrastive learning loss and the InfoNCE loss in Eq. (2)).\n- The proposed architecture and setting are very similar to AERIAL (Phan et al., 2023), so they should be compared. Overall, the use of mutual information to foster diverse policy or trajectory representations has been used in various approaches and does not seem novel (which generally is no issue to me but should be discussed).\n- Conclusion and Limitations fall pretty short and could be extended, e.g., by potential issues of increasingly complex learning architectures. \n\nMinor Comments: \n\n- 2.1 could also be a paragraph, given there is no further subsection in 2.\n- in 2.1: Does $U$ refer to the joint action or the joint action space?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly value your expertise and the effort you put into reviewing our paper. Here are the responses to your concerns:\n\nWeakness 1:The .. being learned.\n\nIn our paper, we discuss a decentralized learning scenario where agents share the same policy network parameters but learn different decentralized policies. Thank you for your thoughtful suggestions. We clarified it in our paper.\n\nWeakness 2: The ... in Eq. (2)).\n\nWe discuss the motivation, the consequences, and the effect of using Equation 4 in Section 3.3. We developed the multi-agent contrastive learning loss shown in Equation 4 because the small size of dataset $C$, equal to the number of agents, results in a wider gap between the true mutual information objective and the contrastive learning lower bound, potentially hurting performance. So we propose the multi-agent contrastive learning loss, which actively increases the number of negative samples from $O(|\\mathcal{C}|)$ to $O\\left(|\\mathcal{C}|^2\\right)$, thus leading to better empirical results. We clarified this further in our paper. Contrastive learning loss is a broad category of loss functions that includes the InfoNCE loss. We typically use contrastive learning loss to demonstrate the employment of contrastive learning.\n\nWeakness 3: The ... be discussed).\n\nThe only similarity between AERIAL proposed in [1] and our method is that we all use the hidden states or trajectory representations output by the RNNs. However, their applications are entirely different. AERIAL processes the hidden states using a simplified transformer in order to automatically learn the latent dependencies over hidden states through self-attention, and then feed the output of the transformer to the mixing network instead of the true state like QMIX. Our method has a totally different objective that encourages multi-agent diversity through learning distinguishable trajectory representations via contrastive learning. \n\nThe idea of learning distinguishable trajectory representations to encourage multi-agent diversity is novel and has not been proposed yet. Moreover, although many previous methods employ the mutual information objective to encourage exploration to improve the cooperation among agents, however, they typically resort to the variational lower bound to solve the mutual information objective, which may lead to serious overfitting as we discussed in the fourth paragraph of Section 1 and Section A in the appendix. We solve this limitation of the mutual information-based methods by using contrastive learning to learn to distinguish trajectory representations. The empirical results demonstrate that our method can lead to better performance.\n\nWeakness 4: Conclusion ... architectures.\n\nWe add some limitations as follows: As our method needs to contrast all agents' trajectories, this necessitates that the training process should be centralized so that the trajectories of all agents can be collected. Thus, our method cannot be applied to fully decentralized MARL methods. Moreover, in contrastive learning, by collecting many negative samples, the model is challenged to distinguish the positive pair from a larger pool of negatives. This helps the model learn more robust and discriminative features. Although we developed multi-agent contrastive learning loss to increase the number of negative samples, however, the number of negative samples is still limited if the total number of agents in multi-agent environments is very small. For our future work, we may develop efficient methods to augment existing trajectory samples to increase the number of samples.\n\nWe add some conclusions as follows: Our simple yet effective method demonstrates the importance of representation learning in promoting efficient exploration. Better representation learning can typically encourage the learning optimal policies. Despite the emergence of multi-agent diversity, we also note the need for homogeneous behaviors. Although our method would not impede the learning of homogeneous behaviors that can lead to more environmental rewards, how to control diversity automatically can be an interesting direction for our future work.\n\n$U$ is a set of agents’ actions as we discussed in Section 2.1.\n\nQ1: If agent ... have?\n\nThe choice of agent identities would not affect the trajectory representation learning. In our paper, we use the learnable identity representations to represent the agent identities, which are randomly initialized are the beginning of the training process. From the InfoNCE loss shown in Equation 3, we note that by minimizing the InfoNCE loss, the identity representations stay close to their corresponding trajectory representations while being far away from other trajectory representations. Thus, the distance between the trajectory representation and the identity representation is irrelevant to the choice of agent identities.\n\n\nQ2: How does your approach improve limited exploration?\n\nDifferent from the previous mutual information-based methods that learn the mutual dependence between trajectories and fixed agent identities, e.g., fixed one-hot vectors, to encourage multi-agent diversity, which may lead to serious overfitting, our method learns distinguishable trajectory representations by minimizing the contrastive learning loss. The learned trajectory representations do not depend on fixed agent identities, thus leading to more efficient exploration. Moreover, from the T-SNE plots shown in Figure 8-10, we note that previous mutual information-based methods such as SCDS and EOI do not learn distinguishable trajectory representations, thereby resulting in inefficient exploration. Compared to previous methods, our method successfully learns distinguishable trajectory representations with large distances. \n\nWe hope to hear from you soon and thank you again for your review.\n\n[1] Phan, Thomy, et al. \"Attention-based recurrence for multi-agent reinforcement learning under stochastic partial observability.\" ICML. PMLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GVmsxqJC4q", "reviewer": "Reviewer_mufx", "summary": "This paper proposes a novel Contrastive Trajectory Representation (CTR) method based on learning distinguishable trajectory\n representations to encourage multi-agent diversity. It introduces contrastive learning to maximize  the mutual information between the trajectory representations and learnable identity  representations of different agents.", "questions": "- What the differences between the proposed method and these methods[1][2].  \n- How to get learnable vector $d$ for each agent?\n- It encouraged for authors to demonstrate the rewards and loss curves of proposed method.", "limitations": "- Lack of implementation details for proposed method. Please make the source codes be available.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper propose a novel strategy to make multi-agent diversity by learning distinguishable trajectory representations.\n- CTR maps the trajectory of an agent into a latent trajectory representation space by an encoder and an autoregressive model.\n- The proposed distinguishable trajectory representations do not rely on fixed agent identities.", "weaknesses": "- The paper lacks a detailed description of the identity representation, only saying that it is a  learnable vector. \n- The paper cited the latest developments in reinforcement learning, does not compare their performance [1]. Newer published works in recent 3 years should get included [2].\n- The quality of the figures needs to be improved such Fig. 2 and 3. The fonts in figures seem to be strange.\nReference.\n[1] Contrastive Identity-Aware Learning for Multi-Agent Value Decomposition, AAAI, 2023.\n[2] Learning Multi-Agent Communication with Contrastive Learning, ICLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and for providing us with detailed and helpful feedback. We response to your concerns below:\n\nWeakness 1: The paper lacks a detailed description of the identity representation, only saying that it is a learnable vector.\n\nPrevious mutual information-based methods typically use fixed agent identities, e.g., one-hot vectors. However, this may lead to serious overfitting as we discussed in the fourth paragraph of Section 1 in our paper. To solve the problem, we instead use learnable identity representations for agents to represent their identities. The identity representations can be trained by minimizing the contrastive learning loss to linearly classify the trajectory representations of different agents. As a result, the identity representations stay close to their corresponding trajectory representations while being far away from other trajectory representations, leading to distinguishability among trajectory representations. We clarified it in our paper.\n\nWeakness 2: The paper cited the latest developments in reinforcement learning, does not compare their performance [1]. Newer published works in recent 3 years should get included [2].\n\nWe conduct a comparison of our method with the CIA method introduced in [1], in Pac-Men and the three super hard scenarios of SMAC. To ensure a fair comparison, we maintain consistent hyperparameters and policy network structures for both methods. The results are presented in Table 4 in the attachment of the global response. Our method consistently outperforms CIA across all scenarios. Specifically, in Pac-Men, CIA learns suboptimal policies and does not identify the top room with the longest path, as illustrated in Figure 1 in the attachment of the global response. Our method enhances multi-agent diversity more effectively, enabling agents to explore different rooms to collect dots, as shown in Figure 2d in our paper.\n\nWe included [2] in our related works. The authors in [2] aim to help agents to learn to communicate for better coordination. To achieve this goal, they uses contrastive learning to maximize the mutual information between messages of a given trajectory, which leads the messages from the same state to be more similar to each other than to those from distant states or other trajectories. We also use the contrastive learning loss. However, we use it to learn distinguishable trajectory representations to encourage multi-agent diversity and further improve it to achieve better empirical results.\n\nWeakness 3: The quality of the figures needs to be improved such Fig. 2 and 3. The fonts in figures seem to be strange. Reference.\n\nWe resized the fonts in the figures to make them clearer.\n\nQ1: What the differences between the proposed method and these methods[1][2].\n\nCIA proposed in [1] realizes credit-level distinguishability using contrastive learning to distinguish the agents' temporal credits that are represented by the agent's gradient-based attributions derived from the TD loss of QMIX. CACL proposed in [2] learns to communicate using contrastive learning to maximize the mutual information between messages of a given trajectory. However, we learn distinguishable trajectory representations from similar trajectory samples using contrastive learning to encourage multi-agent diversity. Further, CIA and CACL directly use the original InfoNCE loss proposed in [3]. We improve the original InfoNCE loss in multi-agent settings, as shown in Equation 4, by increasing the number of negatives, leading to better empirical results.\n\nQ2: How to get learnable vector $d$ for each agent?\n\nAt the beginning of the training process, we randomly initialize alearnable identity representation $d^a \\in \\mathbb{R}^H$ for each agent by building a LearnableDictionary class that wraps a torch.nn.Parameter function in Pytorch to generate learnable tensors.\n\nQ3: It encouraged for authors to demonstrate the rewards and loss curves of proposed method.\n\nWe provide episode returns (the sum of rewards achieved by agents during an episode) and loss curves in Figure 2 and Figure 3 in the attachment of the global response, respectively.\n\nLimitations: Lack of implementation details for proposed method. Please make the source codes be available.\n\nWe provide the PyTorch-style pseudocode for our method and training details in Section C and Section G, respectively. Our source codes, included as supplemental materials, were uploaded when we submitted this paper.\n\n[1] Liu, Shunyu, et al. \"Contrastive identity-aware learning for multi-agent value decomposition.\" AAAI, 2023.\n\n[2] Lo, Yat Long, et al. \"Learning Multi-Agent Communication with Contrastive Learning.\" ICLR, 2024.\n\n[3] Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. \"Representation learning with contrastive predictive coding.\" arXiv preprint arXiv:1807.03748 (2018).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SxQ54iuRpy", "reviewer": "Reviewer_u9xg", "summary": "This paper presents a contrastive approach for learning diverse policies in multi-agent reinforcement learning (MARL). It maximizes the mutual information between trajectory representations and identity representations, formulating this maximization as an InfoNCE loss function. The methodology is tested across several scenarios, including Pac-Men, SMAC, and SMACv2.", "questions": "1. Do agents share the same parameters of the policy network in the proposed method?\n2. How is “diversity” defined in the context of this paper? Does it imply that different agents have different policies?\n3. How does the proposed CTR method compare to the existing CIA method in terms of performance and policy diversity?", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper derives and theoretically substantiates the InfoNCE loss function.\n- The experimental results demonstrate that the proposed method surpasses the performance of existing baseline methods in tested scenarios.", "weaknesses": "This paper has some shortcomings in technical novelty and clarity:\n\n1. **Technical Novelty:** The application of InfoNCE loss to representation learning in this context appears incremental, bearing close resemblance to the previously established CIA method. The distinction between the proposed method and existing techniques is not adequately emphasized, which raises concerns about the paper’s original contribution.\n2. **Unclear Claims About Diversity:** The paper does not specify which MARL framework supports the proposed method. Assuming a shared policy network parameter setting similar to QMIX, the rationale behind diverse policies under such a configuration is questionable. The shared parameters typically lead to the same policies, contradicting the stated objective of enhancing policy diversity among multi-agents.\n3. **Evaluation of Diversity:** Although diversity is a major goal, the paper lacks specific metrics or evaluations of diversity in the experimental setup. The focus is task performance, but short in directly assessing whether the proposed method actually enhances the multi-agent diversity.\n\n**Minor Issues:**\n- The paper incorrectly refers to GRU as an autoregressive model. GRU is a recurrent neural network.\n- Figures 3 and 4 are in wrong aspect ratios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We clarify your concerns and problems below:\n\nWeakness 1: Technical Novelty ... contribution.\n\nWe discussed the CIA method in the related works in our original paper. The differences between our method and CIA are shown below: \n\nFirst, our main idea of learning distinguishable trajectory representations to encourage multi-agent diversity is entirely different from the CIA which distinguishes temporal credits of different agents. We learn a trajectory encoder using contrastive learning to output distinguishable trajectory representations, while CIA treats the temporal gradients of agents derived from the TD loss of QMIX as credits and uses the InfoNCE loss to distinguish them to realize identity-aware credit assignment. Note that CIA can only be used in methods based on the value-based value-decomposition framework. However, our method can also be used in policy-based methods such as MAPPO.\n\nSecond, CIA and our method are all inspired by the InfoNCE loss proposed in [1]. The use of the InfoNCE loss in the field of MARL can also be found in many other works such as [1], [2], and [3]. CIA directly applies the InfoNCE loss to distinguish temporal credits of different agents. However, most importantly, we do not directly apply the InfoNCE loss but a novel multi-agent contrastive learning loss shown in Equation 4. This is because we need to solve the limitation of applying the InfoNCE loss to learn trajectory representations in multi-agent settings that the smaller dataset size used in our method may induce a larger gap between the true mutual information objective and the contrastive learning lower bound, which hurts the agent performance. So, we improve the InfoNCE loss by actively increasing the number of negative samples in the denominator, which leads to better empirical results. \n\nFor the reasons stated above, our method is different from CIA.\n\nWeakness 2: Unclear Claims ... multi-agents.\n\nIn our method, all agents share the same policy network parameters. The claim that learning distinguishable trajectory representations for agents, sharing the policy network parameters, to learn diverse behaviors is reasonable. First, since MARL methods such as QMIX typically adopt techniques such as $\\epsilon$-greedy or entropy regularizers to introduce randomnesses into action selections during the exploration phase, they may not learn the exact same policies. The agents sharing policy network parameters are likely to achieve similar trajectories as we discussed in the third paragraph of Section 1 in our paper. Agents trained by QMIX typically make action decisions based on its historical trajectories. The intuition behind our method is that although similar historical trajectories among agents can typically lead to similar actions when using the shared policy network to make action decisions, we can instead learn distinguishable trajectory representations from similar historical trajectories for action decision making by training the policy network (CTR is a part of the policy network) towards minimizing the contrastive learning loss. This means that although the input historical trajectories are similar, the shared policy network learns distinguishable trajectory representations, thus leading to diverse policies. Contrastive learning gives us a chance to distinguish trajectories of different agents in a representation space. \n\n\nWeakness 3: Evaluation of Diversity ... diversity.\n\n\nThere is currently a lack of commonly used specific metrics to measure diversity among agents. So, we demonstrate the agent diversity as in previous works. To intuitively demonstrate the effectiveness of our method in encouraging multi-agent diversity, we provided some agent's visitation heatmaps, T-SNE plots of trajectory representations, and visualization examples of diverse policies in Figure 2 and Figure 7-11 in our paper.\n\n\nQ1: Do agents share the same parameters of the policy network in the proposed method?\n\nPlease refer to Weakness 2.\n\nQ2: How is “diversity” defined in the context of this paper? Does it imply that different agents have different policies?\n\nMulti-agent diversity refers to the diversity among agents, i.e., the differences between policies of different agents. We discussed the “diversity” in the third paragraph of the Section 1. \n\nQ3: How ... policy diversity?\n\nWe compare our method with CIA in Pac-Men and the three super hard scenarios of SMAC used in our paper. The hyperparameters and policy network structures are kept consistent across the two methods to ensure a fair comparison. We present the experimental results in Table 4 in the attachment of the global response. Compared to CIA, our method maintains its outperformance across all scenarios. In Pac-Men, CIA fails to discover the top room with the longest path as shown in Figure 1 in the attachment of the global response, thus leading to sub-optimal performance. Our method is more efficient in encouraging multi-agent diversity, which enables agents to go to different rooms to collect dots as demonstrated in Figure 2d in our paper.\n\nWe hope to receive your feedback soon and greatly appreciate the time you have taken to review our paper.\n\n[1] Hu, Zican, et al. \"Attention-guided contrastive role representations for multi-agent reinforcement learning.\" arXiv preprint arXiv:2312.04819 (2023).\n\n[2] Zeng, Weihao, et al. \"Multi-Agent Transfer Learning via Temporal Contrastive Learning.\" arXiv preprint arXiv:2406.01377 (2024).\n\n[3] Lo, Yat Long, et al. \"Learning Multi-Agent Communication with Contrastive Learning.\" The Twelfth International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We greatly appreciate the time you have taken to review our paper. The PDF attachment presents the Tables and Figures referenced in the responses. We hope to receive your feedback soon so that we can further improve our paper.

Meta-review: The paper proposes a method for improving behavior diversity in MARL by having the agents learn different trajectory representations using a contrastive loss-based approach CTR while using a shared policy network. The experimental results show that CTR significantly improves the performance of the underlying MARL algorithm. While contrastive learning has been widely for inducing trajectory representations, the specific approach in this paper is novel. Given the performance boost it affords to MARL, the metareviewer recommends this paper for acceptance.
Decision: Accept (poster)