Toward Efficient Multi-Agent Exploration With Trajectory Entropy Maximization
Authors: Tianxu Li
Kun Zhu
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-Agent Reinforcement Learning
Exploration
Cooperation
Trajectory Entropy Maximization
Primary area: reinforcement learning

Abstract: Recent works have increasingly focused on learning decentralized policies for agents as a solution to the scalability challenges in Multi-Agent Reinforcement Learning (MARL), where agents typically share the parameters of a policy network to make action decisions. However, this parameter sharing can impede efficient exploration, as it may lead to similar behaviors among agents. Different from previous mutual information-based methods that promote multi-agent diversity, we introduce a novel multi-agent exploration method called Trajectory Entropy Exploration (TEE). Our method employs a particle-based entropy estimator to maximize the entropy of different agents' trajectories in a contrastive trajectory representation space, resulting in diverse trajectories and efficient exploration. This entropy estimator avoids challenging density modeling and scales effectively in high-dimensional multi-agent settings. We integrate our method with MARL algorithms by deploying an intrinsic reward for each agent to encourage entropy maximization. To validate the effectiveness of our method, we test our method in challenging multi-agent tasks from several MARL benchmarks. The results demonstrate that our method consistently outperforms existing state-of-the-art methods.

Published as a conference paper at ICLR 2025
TOWARD EFFICIENT MULTI-AGENT EXPLORATION WITH TRAJECTORY ENTROPY MAXIMIZATION
Tianxu Li!.²
Kun 2h
!College of Computer Science and Technology. Nanjing University of Aeronautics and Astronautics, China "Collaborative Innovation Center of Novel Software Technology and Industrialization (t (tianxuli, zhukun] @nuaa edu. cn
ABSTRACT
Recent works have increasingly focused on learning decentralized policies for agents as a solution to the scalability challenges in Multi-Agent Reinforcement Learning (MARL), where agents typically share the parameters of policy network to make action decisions. However, this parameter sharing can impede efficient exploration, as may lead to similar behaviors among agents. Different from previous mutual information-basse we introduce a novel multi-agent exploration method called Trajectory Entropy Exploration (TEE). Our method employs particle-based entropy estimator maximize the entropy of different agents' trajectorics contrastive trajectory representation space, encouraging visitations of diverse trajectories and efficient exploration. This entropy estimator avoids challenging density modeling and scales effectively in high-dimensional multi-agent settings. We integrate our method with MARL algorithms by deploying an intrinsic reward for each agent to encourage entropy maximization. To validate the effectiveness of our method, we test our method in challenging multi-agen tasks from several MARL benchmarks. The results demonstrate that our method consistently outperforms existing state-of-the- art methods.
methods that promote multi-agent diversity,
INTRODUCTION
Multi-Agent Reinforcement Learning (MARL) has gained considerable attention in recent years for its potential to solve multi-agent tasks, such as multiplayer video games (Vinyals et al., 2019) and traffic light control (Wu al., 2020). MARL facilitates effective cooperation by training multiple agents together to maximize overall team performance. However, developing efficient cooperative policies for challenging multi-agent tasks remains difficult due to the constraints of partial observation and the need for scalability. A widely adopted solution to these challenges is the Centralized Training with Decentralized Execution (CTDE) framework (Lowe et al. 2017). which each agent takes actions based on local observations while being trained with access global information to ensure robust and stable performance.
In the CTDE framework, each agent learns its own decentralized policy. Training numerous policy networks can be inefficient. To address this, the parameter sharing technique is typically employed, allowing all agents use the same policy network parameters make action decisions. This approach significantly reduces the number of required parameters, thereby lowering computational complexity and speeding up the training process. Additionally, parameter sharing facilitates the experience sharing among agents during centralized training, which not only helps in learning a robust and stable policy but also increases overall learning efficiency (Wang et 2020b).
Taking advantage of these benefits, many MARL algorithms have integrated the parameter sharing technique, including value-based methods (Iqbal et al., 2021 Yang et al.
2021
Wang et al. 2020a
Sunchag et al. 2018 Rashid et al. (018)) and policy gradients (Ma et al.
2021 Wang et al. 2020d
Ndousse al. 202T Zhang et al.] 2021 Yu et al., 2022 Kuba et al.) 2021). However, when agents
share the same policy network parameters, they often learn homogeneous behaviors, as they tend
"Corresponding author.

---

Published as a conference paper at ICLR 2025
to exhibit similar behaviors when faced with similar observations (Hu et al. 2022). This limits the emergence of multi-agent diversity and hinders efficient exploration. Complex mulli-agent tasks typically require extensive exploration and varied policies among agents. For example, in a football game, where agents must work together to score, uniform policies might lead to competition for the ball, resulting in ineffective play. To succeed, agents need to learn diverse strategies and assume different roles to effectively pass the ball and score.
To solve the problem, previous works (Jiang and Lui 2021 Li al., 2021 Charakorn et al. 2023 Jo et al. 2024) typically promote multi-agent diversity based on identity-awee trajectories in a sully-supervieed manner. They maximize the mutual information between trajectories and agent identities to differentiate the trajectories of different agents based on their identities. However, despite their success, these methods are prone to overfitting and falling into local optima, as agents tend to revisit familiar trajectories rich in identity information, hindering efficient exploration. Consequently, the agents' trajectories may become overly focused on aligning with their identities.
In this paper, we introduce a novel exploration method called Trajectory Entropy Exploration (TEE) designed to enhance multi-agent diversity while ensuring efficient exploration. Unlike previous methods, our method does not rely on mutual information or trajectory discriminators. The intuition is that agents must thoroughly explore the environment to visit states where they might obtain rewards. To achieve this, our method focuses Since directly maximizing entropy in the high-dimensional trajectory space is intractable due to the unknown trajectory density model, we employ nonparametric particle-based entropy estimator (ingh et al. 2003 Beirlant et al. 1997). This estimator is asymptotically unbiased for entropy by calculating the mean Euclidean distance between a particle and its neighbors. In multi-agent settings, to make these distances meaningful, we build a trajectory representation space through encoding the trajectory space into a oow-dimesiional representation using contrastive learning (Chen et al.. 2020). Contrastive learning is a commonly used method to learn useful representations, encoding similarities and dissimilarities among sample instances (Chen et al. 2020).
maximizing the entropy of different agents' trajectories.
The contributions of this work are summarized as follows: First, we efficiently distinguish trajectory representations of different agents using contrastive learning. Since agents adopting parameter sharing may induce similar trajectory samples, it is intractable to directly use the vanilla contrastive learning. We further introduce an identity representation for each agent to distinguish trajectory representations of different agents. The identity representations serve as medium variables to contrast different trajectory samples. Unlike fixed agent identities used in previous works, identity representations are vectors, consisting of learnable parameters, linearly classify trajectory representations for minimal contrastive learning loss. Second, to encourage multi-agent diversity, we maximize the trajectory entropy using a nonparametric particle-based entropy estimator in the learned trajectory representation space. We adapt the entropy estimator to multi-agent settings by calculating the average distance between a particle and its k nearest neighbors, achieving more stable and robust empirical results. Third, we integrate our method into MARL algorithms by deploying an intrinsic reward, based on the entropy estimator, encouraging agents to maximize entropy. Fourth, we demonstrate the effectiveness of our method through experiments on various challenging multi-agent tasks, where TEE significantly outperforms existing state-of-the-arr MARL algorithms.
BACKGROUNDS
We model fully cooperative multi-agent tasks as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) (Olichoek and Amato (015). defined by the tuple S,U, R. O,Ω,7) Here, A = [I,...,, ] represents a set of [ agents, s S denotes the environment state, and U is the set of possible actions. At every time step, agent a receives an observation o E Ω2 based on the observation function O(s, a) and choose an action u² E U. The actions of all agents combine to form a joint action u. The environment then transitions to new state s' according to the transition function P (s' s, u). At the same time, agents receive shared reward r = R(s, u) from the environment. The reward discount factor is represented by 7 E [0, 1). Each agent's trajectory, composed of its observation-acto
history, is denoted 7" T. Each
agent learns a decentralized policy T" (u" ("), which together form a joint policy π. The goal is to maximize the joint action-value function Q (s, u) Ex Mo... (E r're
80 8, 2Ag u, π
2

---

Published as a conference paper at ICLR 2025
Q
2. (4
Contrsstive learning loss
Mixing Network
Q g
a (orve))
Fatragy
Ausiongresstvve model
Apent
Particle based estropy extimator
Lacosder
neaghbors
Trajectory representation
QMIX
learning
Figure Diagram of our proposed TEE. Our method trains the encoder and the autoregressive model by minimizing the contrastive loss to learn abstract trajectory representation space from trajectories of different agents. We deploy an intrinsic reward function based on a particle-based entropy estimator in the MARL algorithm such that can train the policies of agents towards maximizing the trajectory entropy, encouraging multi-agent diversity.
TRAJECTORY ENTROPY EXPLORATION
One of the common approaches to encourage multi-agent diversity to maximize the mutual information between trajectories and agent identities (Tiang and Lu 2021 Li et al. 2021). However, these works share limitation that the agents are likely to prefer known trajectories, containing more identity information, than novel trajectories, resulting in inefficient exploration. Consider variational reward r iT = log qo (i T) - log p (i) that maximizes the mutual information objective. go (T) is variational distribution. parameterized by 8, which approximates the true posterior probability of the agent identity i conditioned on the trajectory т. p(i) is assumed to be a uniform distribution, so log (i) log |A| is constant, where |A| is the number of agents. When agents visit known trajectories that have been successfully discriminated by go (T. i.e. go (i 7) - 1, the variational reward is Tknowa log log |A| log [.A]. However, when agents visit new trajectories, the variational reward is Nnew limg. (+) log qo (i T) + log |A| = -o. We note that when agents visit known trajectories, they can achieve more rewards. We provide more detailed theoretical analysis of this limitation in Appendix B To resolve this limitation, in this paper, our aim is to encourage multi-agent diversity by maximizing the entropy of trajectories of different agents in an abstract representation space, unlike the previous mutual information maximization method. First, we map the trajectory space to a latent contrastive representation space with a contrastive learning method. Then, we propose novel nonparametric method to maximize the trajectory entropy by introducing per-agent intrinsic rewards.
3.1 LEARNING CONTRASTIVE TRAIECTORY REPRESENTATIONS
We first construct a trajectory representation space to enable the proper functioning of the nonpara- metric particle-based entropy estimator. To achieve this goal, we resort to contrastive learning, which has shown great promise in learning meaningful representations in RL, as demonstrated in recent works (Laskin et al. 2020 Stooke et al.) 202T). However, the vanilla contrastive learning may not be immediately applicable in our multi-agent setting, as the policy network parameter sharing may lead similar trajectories. By contrasting these similar trajectory samples against each other, the vanilla contrastive learning may fail to distinguish trajectory representations of different agents based on distance between them, which is necessary for entropy maximization to work. To solve this issue, we introduce a learnable identity representation for each agent and instead contrast the trajectory samples with the identity representations to pull the trajectory samples with the same identity representations together while pushing apart those with different identity representations in the trajectory representation space.
Concretely, encode trajectories into the trajectory representations by encoding observations into latent embeddings of = 9o, (of) with a non-linear encoder 9a. These embeddings are then summarized with autoregressive model go,- i.e., of = go, (2g,), which alleviates the non-stationary
3

---

Published as a conference paper at ICLR 2025
issue caused by partial observations. We further denote the trajectory encoder go {96.. 90, }.
[A]
Let the set C=
c7
represents the trajectory representations of all agents at time step t, and a'=1
let d R denote the identity representation for agent a. To train the trajectory encoder go order to learn distinguishable trajectory representations, we minimize a contrastive learning loss, or an InfoNCE loss (Oord al. 2018).
dc) Eoyee (ct'd")
NN
E (d CdnD
log
where f (c,,) :..d)= (c) R. We use the dot product similarity c, d to measure the distance between the identity representation and the trajectory representation. The goal of the contrastive learning loss is to guarantee that the identity representation de remains close with its corresponding trajectory representations of while being far away from the trajectory representations of other agents in C { As result, the identity representations of all agents are uniformly distributed on the trajectory representation hypersphere with their corresponding trajectory representations scattered around them, leading to distinguishability among trajectory representations. Note that the identity representations introduced in our method linearly classify the trajectory representations of different agents for the minimal contrastive learning loss.
3.2 NONPARAMETRIC ENTROPY MAXIMIZATION
Prior work (Hazan et al. 2019) typically uses density estimation to maximize entropy. However, in high-dimensional multi-ageen settings, it intractable and non-trivial to estimate the density. In practice, nonparametric particle-based entropy estimator (Singh al. 2003 Beirlant et al., 1997). which has been extensively studied in statistics (Jiao et 20188, is employed in our method to achieve entropy maximization of trajectory representations across different agents. This estimator quantifies the sparsity of data distribution by measuring the distance between a data point and its k-th nearest neighbor.
We next describe the details of the particle-based entropy estimator within our method. In multi-agent settings, each trajectory representation learned by go is considered partiele. Specifically, for collection of trajectory representations {e] |4| induced by all agents, the particle-based entropy estimator is formulated as:
|A
IAI
H(c) = A
log
[AlVE +
b(k) x
log "
(2)
a=1
where b(k) acts bias correction based on the hyperparameter k and ox represents the volume of hypersphere with radius equal to ||o7
(cf) (x))
|c0 ( (c)) I/1/22
(3)
I ([AA/2+1]
where (cg) denotes the k-th nearest neighbor of of in the set (c4) ||-Il refers to the Euclidean distance, and is the gamma function. Essentially, vá indicates the sparsity around each agent's trajectory representation. The entropy estimator H(c:) thus measures the average of the sparsity surrounding the trajectory representations of all agents.
Based on the definition of v. the particle-based entropy estimator Equation can be rewritten as
[A] Σ log ct (c))
H(c,) x
(4)
where the entropy H(a) is proportional to the sum of the logarithms of the distances between each trajectory representation and its k-th nearest neighbor. However, in multi-agent settings, empirically found that the entropy estimator in Equation [ based on the k-th nearest neighbor typically result in learning unstable policies. To address this, introduce novel entropy estimator
4

---

Published as a conference paper at ICLR 2025
for multi-agent settings, which averages the distances over all & nearest neighbors for each trajectory representation:
[A
H()) == Σ log d+ k
Σ
(c7)6)ENa(e()
In this formula, N& (cf) denotes the set of k nearest neighbors surrounding trajectory representation c. We also incorporate a constant d, set to in all experiments, to enhance numerical stability.
To promote diversity among multiple agents by maximizing the entropy H(c:),, entropy as an intrinsic reward r eentropy. where the representation of of41 is considered as a particle contributing to the entropy. Specifically, given a transition (of, uf. of41) for agent a, we define the intrinsic reward function for agent as follows:
use the
G entrrpy
log
d Ko
Σ
- -00(0=))))0
(6)
gu(o24) (olot,)))EE,, (golog_1)
Intuitively, this intrinsic reward encourages agents to explore diverse trajectories with larger distances the trajectory representation space. For the PyTorch-ttyle pseudocode TEE, please refer to Appendix E The source code of our method can be found in the supplemental material.
Differences to mutual information- based methods It important to note that our objective differs significantly from previous methods (Jiang and Lu [202]] Li et al.) [2021] Charakorn et al. [2023] Jo et al., 2024), which leverage the maximization of mutual information between trajectories T and agent identities by introducing intrinsic reward:
TMI (T, log qo (i -17-logg po
where qo (é () represents a variational distribution trained to maximize the likelihood of (é, --tuples stored in the replay buffer. p (1) a fixed uniform distribution. The mutual information based intrinsic reward Farr motivates agents to visit trajectories that carry more identity information. In contrast, our trajectory entropy-based based intrinsic reward r rentropy in Equation 6 incentivizes agents to explore variety of trajectories with greater distances in the trajectory representation space, thereby resulting in entropy maximization.
3.3 LEARNING ALGORITHM
In this section, present how to integrate algorithm with existing MARL methods. As method introduces a trajectory entropy-based based intrinsic reward for each agent, the agent needs to learn its own decentralized policy independently towards maximizing the intrinsic rewards. We first show how to integrate our method with QMIX (Rashid et al., 2018), a value-decomposiiionn-ieee based MARL method. QMIX co-trains the policies of all agents by optimizing an approximation, Qtot- for the joint action-valle function Q" QMIX uses mixing network to monotonically combine the utility functions all agents (from which the agents" policies are derived) to calculate Qtat- Since the policies of all agents are co-trained by QMIX to maximize the shared team rewards, we cannot simply add the individual intrinsic rewards to the shared team rewards to maximize the individual intrinsic rewards for each agent. To solve this limitation, additionally learn shared intrinsic utility network, Quntropy for each agent. The intrinsic utility network Qentropy uses the agent's s utility (o u&) well the current trajectory representation of as inputs. To train the intrinsic utility network Qentropu we minimize the TD loss using the intrinsic rewards:
Centropy EE of JaD [ [Qentropy (Gi Qalo,, 3) LTD
(8)
where y Pentropy + 2 nenrrpp (ci+1 Qa (oi+1: uf+1))
where Centropy and Q. are target networks used to stabilize training. For each training iteration, we randomly sample mini-batch of trajectory data from the replay buffer D. As the intrinsic utility
5

---

Published as a conference paper at ICLR 2025
Tieatea
Trestags
(a) Illustration of Pac- (b) Learning curves (C) Visitation healmap(d) Visitation heatmap(e) Normalized average Men
of QMIX
of TEE
intrinsic rewards
Figure 2: Performance comparison between our proposed TEE and baselines in Pac-Men.
network takes the agent's s utility Qalof, ug) input, the loss function LTT provides an auxiliary gradient to help train the agent's utility network, leading to trajectory entropy maximization. Since the agent utility in QMIX has no inherent meaning constraints, our method can be safely integrated with QMIX. Thus, the overall loss function for learning optimal agent policies is:
CQMIX LTD + -.Eetrpp
Liotal
where CTD CQMIX is the TD loss function in QMIX that is used to learn the optimal Qior and to update the parameters of the agent utility networks with the goal of maximizing team returns. The coefficient adjusts the weight of Centropy in comparison LQMIX
CTD
During training, we alternately train the trajectory encoder and policies of agents. We first sample trajectories from the replay buffer to train the encoder to learn distinguishable trajectory represen- tations by minimizing the contrastive learning loss. Then we calculate the particle-base based trajectory entropy estimator based on the learned trajectory representations for policy learning. The policies of agents are trained end-to-end in centralized manner by minimizing Liotal. allowing each agent to learn a policy that maximizes both the team returns and the trajectory entropy across different agents. Therefore, coameeerr-hariing for efficient exploration. Additionally, our method can be integrated with policy gradient methods. For details implementing method with policy gradient methods, please refer to Appendix D
method fosters diversity among agents, effectively solving the limitation of
EXPERIMENTS
In this section, we examine the performance of our proposed TEE method using challenging multi- agent tasks from Pac-Men, SMAC, and SMACv2 benchmarks, demonstrating its superior effective- ness. We compare TEE against tate-of--the-ar methods, including value-decomposition methods like QMIX (Rashid et al.; 2018) and QTRAN (Son et al. 2019). well mutual information-basse exploration strategies such as MAVEN (Mahajan et al. 2019). EOI (Jiang and Lu [202T). SCDS variant of CDS (Li al. 2021 with shared policy network parameters), PMIC (Li al. 2022),
LIPO Charakorn et al. 2023). and FoX (Jo et al. 2024). Without loss of generality, we present both the mean and standard deviation of performance for our method and the baseline methods, tested with five random seeds. To ensure fairness, consistent hyperparameters and policy network structures are applied across different methods for each multi-agent task, with detailed experimental settings provided in Appendix H
4.1 PAC-MEN
To highlight the effectiveness of our method fostering multi-agent diversity, we adopt grid world environment called Pac-Men, depicted in Figure 2a to evaluate our method against baseline methods. In this environment, four agents start in the central room of a maze, each with limited visibility. Dots randomly placed in the edge rooms, and the agents must navigate along various paths to collect them. To increase the difficulty, each path to the edge rooms has a different length. Importantly, only the downward path falls within the agents' observable area, making efficient exploration essential.

---

Published as a conference paper at ICLR 2025
1&5- GMIX
MAVEN
LO1
PMICC
LPO
10X
We F20255
Test Wet::aaaee Rate:S
We Rat/S1
800
Timesteps
Timesteps
Timesteps
(a) 3s5z (casy)
(b) 2c 64zg (hard)
(c) (hard)
Tel. Mr Rate/l)
80-
Test We Rateraa
Timeabeps
(d) 6h__6____________ 8z (super hard)
(e) corridor (super hard)
(f)3s57_vs_ v55_6/ (super hard)
Figure 3: Performance comparison between our proposed TEE and baselines in the SMAC scenarios.
As depicted in Figure 2b our method demonstrates significant superiority over QMIX and substan- tially outperforms the other baselines. QMIX fails to learn optimal policies in Pac-Men. For the agents to maximize team returns, they need to disperse to the four edge rooms to collect dots. However, the visitation heatmap for QMIX, shown in Figure [c] reveals that some agents adopt similar behaviors, converging in the same bottom room. This overlap leads to competition for the same dots, resulting in inefficient cooperation. In contrast, our method, as illustrated in Figure 2d] enables the agents to efficiently learn diverse policies, allowing them to respectively move to different edge rooms. This outcome indicates that entropy maximization effectively encourages the development of diverse strategies. Baseline methods like HOI and SCDS, which aim to maximize the mutual information between trajectories and agent identities, perform similarly but fall short of achieving satisfactory results. We believe this is due to their inadequate exploration, as agents may fail to discover the upward room with the longest path. Figure 2e further compares the intrinsic rewards generated by mutual information-base mutual information-bass based rewards offer limited incentives, while our entropy maximization-be based rewards consistently motivate the agents to explore and learn optimal cooperative policies.
methods and our entropy maximization method. The results indicate that
4.2 SMAC
After assessing our method in basic grid world environment, we move on to a more challenging multi-agent benchmark, namely StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019). To demonstrate the effectiveness of our method, we examine our method across six SMAC scenarios with increasing difficulty levels: 3s5z (casy), 2c_vs_6ee 2s_62g (hard), 7sz (hard), 6h_ ___vss_ s__ (super hard), corridor (super hard), and 3s5z_vs_ 1155__s_3$660 _356z (super hard). It is important note that performance comparisons are not valid across different SMAC versions. For our experiments, we use SMAC version SC2.4.10.
The comparisons of performance between our method and the baseline methods in the SMAC scenar- ios are presented in Figure In the super hard scenarios (6h_ vs vs_8z, 87, corridor, and 3s5z_vs, vs________. vs_3s6z), where the enemies are significantly stronger than the agents, our method considerably outperforms the baselines. This suggests that TEE more effective at exploring cooperative policies by maximizing trajectory entropy. Challenging scenarios impose high demand on policy diversity to distribute enemies' attacks. Our method successfully learns diverse policies. To demonstrate this, we provide visualization examples of the diverse policies learned by our method in Appendix M While QMIX performs well in the 3s5z and 2c_vs_ 64zg scenarios, it struggles to learn effective policies in the more challenging scenarios that demand complex cooperation strategies, whereas our method excels.
7

---

Published as a conference paper at ICLR 2025
OMIX
MADEN
BGI
SCD8
PMIC
LIO
Pox
Test (2305)))
Text Aunah
Test BettWir Ranahh)
Timesteps
Timesteps
Timesteps
terran 5_vs_5
(b) protoss_ S_vs_5 (c)zerg 5_ S_vs_>
Figure 4: Performance comparison between TEE and baselines in the SMACv2 scenarios.
(a) MAVEN (b) EOI
(c)SCDS
(d) LIPO
(c) FoX
(f) TEE
Figure 5: Visitation heatmaps of different algorithms in the terran 5 v_vs__ scenario.
MAVEN proves less effective at exploring cooperative policies, highlighting that the trajectory entropy maximization objective leads to more efficient exploration compared to the strategy of encouraging diverse joint behaviors by MAVEN. While EOI and SCDS deliver promising results in the 3s5z and 2c_vs_ vs__47g scenarios, they fall short in more challenging scenarios. We believe this is due to the strong mutual dependence between trajectories and agent identities, which hinders the exploration of complex cooperative strategies. Similarly, the formation diversity based on mutual information in FoX also encounters this issue.
Our method also shows superior performance in the easy 3s5z scenario, where agents sometimes need to act similarly to master the 'focus fire' trick. This indicates that our method efficiently trades off exploration and exploitation, and does not hinder the learning of homogeneous behaviors that are beneficial for maximizing environmental rewards. For additional evaluations of our method in scenarios that require homogeneous behaviors, please refer to Appendix
Homogeneous behaviors
U
Stochasticity and Exploration Despite the challenging settings in the SMAC scenarios, notable limitation the insufficient stochasticity combat scenarios due to the fixed initial positions of units and team compositions. This limits the ability to fully test the exploration capabilities of MARL algorithms. To overcome this, we adopt the more demanding SMACv2 benchmark (Ellis et al., (022) which introduces stochastic elements by randomizing start positions and team compositions each episode.
We examine method in three SMACv2 scenarios including terran 5 vs_5, protoss_ 5_ vs_5, and vrrg___v_5... The experimental results, shown in Figure 4 indicate that our method significantly outperforms the baselines across all scenarios. Notably, QMIX struggles to learn optimal cooperative policies and lacks the necessary exploration to adapt to the stochasticity present in the SMACv2 scenarios. However, when combined with our method, QMIX shows marked improvement in performance, learning more exploratory and diverse policies. Similarly, mutual information based baselines like MAVEN, EOI, and SCDS tend to get stuck in local optima. We believe this occurs because the mutual dependence between agent identities and trajectories in these methods restricts agents to known trajectories rather than encouraging the discovery of new ones. contrast, our method consistently explores new trajectories and seeks out exploratory policies. Additionally, provide agent's 's visitation heatmaps Figure 5 The results clearly show that agents trained with the baselines confine their movements to only partial areas, whereas method motivates the agents to explore all possible states that offer environmental rewards, ensuring thorough exploration of the entire environment.
8

---

Published as a conference paper at ICLR 2025
100-
1est Winn Ratey5)
80-
ranzem mearss: neightrr neares: neigtiorss
40
20
0.6
Scenarios SMAC
TCE
TEE w/ varilla contrastive learning
TEE w/o autoregressive model
TEE w/ i-th neaaeett neighbor
TEE w/ fixed encoder
TFT w/ rand ne weiighhor
TEE w/ identity prediction
QMIX
TEE w/ fixed agent identity
Timesteps
(b)
Figure 6: (a) Performance comparisons of our method against different variants in the scenarios of SMAC. (b) Different kinds of intrinsic rewards in the corridor scenario.
4.3 ABLATION STUDY
We perform several ablation studies to examine the contribution of each component in our method. To evaluate the contribution of the autoregressive model used for learning trajectory representations, we create a variant that omits the autoregressive model, relying solely on the observation encoder. To evaluate the importance contrastive representation learning, we develop a variant where trajectories encoded by encoder with fixed randomly initialized parameters. To test the effectiveness of the identity representations introduced in our method, we design two variants: the first employs the vanilla contrastive learning that directly contrasts trajectory samples against each other rather than using identity representations as medium variables to contrast different trajectory samples; the second replaces the learnable identity representations with fixed agent identities, such one-hot vectors, in the contrastive learning loss. Additionally, we introduce a variant that learns trajectory representations through predicting the agent identities from trajectories trajectory entropy, we develop two variants that respectively utilize randomly selected neighbors and the k-th nearest neighbor in the trajectory entropy.
supervised manner. Finally, to test the
We test these variants in three SMAC scenarios: 3s5z (easy). 2c_ so_vs___e vs_64zg (hard), and corridor (super hard). The performance results are presented in Figure 6aL Employing the k-thh nearest neighbor in the trajectory entropy damages performance and introduces significant variance. We also observe noticeable decline in performance when using randomly selected neighbors. However, both variants still receive higher win rates than QMIX, demonstrating the robustness of our representation learning method. As shown in Figure 6b, employing & nearest neighbors in the trajectory entropy provides more efficient intrinsic rewards compared to the other two methods, promoting thorough exploration.
Employing fixed encoder for trajectory encoding results in suboptimal performance, demonstrat- ing the importance of contrastive representation learning. This performance degradation occurs because the representations encoded by a fixed encoder provide inefficient intrinsic rewards for agent exploration. Compared contrastive representation learning, the representations learned via predicting agent identities from trajectories lead to a significant performance decline. This decline occurs because representations guided by fixed agent identities result in strong mutual dependence, hindering efficient exploration. Using fixed agent identities in contrastive learning loss also leads poor performance. Simply using the vanilla contrastive learning without identity representations results in significant performance drop. These variants demonstrate the importance of our contrastive learning loss using identity representations to learn distinguishable trajectory representations. We further provide the trajectory representations learned by different variants in Figure 7 We note that these variants do not necessarily learn distinguishable trajectory representations for the entropy maximization to work.
Furthermore, removing the autoregressive model results in performance comparable to our method in the 3s5z and 2c 64zg scenarios, but it causes significant performance decline in the super hard corridor scenario. This suggests that utilizing the autoregressive model for learning trajectory representations enhances robustness, particularly more challenging multi-agent tasks.

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
(d)
(e)
(f)
Figure 7: T-SNE plots for different agents" trajectory representations learned by different variants
of TEE ((a) QMIX (b) TEE fixed encoder (c) TEE w/ identity prediction (d) TEE w/ fixed agent
identity (e) TEE w/ vanilla contrastive learning (f) TEE), in the corridor scenario of SMAC.
RELATED WORKS
In MARL settings, fostering diversity aims to encourage agents visit diverse trajectories, resulting to varied policies among agents. SVO (McKee et al. 2020) uses social value orientation to address multi-agent social dilemmas by introducing intrinsic rewards that incentivize diverse policy learning. RODE (Wang et al., 2020c) enhances diversity by assigning agents to specific roles with distinct actions, though may struggle in scenarios with continuous large action spaces. MAVEN (Mahajan et al. 2019) adopts value-based approach, conditioning agents' joint behaviors on shared latent variable, controlled by hierarchical policy, through maximizing mutual information. EOI (Jiang and Lu (02T)) trains probabilistic classifier to predict the agent identities' probability distribution based on their observations, using the correct predictions as intrinsic rewards for policy training. CDS (Li et al. 202T) encourages diversity by optimizing mutual information through lower bounds based on the Boltzmann softmax distribution and variational inference. PMIC (Li et al.) 2022) aims to foster the learning of superior policies by maximizing mutual information related to effective cooperative behaviors while minimizing it for less effective ones. LIPO (Charakorn et al. 2023) uses policy compatibility to develop diverse policies and further introduces variations in each agent's 's policy by maximizing mutual information. FoX (Jo et al. 2024) promotes formation based exploration, fostering agents' understanding of their formations by encouraging them explore diverse formations. Despite their achievements, these methods often overemphasize the dependence between agent identity and trajectories or formations, leading agents repeatedly visit similar observations, which can limit their exploration of new possibilities. We refer the reader to extensive related works about entropy maximization in Appendix.
LIMITATIONS AND FUTURE WORKS
In this work, we simply use the distances between trajectory representations of different agents to measure the policy differences among agents. It could be an interesting direction to develop more efficient policy difference measurement that could be used in the trajectory entropy estimator to significantly improve the performance of our method.
7 CONCLUSION
We propose novel multi-agent exploration method to solve the limitation of homogeneous behaviors of agents caused by parameter sharing. Our method introduces an intrinsic reward based on pattcle-bbased entropy estimator to maximize trajectory entropy in a contrastive representation space, which promotes efficient exploration. We test our method in a variety of challenging multi-agent tasks. The experimental results demonstrate the outperformance of our method compared to existing state-of-the-al methods.
ACKNOWLEDGMENTS
This work supported in part by the Fundamental Research Funds for the Central Universities (Grant No. NS2024055), in part by National Natural Science Foundation of China ( 62061 146002), and in part by Natural Science Foundation of Jiangsu Province (Grant No. BK20222012) )
10

---

Published as a conference paper at ICLR 2025
REFERENCES
J. Beirlant, E. J. Dudewicz, L. Györfi, E. C. Van der Meulen, et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 671), 1997.
R. Charakorn, P. Manoonpong. and N. Dilokthanakul. Generating diverse cooperative agents by learn- ing incompatible policies. In The Eleventh International Conference on Learning Representations, 2023. URL http:aas///penreiiwwee.ee.. net forsst//orumpaa UKU055GHT 6.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
B. Ellis, S. Moalla, M. Samvelyan, M. Sun, A. Mahajan, J. N. Foerster, and S. Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2212.07489 2022.
E. Hazan, S. Kakade, K. Singh, and A. Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pages 2681-2691. PMLR, 2019.
S. Hu, C. Xie, X. Liang, and X. Chang. Policy diagnosis via measuring role diversity in cooperative multi-agent rl. In International Conference on Machine Learning, pages 9041-9071. PMLR, 2022.
S. lqbal, C. A. S. De Witt, B. Peng, W. Böhmer, S. Whiteson, and F. Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference Machine Learning, pages 4596-4606. PMLR, 2021.
J. Jiang and Z. Lu. The emergence of individuality. In International Conference on Machine Learning, pages 4902-5001. PMLR, 2021.
J. Jiao, W. Gao, and Y. Han. The nearest neighbor information estimator is adaptively near minimax rate-optimal. Advances in neural information processing systems, 31, 2018
Y. Jo, S. Lee, J. Yeom, and S. Han. Fox: Formation-ave aware exploration in aulti- reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12985-12994, 2024.
J. G. Kuba, R. Chen, M. Wen, Y. Wen, F. Sun, J. Wang, and Yang. Trust region policy optimisation in multi-agent reinforcement learning, arXiv preprint arXiv:2109.11251, 2021.
M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for rein- forcement learning. In International Conference on Machine Learning, pages 5639-5650, PMLR, 2020.
C. Li, T. Wang, C. Wu, Q Zhao, J. Yang, and C. Zhang. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:3991-4002, 2021.
P. Li, H. Tang, T. Yang, X. Hao, T. Sang, Y. Zheng, J. Hao, M. E. Taylor, W. Tao, Z. Wang, al. Pmic: improving multi-agent reinforcement learning with progressive mutual information collaboration. arXiv preprint wxXi::2003.08533 2022.
H. Liu and P. Abbeel. Behavior from the void: Unsupervised active pre-training. Advances Neural Information Processing Systems, 34: 1845998988847, 2021.
R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-commptttive environments. Advances in neural information processing systems, 30, 2017.
X. Ma, Yang, C. Li, Y. Lu, Q. Zhao, and J. Yang. Modeling the interaction between agents in cooperative multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 853-861, 2021.
A. Mahajan, T. Rashid, M. Samvelyan, and S. Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2025
K. R. McKee, I. Gemp, B. McWilliams, E. A. Duéfiez-Gzziin E. Hughes, and J. Z. Leibo. So- cial diversity and social preferences in mixed-motiv reinforcement learning. arXiv preprint arXiv:2002.02325, 2020.
K. K. Ndousse, D. Eck, S. Levine, and N. Jaques. Emergent social learning via multi-agent rein- forcement learning. In International conference on machine leurning, pages 7991-8004. PMLR, 2021.
F. A. Oliehoek and C. Amato. A concise introduction to decentralized pomdps, 2015.
A. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1rXiv:1807,03748, 2018
T. Rashid, C. De Witt, G. Farquhar, J. Foerster, S. Whiteson, and M. Samvelyan. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In 35th International Conference Machine Learning, ICML 2018, pages 6846-6859, 2018.
M. Samvelyan, an. T. Rashid, C. S. De Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr, J. Foerster, and S. Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.000 90..00000444, 2019.
Y. Seo, L. Chen, J. Shin, H. Lee, P. Abbeel, and K. Lee. State entropy maximization with random encoders for efficient exploration. In International Conference Machine Learning, pages 9443-9454. PMLR, 2021.
H. Singh, N. Misra, V. Hnizdo, A. Fedorowicz, and E. Demchuk. Nearest neighbor estimates of entropy. American journal of mathematical and management sciences, 23(3-4):301-3211 2003.
K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pages 5887-5896. PMLR, 2019.
A. Stooke, K. Lee, P. Abbeel, and M. Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pages 9870-9879, PMLR, 2021.
P. Sunchag. G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Son- nerat, J. Z. Leibo, K. Tuyls, et al. dalop-decoooooiiioon networks for cooperative multi-agent learning based on team reward. In Proceedings of the 77th International Conference Autonomous Agents and MultiAgent Systems, pages 2085-2087, 2018.
O. Vinyals, L Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350- 354, 2019.
Wang, Z. Ren, T. Liu, Yu, and C. Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.0 2020a.
T. Wang, H. Dong, V. Lesser, and C. Zhang. Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003. 080399 2020b.
T. Wang, T. Gupta, A. Mahajan, B. Peng, S. Whiteson, and C. Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020c.
Y. Wang, B. Han, T. Wang, H. Dong, and C. Zhang. Dop: Off Off-policy multi-agent decomposed policy gradients. In International conference on learning representations, 2020d.
T. Wu, P. Zhou, K. Liu, Y. Yuan, X. Wang, H. Huang, and D. O. Wu. Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks. IEEE Transactions on Vehicular Technology, (2(3))8243-8256 2020.
Y. Yang, X. Ma. C. Li, Z. Zheng, Q. Zhang, G. Huang, J. Yang, and Q. Zhao. Believe what you Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 101199100333 2021.
12

---

Published as a conference paper at ICLR 2025
D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Reinforcement learning with prototypical rep- resentations. In International Conference on Machine Learning, pages 11920-11931. PMLR, 2021.
C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo in cooperative multi-agent games, Advances in Neural Information Processing Systems, 35: 24611-24624, 2022.
T. Zhang, Y. Li, C. Wang. G. Xie, and Z. Lu. Fop: Factorizing optimal joint policy of maximum- entropy multi-agent reinforeement learning. International Conference on Machine Learning. pages 12491-12500 PMLR, 2021.
A RELATED WORKS ABOUT ENTROPY MAXIMIZATION
Entropy maximization has been adopted in various RL works to effectively promote state exploration. RE3 Seo et al. (202T) aims to improve sample efficiency by facilitating exploration. It achieves this by transforming high-dimensionn dimensional representation space using a fixed encoder, then employing an entropy estimator to assess state entropy within the representation space. Unlike RE3, we use contrastive learning to create a contrastive representation space, which includes more relevant information. APT Liu and Abbeel (2021) introduces pre-training method that maximizes state entropy for sufficient exploration in unknown areas in the task-agnostic environment, utilizing an entropy-based intrinsic reward to train the agent's policy in reward-free setting. ProtoRI. Yarats et al.. (2021) learns representations via prototypes that summarize the agent's 's exploration experience, which not only generalize across tasks but also accelerate exploration efficiently. Drawing inspiration from these approaches, our method in multi-agent settings promotes the diversity among agents by maximizing trajectory entropy contrastive representation space, thereby fostering both efficient exploration and effective collaboration.
Entropy Maximization:
observations into compact, low-
B LIMITATION OF MUTUAL INFORMATIIONBBASEI MULTI-AGENT EXPLORATION
In this section, analyze the limitations of existing mutual information-basee exploration methods from a theoretical perspective, particularly focusing on how agents tend to revisit familiar trajectories rather than exploring new ones. We present the reward functions associated with exploring both familiar and new trajectories. The theoretical results indicate that agents receive higher rewards when revisiting known trajectories compared when they explore new ones.
The mutual information between the trajectory т and agent identity is defined as:
[(i:T) E., [log p [)] E, [log p(i)
(10)
E,,r [log qo [)] - I., (log
where the unknown posterior distribution p(i T) is approximated by variational distribution qa r). We parameterize qo(i T) with 0 and update 0 to maximize the likelihood of (i, ( --tuples stored in the replay buffer. Prior works maximize mutual information by using the variational lower bound as an intrinsic reward:
r (T, = log qu (i' 7) log (i')
(11)
loog 9o (E' 7) + llog |A|
where p(i) is assumed to be a uniform distribution, so - log (i') log [A], where |A] is the number of agents. We assume access perfect distribution go r). ensuring that E!4 go (ia 7)
Intrinsic reward for familiar trajectories The intrinsic reward function encourages agents to visit familiar trajectories where qo (6' ()) + 1. As result:
Tknown
log + +og |A| log |A].
(12)
13

---

Published as a conference paper at ICLR 2025
Intrinsic reward for new trajectories When agents visit trajectories, where qo (t' () is unknown, we assign null probability to unseen trajectories by adding a "background' class to the model. The penalty agents receive when visiting these unseen trajectories is:
new
log qo (i' () + log |A] 9o (7
lim
-00
(13)
This analysis shows that as the distribution qo (V' T) converges, agents are incentivized to revisit known trajectories, where they can achieve much higher rewards compared to exploring new, unfa- miliar trajectories.
C
THE TD LOSS OF QMIX
QMIX trains the policies of agents jointly by optimizing Qtot using the TD loss, defined as:
27
Σ ( max Qeat (St+1, Up+1) Qtor (St, mz)
CTD
y--xmmxxu u1++1
(14)
where Qtot represents the target network, b is the batch size of transition samples, and D is the replay buffer storing trajectory samples. denotes the global reward shared by all agents. Notably, since the policies of all agents are trained jointly by minimizing the TD loss, it is not possible directly incorporate each agent's intrinsic reward r entropp into the global reward r to independently train each agent's 's policy. This limitation necessitates the addition of an intrinsic utility network, Qentropy which is specifically designed to optimize the intrinsic reward r rentropy"
D THE IMPLEMENTATION OF TEE WITH POLICY GRADIENT METHODS
In our paper, we integrated our method with the value-based method QMIX. Here, we demonstrate how to integrate our proposed TEE with policy gradient methods. Specifically, we integrate TEE with MAPPO, a state-of-the-ar policy-based based MARL algorithm measured by SMAC. MAPPO involves training an actor network and critic network shared among all agents. Because each agent trains its own critic, we can thus adopt a shaped reward, P'neo + or ward and ra entrrop is the intrinsic reward generated by our method), when computing the reward-to-go Å to train each agent's s critic network. The other components of MAPPO remain unchanged. We con- duct experiments Pac-Men SMAC, and SMACv2 to evaluate the performance of TEE+MAPPO. The results, presented Table show that TEE+MAPPO outperforms the baselines significantly.
entropy (where Tene is the environmental re-
E PYTORCH-STYLE PSEUDOCODE FOR TEE
The pytorch-style pseudocode for TEE is provided in Algorithm
F ENVIRONMENTAL DETAILS AND ADDITIONAL EXPERIMENTAL RESULTS
In Pac-Men, we initialize four agents positioned in the central room of maze. Each agent can only observe a 4x4 grid around them. There are some randomly initialized dots distributed each edge room. The goal of the agent is to collect as many dots as possible in each edge room. We set different lengths for the paths to investigate the exploration of different MARI algorithms. Specifically, for the downward, left, right, and upward paths, the path lengths are 3, 6, 6, and 10, respectively. Only one path is within the agent's observation scope. The dots each room will refresh when all of them eaten by agents. The environmental reward received by the agent equals the total number of dots eaten in each time step.
SMAC benchmark is set of cooperative tasks built StarCraft II, aimed evaluating the ef- fectiveness of various Multi-Agent Reinforcement Learning (MARL) algorithms. The agent-level control in SMAC is achieved through the Machine Learning APIs provided by both StarCraft II and
14

---

Published as a conference paper at ICLR 2025
Algorithm 1: PyTorch-style pseudocode for TEE
batch: collected trajectories
[Al: number of agents
H:
dimension of the identity zepresentation
perrritttyreeeeeetatiiiiio = Paraaetere (th.randn, (IAI, H) def TEE (batch):
str_out =
for t in range (batch. seggenghh) z_embedding encoder (obba t|)
c_embedding, hidden states hidden_states autoregressive saarre_mmeee(( (2 (2_embedding hidden_stated
etr wtw.oout.appeddc___mmeeidoo etr_out th.stack (ctr_out, dim=1) # Concat trajectory
representations over time
srajeccoc.reereeettttiioooosssss
contraatiiollossstooottteeeeeeee c_embedding)
ogtimizerr.zeroggraddd [aajercoyy_eeceeenttttoooooossss
optimizer.step()
for t sangee >_embedding (batch
t+1])
c_embedding, hidden_stater auttoregresaivre_ooddeeeeeeeeeee hidden_states)
intrinsic_rewarte entropy_estimatorrco (c_emedddoooeeedddiidddddddd
intrinsic_rewards ancclaiic.eerrd..pppppaiiiiiiiit
intrinsic_rewarde torch. stack
dim=1) #
Concat intrinsic rewards over time
return intrinsic_revards
DeepMind's PySC2. Each task features combat scenario with two armies: one controlled by allied RL agents and the other by non-learning game AI. The game is over when all units of any army die predefined time limit is reached. The objective for the allied agents is to learn a policy that maximizes the game's win rate. achieve this, agents must learn sequence actions to collaborate with allies in defeating enemy forces. An illustrative example of such collaboration is the mastery of kiting skills, where agents form formations based on their armor types, compelling enemy units to pursue while maintaining safe distance minimize damage. The SC2.4.10 version of StarCraft II used, and it's important to note that performance comparisons between different versions are not applicable. We conduct experiments
six scenarios including 3s5z, _e_vs___________________________ 64zg. 7sz, 6h _s________ 8z, corridor, and 3s5z_vs vs vs__v______ S_s66zzz with various difficulty levels.
SMAC has notable drawbacks due the lack of stochasticity, and address this issue, SMACv2 proposes some changes: introducing random team compositions, and random start positions. These adjustments aim increase stochasticity efficiently
the exploration of MARL algorithms. We conduct experiments on three scenarios of SMACv2 including terran 5 vs__, 5, protoss 5_ _s__,,___ and zerg 5_vs_5 vv___ SMACv2 employs three unit types for each race. Units are algorithmically generated in teams, with each unit type having a fixed probability. consistent at both test and train times. The unit types of allied agents in these scenarios are identical to those of enemies. the beginning of each episode, the allied agents are randomly spawned in the map with reflect surround style.
We show the average returns of all algorithms in Pac-Men, SMAC, and SMACv2 with standard deviation over five random seeds in Table [ of our method compared to baselines.
The results demonstrate the significant outperformance
G ENVIRONMENTAL DETAILS AND EXPERIMENTAL RESULTS
In Pac-Men, four agents are initialized in the central room of a maze. Each agent has a limited observation range, confined to 4x4 grid around them. Dots are randomly distributed in the edge
15

---

Published as a conference paper at ICLR 2025
rooms of the maze, and the agents' goal is to collect many dots as possible from these rooms. To examine the exploration capabilities of different MARL algorithmss, we set varying path lengths: 3, 6. 6, and 10 steps for the downward, left, right, and upward paths, respectively. Only the downward path falls within the agents' observation range. The dots in each room respawn after all have been collected by the agents. The environmental reward each agent receives corresponds to the number of dots collected in each time step.
The SMAC benchmark consists of a of cooperative tasks built on StarCraft II, designed to evaluate the effectiveness of various MARL algorithms. In SMAC, agent-level control is facilitated through the Machine Learning APIs provided by both StarCraft II and DeepMind's PySC2. Each task presents a combat scenario involving two armies: one controlled by allied RL agents and the other by a non-learning game AI. The game ends when all units of one army are eliminated or when a predefined time limit is reached. The allied agents aim to learn policies that maximize the win rate by collaborating effectively to defeat enemy forces. An example of such collaboration is mastering kiting skills, where agents form formations based on their armor types, forcing enemy units to pursue them while maintaining a safe distance to minimize damage. The experiments utilize version SC2.4. 10 of StarCraft II, and it's important note that performance comparisons across different versions are not applicable. We conduct experiments on six scenarios with varying difficulty levels: 3s5z, 2c_vs_647g, 7sz, 6h_vs_8r, corridor, $s5%_vs_3s$66 MMM2, 10m_ ________lie
1 m, and 27m_vs_30m,
SMAC has some limitations due to its lack of stochasticity. To address this, SMACv2 introduces modifications such as random team compositions and random start positions. These changes aim to enhance stochasticity, making it more effective benchmark for evaluating the exploration capabil- ities of MARL algorithms. We conduct experiments on three SMACv2 scenarios: terran_5 (erran_5_vs_55 $_vs_5, protoss_5 5_vs_5, and zerg___vs._ $_vs_5. SMACv2 sets three unit types for each race, with teams being algorithmically generated based on a fixed probability for each unit type. consistent during both testing and training phases. The allied agents' unit types are identical to those of the enemies. At the start of each episode, the allied agents are randomly spawned on the map in either a reflective or surround formation.
We also test our method in two scenarios of Google Research Football (GRF). GRF provides a challenging, physics-based environment that simulates football game where agents need to learn strategic planning, coordination, and precise timing to succeed. The left side players (except the goalkeeper) agents are trained to learn cooperative policies. The right side palyers are controlled by the game engine. The agents operate within a discrete action space of 19 options, including moving in eight directions, sliding, shooting, and passing. Agent observations include the positions and movement directions of the controlled agent, other agents, and the ball.
We present the average returns of all algorithms in Pac-Men, SMAC, SMACV2, and GRF along with the standard deviation over five random seeds, in Table [] The results demonstrate that our method significantly outperforms baseline methods.
Table 1: Average returns of all algorithms in Pac-Men, SMAC, SMACv2, and GRF, + denotes the standard deviation over five random seeds.
SNY
GAI
BL
SA Aah
18102
The FLTT
wadeyy 1.5 koae Lodoee J.N. Loger
058X
A17.00
030.341
548000
831.805
0OLMM
$414405
437660
GRRA
115000
M0000
08:38
60600
50000
02:18
02638
KC5
3171005
03:361
E5508
371600
00:18
15003
08:18
31:00
60000
09:30
10000
30:105
09914
02:10
131000
00.00
17608
101000
05033
19005
64008
00.36
50500
02638
152415
18300
19003
356660
657-344
6650C30
0N-MI
33650
010000
04038
08/38
MUSIC 557000
H TRAINING DETAILS AND HYPERPARAMETERS
The overall trajectory encoder used in our method is composed of an encoder and an autoregressive model. For the encoder, we utilize two MLP layers with a hidden size of 64, followed by batch normalization. The autoregressive model is implemented using GRU unit. We use randomly initialized learnable vectors as identity representations, which have the same dimensions as the trajectory representations. We integrate TEE with QMIX by adding an intrinsic utility network, which
16

---

Published as a conference paper at ICLR 2025
is composed of two-layer MLP with hidden size of 64, optimized to maximize the total intrinsic rewards. All other components remain identical to the standard QMIX.
To ensure fair comparison, the policy networks for all agents constructed using Deep Recurrent Q-Networks. At each time step, an agent's ' policy network processes a local observation as well as the last step action through fully connected hidden layer, a GRU unit, and final fully connected layer that generates [U] outputs, corresponding to the available actions. We set the evaluation interval to 10K steps followed by 32 test episodes. We run all methods for million steps. In SMAC and SMACv2, target networks are updated using hard updates every 200 episodes, whereas in Pac-Men, soft updates with momentum of 0.01 are employed. The hyperparameters for TEE and baseline methods in Pac-Men, SMAC, and SMACv2 are detailed in Table 2 Parameter sharing is applied across all methods to enable agents to make action decisions. For generality, we report both the mean and standard deviation of performance results, averaged over five random seeds. To ensure fair comparison, consistent hyperparameters are used across different methods. The replay buffer size is set to 5K. We implemented our method using NumPy and Py and all experiments are conducted on a single NVIDIA GeForce RTX 4090 GPU.
Table 2: Hyperparameters
331RU12
loaisiu rale
0,0008
CJMS
sytisnses
(.looh)) 13
(000haat)
barch dioo
64
O for Tebs ou blay, Iin fon ve ve and Iien sou lim. ouniken
0.0
0.01 fo 3ake, 2e Skey, Bi, Suays dea,
0.00 for Tu Mius cuikkn and bi3,, ra Sife for 22, Aigg. th
und Iin
0.01
Torsme, Hll semin.1 15, Strumas,31.s 30.
chilas snncal tine
2000000
200 sdt, 2e MMM, fm, fri sx..t. Sm, vi, Wr. and Imm 3000900 Ter.
corade and v.
SUALC00
I EVALUATIONS OF TEE WITH DIFFERENT VALUES OF k:
The values of ls used in different experimental environments are listed in Table 2 To investigate whether the performance of our method is strongly sensitive to k. we show the performance of our method with different values of k in the terran_5_v__5 (including 5 agents) and terran_20_vs_2 (including 20 agents) scenarios in Table 3 We note that different values of only lead to small differences in performance in both scenarios. The performance of our method remains robust across different values of k.
J EVALUATION OF TEE IN SCENARIOS REQUIRING HOMOGENEOUS BEHAVIOR
While our proposed TEE succeeds in promoting diversity among agents to encourage exploration, there are situations where agents may benefit from behaving uniformly, especially simpler scenarios. For instance, allied agents might employ the same tactic, such simultancously firing at a single enemy to quickly eliminate it. To demonstrate our method's ability to learn such behaviors, we test in four homogeneous SMAC scenarios where the focus fire tactic is advantageous. The results, shown in Table 4 indicate that our method consistently outperforms QMIX in all scenarios, confirming that it can support uniform behaviors when they lead to higher environmental rewards. This phenomenon demonstrates that our method can efficiently balance exploration and exploitation, leading to optimal cooperative behaviors.
Table 3: Performance of our method with different values of
Methods
leman 5
terran 20 20
k=10
k=
k=18
TEEX 0.92 20.03 0.90 20.03 0.93 10.04
0.96 #0.02
0.85 20.03
0.86 10.04
0.83 2004
0.88 20.05
0.89 =0.03
17

---

Published as a conference paper at ICLR 2025
Table 4: Performance of our method and QMIX in homogeneous scenarios.
Method
8m
5m__ 6m
8m vs 9m
10m Ilm
TEE+QMIX
0.95
0.92:10.04 0.65110.04
0.94 0.58
0.91 0.06
QMIX 0.87+0003
0.43:10.04
Table Performance comparisons of our method against QMIX with different values of
Methods
corridor
3s5z vs 3s6z
Trajectory entropy maximization (Ours)
0.90 10003
0.87 10.04
0.05
0.57 +0.07
0.36 #0.12
= 0,08
0.61 +0.4
0.39 +0.09
0.12
0.65 0.06
0.45 10.14
K COMPARISON WITH C-GREEDY
e-greedy is widely used exploration technique in many RL approaches. Increasing the value of e generally promotes better exploration. In this section, we compare our entropy maximization method with e-greedy to demonstrate the effectiveness of our method in promoting exploration in the domain of MARL. To achieve this, we set the values 0.05, 0.08, and 0.12 for QMIX, respectively. We test different values of in the super hard scenarios corridor and 3s5z_vs 3s6z. The results are shown in Table 5 Our entropy maximization method more efficient than increasing the values encourage effective exploration in multi-agent environments. We note that increasing the values of does not improve the performance significantly. multi-agent settings, larger e values only increase the stochasticity in action selections of single agent and ignore the diversity distinguishability among agents as they do not consider the trajectories of other agents, thus leading to inefficient exploration.
to
L SCALABILITY
Many MARL methods suffer from poor scalability, i.e., the agent performance decreases significantly as the number of agents increases. This occurs because the state-action space expands exponentially as the number of agents increases, making efficient exploration crucial. In this section, we evaluate the scalability of our method across four SMACv2 scenarios with increasing numbers of agents: terran_5 5_vs__, per_an_ 10, terran_15_ _rrran_1_vs____ 15, and terran_22 _0________________ vs__0. The results, presented in Table 6, show that method substantially outperforms QMIX across all scenarios. QMIX faces challenges in scalability and struggles to achieve satisfactory performance due to insufficient exploration. In contrast, our method shows strong scalability, demonstrating that maximizing trajectory entropy enables efficient exploration.
M VISUALIZATIONS
We present some visualization examples of the diverse policies learned by method, as shown in Fig- ure g which emerge in the extremely challenging scenarios (6h_vs_8z, corridor, and 3852_vs_3s666 For instance, in the 6h_ vs_8z scenario, agent moves in the opposite direction of the team cover other agents. Most of the enemies are then drawn to the agent's movements. The agent continues kiting the pursuing enemies, drawing the majority of their fire. Meanwhile, the remaining enemies are surrounded by the other agents. This strategy allows agents to cooperatively distribute enemies' attacks. In contrast, if all agents behaved similarly and rushed toward the enemies, they would be
Table 6: Performance of our method and QMIX in scenarios of SMACv2 with different number of agents
Method TEE+QMIX
terran 5 _s___
terran 10. vs_10
terran 15 vs_15
terran 20 _s__v___000 20
0.96:10.2
0.95 =0.03
0.92 +0.04
0.89 =0.03
QMIX
0.68++0.0
0.39+10.0
0.24 +0.06
0.11=00.05
18

---

Published as a conference paper at ICLR 2025
6h vs 8z
corridor
3s5z vs 3s6z
Figure 8: Visualization examples of diverse policies emerging 6h vs 8z (top), corridor (medium), and 3s5z_vs ss__vs________ vs_3s6z (bottom) from initial (left) to final (right). Green and red shadows represent agents and enemies, respectively. Green and red arrows represent the moving directions of agents and enemies, respectively.
quickly defeated by the stronger opponents. Similar diverse policies are observed in the other two scenarios. These results highlight the effectiveness of method in learning diverse policies, helping agents cooperatively defeat enemies.
We also provide additional visitation heatmaps for TEE+QMIX and the baseline methods in SMACv2 scenarios, as shown in Figures D and I0 These heatmaps illustrate that our proposed TEE achieves more efficient exploration compared the baselines. The mutual information-based baselines such as MAVEN, EOI, and SCDS do not sufficiently explore the environment, making them less effective defeating enemies that appear randomly on the map. While the mutual information objective encourages multi-agen diversity, it can also impede effective exploration. In contrast, our proposed TEE leverages the trajectory entropy maximization objective, which drives agents adopt exploratory policies without being constrained by mutual dependence.
(a) MAVEN
(b) EOI (c)SCDS
(d) LIPO
(e) FoX
(f) TEE+QMIX
Figure 9: Visitation heatmaps of different algorithms in the protoss 5 $_vs_5 scenario.
19

---

Published as a conference paper at ICLR 2025
(a) MAVEN
(b) EOI
(c)SCDS
(d) LIPO
(e) FoX
(f) TEE+
Figure 10: Visitation heatmaps of different algorithms the 5erg___vs vs_5 scenario.
20


Reviews:
[{"review_id": "aB8P79tp6g", "reviewer": "Reviewer_posU", "summary": "This paper introduces Trajectory Entropy Exploration (TEE), a multi-agent exploration method that maximizes the entropy of different agent's trajectories in a contrastive trajectory representation space. Prior approaches that incentivize exploration by maximizing the mutual information between the agents' identities and their trajectories can lead to insufficient exploration, where each agent revisits familiar trajectories. In contrast, TEE trains a trajectory representation using *contrastive learning on trajectories and agent identities*, where trajectories from the same agent are pulled closer. This latent representation is then used in a *particle-based entropy estimator*, which estimates the entropy by averaging the distance of each particle to its top $k$ nearest neighbors in the representation space. The proposed method is instantiated with the QMIX algorithm, where an additional $Q$-value for the entropy rewards is learned and added to the standard $Q$-value. TEE exhibits strong empirical results across a suite of MARL environments including Pac-Men, SMAC, and SMAC 2, outperforming baselines in exploration and sample efficiency. The design choices in TEE are thoroughly validated via exhaustive ablation studies.", "questions": "1. If the limitation of methods maximizing the mutual information between trajectories and agent identities is insufficient exploration, why can't we add a max entropy term to each agent's objective, so that they explore as much as possible while being distinguishable from each other? Are there related works that do this?\n2. The autoregressive trajectory encoder is trained on trajectories but only applied to a single state when estimating the particle-based entropy (according to Algorithm 1). So in principle one only needs a state encoder. Why do you train a trajectory encoder when it's only applied to states?\n3. What happens if you replace the L2 distance in the entropy estimator with negative cosine similarity? Since the encoder is trained with contrastive loss, cosine sim seems a more natural metric in the latent space. \n4. In scenarios requiring homogeneous behavior (Appendix. F), do you need to tune the coefficient $\\beta$ of the entropy loss?\n5. Section 3.1 on learning contrastive trajectory representations is disconnected from Sections 3.2 and 3.3 on the particle-based entropy estimator and its integration with MARL algorithms. I suggest adding a paragraph in Sec. 3.3 stating that the representation is learned jointly with the policy, to provide a more complete picture.\n6. I would also suggest adding a more detailed description of TEE in the caption of Figure 1 to make it self-contained.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is motivated by carefully analyzing existing multi-agent exploration methods and identifying a potential degeneracy in those methods, i.e., that maximizing the mutual information between agent identities and trajectories can lead to insufficient exploration. The proposed method -- maximizing the entropy of the mixture of trajectories while being informative about agent identities -- is a convincing solution. \n2. TEE demonstrates strong empirical results on a suite of challenging multi-agent RL environments, significantly outperforming standard MARL and MI-based exploration methods. Figure 2 in particular shows that TEE fully explores all four rooms in the Pac-Men environment whereas Q-MIX only explores two of the four rooms. Figure 5 further shows that TEE explores the map in SMAC more thoroughly than MI-based exploration methods. These lend support to the effectiveness of the exploration method.\n3. The design choices in TEE are backed by thorough ablation studies in Section 4.3, Figure 6, and Table 1. The authors show the importance of parameterizing the trajectory encoder using an autoregressive model, learning the agent identities instead of fixing to one-hot, and so on.", "weaknesses": "1. The proposed method relies on a nonparametric entropy estimation that introduces additional hyperparameters, e.g. the number of neighbors $k$. This procedure also introduces additional computational overhead, since it needs to run batch KNN in each training step.\n2. Some design choices of the method are not straightforward. See questions 1, 2, and 3 below.\n2. The presentation of the paper can be improved. Some paragraphs are too dense (e.g. Section 4.3) and figures need more informative captions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FVcRvVYRkZ", "reviewer": "Reviewer_1kFd", "summary": "This paper introduces a novel multi-agent exploration method, Trajectory Entropy Exploration (TEE), to address the issue of homogeneous agent behaviors resulting from parameter sharing. TEE leverages contrastive learning to encode trajectories and identities into distinguishable embeddings, utilizes a nonparametric particle-based entropy estimator to calculate intrinsic rewards, and integrates seamlessly with existing MARL algorithms such as QMIX and MAPPO. Experiments on the Pac-Men, SMAC, and SMACv2 benchmarks demonstrate its superior effectiveness.", "questions": "- As mentioned in the introduction, football games require agents to learn diverse strategies, and Google Research Football is a widely used MARL benchmark. This approach should include additional experiments on this benchmark, similar to the baselines SCDS and FoX.\n- I believe it should become a community standard to adopt the evaluation protocol of [Gorsanne et al.](https://arxiv.org/abs/2209.10485) in the multi-agent setting. Not adhering to these standard practices, in my view, weakens what could otherwise be a compelling case for the effectiveness of these algorithms.\n- Figure 1 is somewhat unclear. Are the agents' policies derived from the utility functions $Q_a$ or from the combined term $Q_a + \\beta Q_a^{entropy}$? If it’s the latter, why not directly update $Q_a$ with $r^a + r^_{entropy}$ instead of using a separate intrinsic utility network?\n- Minor items: In Line 184, should $v_a^k$ be written as $\\text{v}_a^k$?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-motivated and clearly presented. The method effectively combines maximum entropy with contrastive learning to address the diversity challenge in MARL, demonstrating strong empirical performance.\n- The tricks used are both practical and easy to implement, and the authors provide thorough ablation experiments to support their approach.", "weaknesses": "- Lack challenging MARL tasks that require diversity strategies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bGi85wTuAT", "reviewer": "Reviewer_17Sh", "summary": "TEE promotes agent diversity in multi-agent RL by learning a contrastive trajectory representation and using it to compute a particle-based entropy of the agent trajectories as intrinsic rewards. Combined with QMIX, TEE is evaluated in two multi-agent domains, showing increased diversity and superior empirical performances as a result.", "questions": "See other sections.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The writing is overall clear and well-organized despite some small issues, which I will mention in other sections. Figures in this paper are helpful as well, although the authors might want to change some font colors in Figure 1 to make it more readable.\n- The method is interesting and novel to my knowledge, addressing issues existing in prior methods under the same topic.\n- The experiments in the PacMan environment clearly demonstrate the advantages of promoting agent diversity compared with baselines.\n- In SMAC, TEE has significant improvement over baselines in the harder scenes. Even in easier scenes where diversity is not required, TEE still has a small advantage, showing that empirically it can balance the diversity maximization and reward exploitation.\n- Overall for empirical evaluations, there are a variaty of baselines and ablations, providing sufficient information about the proposed method.", "weaknesses": "- I appreciate the theoretical analysis of why mutual information-based methods can have insufficient exploration in the appendix, but it would be more helpful if the authors could include some insights of the analysis into the main text.\n- I would like to hear more explanation on why using $Q_a(o_t^a,u_t^a)$ as an input of $Q^{entropy}_a$ can lead to trajectory entropy maximization. This is not straightforward to me. Yes, there will be a gradient passed to $Q_a(o_t^a,u_t^a)$, but why can this gradient make $Q_a$ assign higher values to actions that have higher trajectory entropies?\n- TEE is evaluated in two domains, PacMan and SMAC. PacMan to me is more of a demonstrative domain, where behavioral diversity is vital to get the maximum reward. As such, the diversity in evaluated domains feels a bit lacking.\n- In the evaluations, TEE is only combined with QMIX. I'm wondering why it's not tested with other MARL algorithms as QMIX is one of the less intuitive one to combine with.\n\nMinor writing issues:\n- Using same characters for encoder $g_{\\theta_e}$ and $g_{\\theta_g}$ is confusing. It feels like two same networks with different parameters.\n- L184 the $v$ in $v_a^k$ is italic.\n- L248 you state that $Q^{entropy}_{a}$ is \"shared\". Why does it still have a subscript $a$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7rQQXo9Dz2", "reviewer": "Reviewer_6aVA", "summary": "This paper aims to enhance exploration in MARL by maximizing trajectory entropy (TEE). The authors propose encoding trajectories into low-dimensional vector representations through contrastive learning, calculating particle-based entropy on these representations, and maximizing it as an intrinsic reward. This method is incorporated into the QMIX algorithm and tested in grid-world and SMAC/SMAC-v2 environments, with experimental results indicating that TEE outperforms other QMIX-based exploration techniques.\n\nAlthough the paper is well-structured with a coherent logic flow, I have concerns regarding its unclear motivation and potentially problematic experimental results. It shares a common shortcoming with MARL research from a few years ago. I recommend a rejection.", "questions": "* Is the trajectory representation learned on-the-fly from the replay buffer, or is it pre-trained?\n\n* Can your exploration method be applied to policy-gradient methods?\n\n* Why did you choose QMIX as the base algorithm instead of QPLEX[6], which is more advanced and theoretically general?\n\n[6] QPLEX: Duplex Dueling Multi-Agent Q-Learning, https://arxiv.org/abs/2008.01062", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written with a logical flow.\n\n* The authors include an extensive set of experimental baselines; however, some important baselines are missing, as I elaborate in the next section.", "weaknesses": "* **Policy-Gradient-Based Algorithms**. This paper does not address any policy-gradient-based (PG) MARL algorithms, raising two issues. First, it's unclear whether insufficient exploration is a significant problem for PG algorithms as well. Could you also include policy gradient algorithms like MAPPO [1] and HAPPO [2] in the Pac-Men experiments? At the very least, these works are relevant and should be cited. Second, policy gradient methods have proven competitive on benchmarks such as SMAC. Even if these algorithms aren't directly compared on the learning curve, their final performance should be used as a baseline so readers can see that your approach advances the MARL field. Omitting them weakens the paper's overall rigor.\n\n[1] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games, https://arxiv.org/abs/2103.01955\n\n[2] TRUST REGION POLICY OPTIMISATION IN MULTI-AGENT REINFORCEMENT LEARNING, https://arxiv.org/pdf/2109.11251\n\n* **Pac-Men Results Conflict.** Results in the Pac-Men environment appear to conflict with the published CDS paper [3]. The CDS paper shows this environment is solvable with 2 million timesteps, but in your results, it isn't. Please explain this discrepancy in detail. Additionally, while explaining your algorithm's success, you should analyze why baseline methods fail rather than providing vague reasons like \"insufficient cooperation\". Consider analyzing trajectories in the replay buffer, visualizing the Q-value heatmap, or conducting a more thorough breakdown.\n\n[3] Celebrating Diversity in Shared Multi-Agent Reinforcement Learning, https://arxiv.org/pdf/2106.02195\n\n* **Insufficient Evaluation.** The evaluation approach needs refinement. First, only six maps were selected in the SMAC environment, which raises concerns about potential cherry-picking. Second, Figure 3 shows a significant divergence from the curves in the CDS paper. Although the authors suggest this is due to differences in StarCraft engine versions, it would be best to match the version used in the CDS paper to ensure fair performance comparisons. Third, recent work has introduced more reliable MARL evaluation protocols [4,5], recommending evaluation across multiple domains (e.g., GRF, Melting Pot, SMAC) and reporting IQM scores/curves for each domain. Adopting these standards would improve the reliability of your evaluations.\n\n[4] JaxMARL: Multi-Agent RL Environments in JAX, https://arxiv.org/abs/2311.10090\n\n[5] Towards a Standardised Performance Evaluation Protocol for Cooperative MARL, https://arxiv.org/abs/2209.10485", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)