Measuring Mutual Policy Divergence for Multi-Agent Sequential Exploration
Authors: Haowen Dou
Lujuan Dang
Zhirong Luan
Badong Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-agent reinforcement learning
sequential updating
exploration
Cauchy-Schwarz divergence
Primary area: reinforcement_learning
TL;DR: A method that maximizes conditional Cauchy-Shwarz policy divergence between agents and between episodes to enhance exploration and heterogeneity for MARL.

Abstract: Despite the success of Multi-Agent Reinforcement Learning (MARL) algorithms in cooperative tasks, previous works, unfortunately, face challenges in heterogeneous scenarios since they simply disable parameter sharing for agent specialization. Sequential updating scheme was thus proposed, naturally diversifying agents by encouraging agents to learn from preceding ones. However, the exploration strategy in sequential scheme has not been investigated. Benefiting from updating one-by-one, agents have the access to the information from preceding agents. Thus, in this work, we propose to exploit the preceding information to enhance exploration and heterogeneity sequentially. We present Multi-Agent Divergence Policy Optimization (MADPO), equipped with mutual policy divergence maximization framework. We quantify the policy discrepancies between episodes to enhance exploration and between agents to heterogenize agents, termed intra-agent and inter-agent policy divergence. To address the issue that traditional divergence measurements lack stability and directionality, we propose to employ the conditional Cauchy-Schwarz divergence to provide entropy-guided exploration incentives. Extensive experiments show that the proposed method outperforms state-of-the-art sequential updating approaches in two challenging multi-agent tasks with various heterogeneous scenarios. Source code is available at \url{https://github.com/hwdou6677/MADPO}.

Measuring Mutual Policy Divergence for Multi-Agent Sequential Exploration
Haowen Dou 1,2,3 1,,3
Lujuan Dang
1,,3, 12,3,
Zhirong Luan
Badong 1.,3,,
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications,
Institute of Artificial Intelligence and Roboties, XI' an Jiaotong University,
"School of Electrical Engineering, Xi'an an University of Technology douhaowen@stu.xxjtu.uu edu. en, dangl j ju. edu. en, luanzhirongéxaut.o edu. tu. edu. cn chenbd@mail xjtu.
Abstract
Despite the success of Multi-Agent Reinforcement Learning (MARL) algorithms in cooperative tasks, previous works, unfortunately, face challenges in heterogeneous scenarios since they simply disable parameter sharing for agent specialization. Sequential updating scheme was thus proposed, naturally diversifying agents by en- couraging agents to learn from preceding ones. However, the exploration strategy in sequential scheme has not been investigated. Benefiting from updating one-by-one, agents have the access to the information from preceding agents. Thus, in this work, we propose to exploit the preceding information to enhance exploration and het- erogeneity sequentially. We present Multi-Agen Divergence Policy Optimization (MADPO), equipped with mutual policy divergence maximization framework. We quantify the discrepancies between episodes enhance exploration and between agents to heterogenize agents, termed intra-agent divergence and inter-agent diver- gence. To address the issue that traditional divergence measurements lack stability and directionality, we propose to employ the conditional Cauchy-Schwarz diver- gence to provide entropy-guide exploration incentives. Extensive experiments show that the proposed method outperforms state-of-the-e sequential updating ap- proaches in challenging multi-agent tasks with various heterogencous scenarios. Source code is available at htttt:t////. /ttpt/gttttt.t//////// hawbbbccom/wdoo6677//W0a
Introduction
Multi-Agen Reinforcement Learning (MARL) plays an increasingly important role in numerous real-world cooperative problems, such as smart grid management [Zhang al., 2022b). autonomous driving WWang et al. 2023b), unmanned system control [Feng et al.) 2023b and games [Zhang ct al. 2022a]. Centralized and decentralized MARL methods have been investigated as the first two extensions from single-agent to multi-agent systems. However, challenges have arisen regarding curse of dimensionality and non-stationary training as the number of agents increasing (Mao al. 2022). To address this issue, Centralized Training with Decentralized Execution (CTDE) developed to disentangle training and execution phases FFerster et al. 2018), In the CTDE scheme,
Coorresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the centralized critic provides global information, guiding agents during training but not during execution. CTDE significantly simplifies and stabilizes the training process, providing an effective and efficient paradigm for policy-gradient cooperative MARI..
In CTDE, agents share parameters for homogeneous tasks, such multi-particle coordination, and then take actions sampled from the same policies. For heterogeneous tasks, such as multi- joint coordination in robotic control, they learn distinct policies without sharing parameters and exhibit different behaviors. However, in these scenarios, relying solely on the non-paramette--hariin setting to achieve cooperation is an oversimplification [Bhattacharya et al., 2023]. This is because agents can never learn optimal policies that depend on trajectories from other agents when updating simultaneously. To tackle this problem, sequential updating [Bertsekas] 2021] has been proposed to improve heterogeneity and collaboration. This updating scheme originates from the insight that agents in one rollout update their policies one-by-one, rather than simultaneously, to retain preceding agent information. Several sequential methods have been proposed by leveraging the multi-agent advantage decomposition lemma[Kubs et al.) 2022], the multi-agent performance difference lemma [Wang al. 2023a]. and rollout policy iteration [Bertsekas] [021], to not only adapt the sequential updating scheme but also maintain the monotonic improvement property.
Despite the success of sequential policy updating, the exploration towards further heterogeneity improvement remains unexplored and challenging (Zhang et al. 2022a]. In MARL, agents struggle to learn globally optimal policy due to the huge exploration space complexity, which is, unfortunately, further amplified in heterogeneous tasks. Existing multi-agent exploration strategies typically require parameter sharing in homogeneous scenarios. However, when applied to heterogeneous scenarios, they suffer from performance degeneration despite employing the non-parameter sharing setting. This is because these methods fail to fully leverage the main advantage of sequential updating, i.e. the preceding information. To the best of our knowledge, there is no exploration method that can adapt to both heterogeneous scenarios with sequential updating and homogeneous scenarios with simultaneous updating.
To this end, this paper presents novel sequential MARL framework, termed Multi-Agen Divergence Policy Optimization (MADPO), where a simple yet efficient exploration strategy is equipped to enhance sample efficiency, particularly in heterogeneous scenarios. In MADPO, we first propose a Mutual Policy Divergence Maximization (Mutual PDM) strategy to heterogenize agents. Specifically, mutual PDM consists of the intra-agent divergence and the inter-agent divergence. The intra-agent divergence measures the policy discrepancy between episodes, encouraging agents to learn diversified policies. The inter-agent divergence measures the policy discrepancy between agents, enhancing heterogeneity and promoting greater diversity. However, simply applying classical divergence measures to the proposed framework may trap the exploration in local optima due to the lack of positive incentives. To address this issue, we propose to employ conditional Cauchy-Schwarz (CS) divergence to provide cntropy-guides incentives. Compared to the famous Kullback-Leti Leibler (KL) divergence, the conditional CS divergence implicitly maximizes the entropy of current policy and is more stable. The main contributions can be summarized as follows:
1.
We develop a novel multi-agent divergence reinforcement learning model equipped with mutual policy divergence maximization, termed MADPO, to enhance exploration and heterogenize agents in heterogeneous scenarios. To the best of our knowledge, we are the first to demonstrate the efficacy of policy divergence maximization in sequential MARL.
2. We propose to maximize conditional Cauchy-Shwarr policy divergence to provide entropy- guided incentive and stabilize multi-agent sequential exploration.
3. We evaluate the proposed method through extensive experiments. The results show that MADPO outperforms state-of-the-ar art sequential methods in two multi-joint coordination tasks with various heterogeneous scenarios.
2 Related Works
2.1 Multi-Agent Reinforcement Learning with CTDE
Benefiting from the CTDE framework, multi-agent policy gradient algorithms have paved promising path for cooperative games CCai et al.] 202T Qiu et al., 202T] Li et al. 202T]. For example, [u et al.]
2021 proposed CoPPO which guarantees the joint policy improvement by adapting the step size
2

---

dynamically. Yu et al. 2022 proposed Multi-Agent Proximal Policy Optimization (MAPPO) which applies PPO to multi-agent scenarios without violating the guarantee of monotonic improvement in the individual policy level. Policy entropy incentive in MAPPO is one of the most related parts to our method, providing diversified policy learning from an information theory perspective. [Li and He [2023] proposed MATRPO to extend Trust Region Policy Optimization (TRPO) to multi-agent tasks through a fully decentralized setting and distributed optimization. However, when the number of agents becomes large, MATRPO may encounter the challenge of the connecting link dimension curse. This is because it relies on communication rather than global information to facilitate cooperation. Guo et al. [024] proposed MASPG, trust region- based MARL algorithm in the off- policy manner, to enhance the sample efficiency of trust region methods. However, these methods require homogeneity of agents, i.e. parameter sharing, to ensure monotonic improvement. This homogeneity assumption can impose significant restrictions on agents, limiting their ability to explore the joint policy space adequately (Ding et al. 2022]. Consequently, if the sharing of parameters is canceled, it can lead to violations of the monotonicity guarantee and result in performance degradation (Zhan et al. 2023
2.2 Sequential Updating MARL
The sequential updating scheme originates from single-agent rollout and policy iteration [Bertsekas 202I], aiming to update policies of agents one by one, as shown in Fig. Ia This structure encourages agents to learn different policies based on information from preceding ones, thereby naturally generalizing the homogencous MARL to heterogeneous MARL. To build the multi-agent sequential updating scheme, attempts have been addressed from both joint and individual policy perspectives. For example, [Bertsekas] [2021] proposed rollout and policy iteration method, which was the first to consider sequential updating in MARL. Kuba et al. [2022] observed the multi-agent advantage decomposition lemma and proposed HAPPO. Leveraging this powerful lemma, HAPPO estimates and decomposes the joint advantage function to implement sequential updating and ensure the joint monotonic improvement. On the other hand, A2PO proposed by Waag et al. [2023a] focuses policy difference lemma. A2PO
on individual policy improvement by leveraging the maintains distribution invariance during each agent's 's advantage estimation process and consider a more refined updating order. Zhao al. (2023) introduced localized action value function as the surrogate optimization objective, offering provable convergence guarantee for multi-agent PPO.
2.3 Information Theory Induced RL
Information-thenn -theoretcc principles serve as a powerful regularization technique for providing valuable guidance in intrinsic reward-driven RL [Liu and Zhang 2023 Subramanian et al., 2022 Russo and Proutiere 2024]. J. including both policy and state exploration [Cn et al.) [02 Jacob al.) [022]]. For example, the Soft Actor- Critic (SAC) is the first to maximize the Shannon entropy of policies, promoting randomness and encouraging exploration [Haarnoja al. 2018). On-policy methods, such as PPO, MAPPO and HAPPO, embrace the same concept by incorporating entropy regularization into the optimization objective. Additionally, recent advancements have explored the utilization of various entropy forms, such as encoder estimated stable entropy [Liu and Abbeel 2021]. value conditional entropy (Kim et al., [203) and Rényi entropy (Yuan et al. [023] to model environmental dynamics and accelerate novel state discovery. However, maximizing entropy only introduces stochasticity for measuring uncertain dynamics. To address this limitation, policy divergence regularization
between episodes [Su and Lu 2022 Xu et al.] 2023) has been proposed. This regularization method calculates the policy divergence based on a fixed policy and offers more directed guidance compared to entropy alone. Furthermore, the efficacy of state divergence in combating local optima and fostering state novelty has been demonstrated (Hong et al. 2018 Yang et al., 2021]. However, the existing divergence RL methods cannot be effectively extended to the sequential updating paradigm.
In this work, we pursue an on-policy method to enhance exploration and heterogenize agents in a sequential updating paradigm, termed Multi-Agent Divergence Policy Optimization (MADPO). In contrast to the aforementioned policy divergence- maximizes inter- and intra-agent policy divergence, thereby incorporating policy information. To further address the deficiency of exploration direction in the traditional divergence RI., we propose to employ the conditional Cauchy- Cauchy-Schwarz divergence to provide an entropy-guidee incentive.
methods, we introduce a novel approach that
3

---

7Ck
1
Mutual PDM
R
2
π
π
πt
π
Mutual PDM
Updats
7t
π
Agent 1-1
Agant
it
IT
Update
Episoce
7
π
Mutual PDM
Enkood
Updite
Updaet
π
πt
π
x
Div(nille 1) DivCeslinE..)
(a) Sequential Updating MARL
(b) Multi Aulti-Agent Divergence Policy Optimization
Figure 1: A three-agent example of traditional sequential updating MARL and our MADPO. The white boxes represent the policies to be updated n', and the orange boxes represent the updated policies #.. The white boxes with dashed lines represent the joint policies to be updated and the orange ones represent the updated joint policy म. Compared the traditional sequential updating MARL, our method takes the intra-agent and inter-agent divergence into account, shown in the blue boxes. The intra-agent divergence directs agents to explore novel policies based on their former policies, while the inter-agent divergence heterogenizes agents sequentially.
3 Preliminaries
3.1 MARL Porblem Formulation
In this paper, we consider multi -ggntt sequential decision decision-making problem, which can be described as decentralized Markov decision process (DEC-MDP). A DEC-MDP with agents can be formulated as the tuple: (S, A, T,, T,7), where S represents the state space. We denote N (1....,n) n) the set of finite agents. A A x ... x Ar is the joint action space by taking the x Ax + [0, 1] is the state transition function
product of actions spaces of n agents. S x A + IR is the reward function. 7 is the discount factor.
of the environment dynamics. At time step t, to interact with the environment, each agent at state s, S takes an action af from its policy x' x-(ss) to form joint action a; fa!..... a²") and joint policy π (-1ss) n² X x... .... the probability T(s++1 $t, at). The objective is learn optimal joint policy by maximizing the expected cumulative reward: म = arg max E E Earrpn (n"r(st. a.)], where Pn is the
л". The agents then receive a joint reward r(st, a,) and step to the new state $141 with
marginal state distribution. Following Bellman Equations, the state-action value function and the state function state 8; are defined as Q az) = r(st, ar) + Eis, Es,ren (r=,,, a;)). and V (st) = Σie, Ear-pa,, (r as)]s0 = 81). And the advantage function is defined as A (s a.) = Q" ( a,) (st).
3.2 Multi-Agent Sequential Policy Updating Paradigm
Sequential updating paradigm was introduced to alleviate homogeneity in multi-agent reinforcement learning. The overview of sequential updating with a three-agent setting is shown Fig. la For instance, Heterogeneous-Agent Proximal Policy Optimisation (HAPPO) takes preceding agent information into account by employing the multi-agent advantage decomposition lemma and the joint At episode k, agent in HAPPO maximizes the extrinsic
advantage estimator Kuba et al. 2022

---

multi-agent clipping objective as formulated in Eq. 1
(a
n (aa1s)
T Bom amtp
min
M 5 (s, a), clip
M
(s
& (a
a (a*)
where म (a*1s) is the policy of the m agent updated at episode kt 1, the superseript of r represents Extrinsic, and M (s, a) is the joint advantage estimator of the first to m agents, which is defined as follows.
M -11m-1 A(s, a),
where π fimm1
II e is the joint policy of the first to m agents, and A(s, a) is an individual pm1
advantage estimator, such as Generalized Advantage Estimation (GAE). Additionally, Eq. 1 is also incorporated with an intrinsic reward term, i.e. the policy entropy, defined as r' = H(1 fm ss).
Method
4.1 Mutual Policy Divergence Maximization
Most existing divergence RL methods only consider state or policy divergence between episodes in a simultancous updating scheme, which lacks practicality in heterogeneous scenarios. To address this issue, We introduce the main framework of MADPO in this section, i.e. Mutual Policy Divergence Maximization (Mutual PDM), and the framework is shown Fig Tb Specifically, we consider mutual intrinsic reward which consists of two types of policy divergence: inter-agent and intra-agent policy divergence. episode kt agent maximize mutual policy divergence as follows,
A ( 1 + (1 (1-A) Dio (
Imttaal
where Dir(-) is one divergence measurement, is the coefficient to control the influence of the two divergence, and the superscript / represents Intrinsic.
The first term in Eq. is the inter-agent policy divergence, quantifying the discrepancy between policies of the current agent and the preceding agent. In heterogeneous tasks, such as multi-joint control in robotics, each agent has its own specialization. Hence, learning diversified policies for different agents is more desirable in these scenarios. By maximizing the inter-agent divergence, agents are provided with a novel optimization direction towards heterogeneity, resulting in significant diversification. Note that in the simultaneous updating manner, the inter-agent divergence maximiza- tion becomes theoretically challenging, since these methods lack access the information from the preceding agents.
The second term in Eq. is the intra-agent policy divergence, which measures the difference between the current policy and the former policy of an agent. The intra-agent policy divergence encourages agents learn new and diverse policies based on their previous policies. Consequently, we provide agents an optimization direction towards policy novelty, greatly enhancing exploration.
4.2 Conditional Cauchy-Schwarz Policy Divergence
To measure the discrepancy between policies, natural idea is to the KL-ddivergence. At episode k. agent optimizes the KL-divergence between the current policy गÊ and a fixed policy म, which can be defined as follows,
(aylss;) [(a,18))
DKL(TillT) Beympro as ax
Σ m (sy) log
(4)
H(m H((T())
(5)
where H( #) is the cross entropy between oं. and न. H(mEt) is the policy entropy. However, optimizing -il-diveeeeeee between policies raises problems regarding instability and inhibition of exploration in MARL. Specifically, when approaching 0, the fixed policy म ((ayss)) in Eq. 4 may
5

---

lead to uncontrollability and unreliability of the log term, which is common in initialization and converged phrase MARL. Moreover, the second term in Eq. [0..mmmmiizee the entropy of the current policy, which brings an opposite optimization direction, thus adversely affecting exploration. To address these issues, we introduce Conditional Cauchy-Schwarz Divergence for policy divergence maximization.
The Conditional CS divergence, recently proposed by Yu al. [2023] an extension from classic CS divergence for quantifying the discrepancy between two conditional distributions. Formally, given random variable A and with finite data set, the CS inequality is defined follows,
/f pla)g(a)da = </Dlaa)Pd (q(()) da,
(6)
where p(a) and g(a) the probability density functions. By leveraging Eq. 6 can obtain the CS (J p(a)ola))²
derive the conditional CS divergence two policies, i.. the action distributions conditioned by the states, defined as follows, ((ss(T(as)]|[(())) [
divergence, defined Dcs - 2/2 log pa(a) [' (a) Similarly,
2 log
2 SA (aals)dT)) (Js SA (a(as)ddT)
--2 log (1 (alls) + log (g/ / x² (a(ca))d)) + log (1s/A 7 (a(()s)
We then present some desirable properties of Eq Proposition 1. Given policy to be updated π and a fixed policy म., and a-order Rényi policy entropy (T) -T-a log SA T" (als)da T-a Beos v log aa-1aaa-0 (a|s), then we have:
H2(T)) + H2((T) 2 Des (T|*).
(8)
Proofs can be found in Appendix A. Proposition. is motivated by [Li et al.) and indicates that the CS divergence between distributions is lower bound of the sum of 2nd Rényi entropy of distributions. Consequently, in MARL, maximizing the CS divergence between a target policy and a fixed policy behaves like maximizing the Shannon policy entropy Yuan et al. 2023) In this way, by taking the conditional CS divergence into account, agents encouraged enhance their policy entropy while diversifying their policies. Thus, maximizing the CS policy divergence can provide agents with 2nd-Rényi cerropy-guided exploration incentives.
entropy of the target policy, which is generalized form of
Proposition 2. Given policy to be updated π and fixed policy with a finite action set A {ao...., an} ) at state s, then the CS divergence lower bounded by:
Dcs DCs(TII 2 - loog n.
Proofs can be found in Appendix A.2 Recall that in Eq. is obvious that the KL-divergence is unstable when the probability of one action approaching 0, a common occurrence in MARL. In contrast, Proposition. 2 demonstrates that the CS divergence has a deterministic lower bound unless the number of actions approaches infinity, which is not feasible in practical MARL. Even if in continuous action tasks, the trajectories sampled from policies have finite actions. Thus, maximizing the CS divergence can provide a more stable guidance for policy optimization.
4.3 Multi-Agent Divergence Policy Optimization
We first present the overall optimization objective of MADPO this section. At episode k. agent in MADPO maximizes the practical objective as follows,
[(r'(a'ss)) = = (m)((aa)ss) ss) + Dcs(TT | :0)
1-
(10)

---

where (-s(-1l-) is the estimator of the conditional CS divergence, and is the parameter of the estimator. Given trajectories T" { of .... s a, and T" {sō aj ... s a sampled from fixed policy т and the current policy The empirical estimator of Eq. 7 can be formulated by using kernel density estimation:
A
)) log
n
ST &
Des(T(a(s](+(dss) log
أラ=N )
Σ
log
==
log Σ
(1I)
where S" and A represent the Gram matrices of states and actions sampled from the policy m: ; = k(S - x3). k(aa- -aa) where x(.) is Gaussian kernel denoted as x(+))))))))) exp( and is the parameter of s(-). Moreover, S and A - represent the Gram matrices from distribution (i.e. policy) क to distribution मं formulated as S = k(s $ Detailed proofs can be found in Yu et al. 2023
|1:1
-n
In contrast existing sequential methods. starting from the second agent in the first episode. MADPO maintains the buffer data for more time. Specifically, for mutual policy divergence maximization, when finished training episode k. MADPO maintains the minibatch of the updated k-th policies for one more episode. We summarize the whole algorithm in Algo.
Algorithm 1 Multi- Divergence Policy Optimization
Input: Initial joint policy To Tb X To parameters of Mutual PDM, a and X.
1: for iteration & = I.., K do
2: Collection trajectories Ta (...,.. Th) by running H& m..
ak.
3: Restore Tç into the buffer.
4: Compute the advantage Á(s, a) by using the network.
5:
for agent = I,..., do
6: not then
7:
Compute the inter-agent divergence 1= Des(T) via trajectories Tk i-1 [T
T and
8:
end if
9:
Compute the intra-agent divergence
via trajectories T- T and
Eq.
10:
Compute T rf mutual and update the actor network by maximizing Eq. 10
11:
Compute the joint advantage via Eq. 2
12:
end for
13:
Update the V network.
14:
Delete T from the buffer.
end for
5 Experiments
We evaluate the proposed MADPO on two challenging multi-agent heterogeneous environments, Multi-Agent Mujoco (MA-Mujoco) de Witt et al., 2020) and Bi-DexHands [Chen al.) 2022) Multi-Agent Mujoco a complex and widely used task which necessitates up to 17 different joints of one robot coordinate for human-like behavior imitation, such running and walking. Bi-DexHands is a bimanual dexterous manipulation environment, where agents are in control of fingers, hands or joints. Sub-scenarios in Bi-DexHands require agents to collaborate for more complex bimanual tasks, such as opening door inward and outward, passing an item from one hand to another. We compare MADPO with state-of-the-as MARL algorithms, including one simultaneous method MAPPO (Yu et al. 2022 and sequential methods, such as HATRPO (Kuba et al.) 2022), HAPPO (Kuba et al. 2022 and A2PO [Wang et al., 2023a]. Clearly, different agents the two benchmarks should learn diversified policies. Hence, we switch off the parameter sharing setting for HAPPO, HATRPO, A2PO and our MADPO, and keep sharing parameter MAPPO. this work, we conduct experiments of random seeds 10 scenarios of MA-Mujoco and 10 scenarios of Bi- Gi-DexHands.

---

We also conduct statistical testing experiments by using rliable [Agarwal et al. 2021] Since the environments we used in this work do not have round end score, we choose the aggregate interquartile mean (IQM) sample efficiency test of rliable for evaluation. The interquartile mean (IQM) computes the mean scores of the middle 50% runs, while discarding the bottom and top 25%. Here, we evaluate the performance across multiple tasks, and the total number of runs is n X m, where r is the number of trials for one task, and n = this paper. m is the number of tasks. IQM test is more robust than the mean and has less bias than the median. The experimental details can be found in Appendix
5.1 Results on MA-Mujoco and Bi-DexHands
Fig. and Fig. show the results on MA-Mujoco and Bi-DexHands. The shaded areas represent the 95% confidence interval. we observe that MADPO consistently outperforms all baselines in MA-Mujoco, especially when the number of agents is large, indicating its efficiency highly complex scenarios. Additionally, MADPO shows superior in challenging bimanual coordination tasks in Bi-DexHandd, while other methods like HAPPO suffer from local optima due to insufficient exploration.
A2PO
13tpp3
Tesppo
2007
308
250
110
Figure 2: Performance comparison against baseline methods on Multi-Agent Mujoco. Benefiting from the heterogeneity and exploration enhanced by mutual policy divergence maximization, the proposed MADPO consistently outperforms all baselines.
Fig. 4 shows the IQM rewards comparison against other baselines. The lines in the figures represent the IQM, while the shaded areas indicate the confidence intervals. The 10 tasks in MA- MA-Mujoco include all the tasks of MA-Mujoco used and 10 tasks in Bi-Dexhands include all the tasks of Bi- Bi-Dexhands used. The 3 tasks of MA MA-Mujoco Ant include Ant-v2-2x4, 4x2, and 8x1. The 3 tasks of MA MA-Mujoco Halfcheetah include
2itchheetah-2222x3 3x2, and 6x1. The 3 tasks of MA-Mooo Walker2d include Walker2d-v2-2x3. 3x2, and 6x1. The results in Fig/Findicates that the proposed MADPO consistently outperforms the state-of-the-an MARL methods terms of best episodic reward across multiple tasks. The results also show that, MADPO has higher sample efficiency compared to other methods and achieves an improvement gap in most tasks.
5.2 Ablation Study
We also investigate the efficiency of conditional CS policy divergence compared other widely used exploration incentives as shown in Fig.5alan 5aland Fig 5b Here, no incentive presents disabling the intrinsic reward for training. We can clearly observe in Fig. Sa that the conditional CS policy diver- gence and KL-divergence achieve significant improvements compared to the popular policy entropy. These results indicate the effectiveness of mutual PDM framework, which takes the information from preceding agent into account. Additionally, the conditional CS policy divergence outperforms
8

---

Tucken SibeeteeSetbeeeiedeo
Studeeter doadd woorm
happe
rudico
Dadeon www.erCoeeeee
Sedeelir Cortpesiaa
Figure 3: Performance comparison against baseline methods Bi-DexHandk The proposed MADPO achieves superior performance compared to other MARL methods.
50 tono Dasl
Figure 4: IQM performance comparison against baseline methods on tasks of Bi-DexHands and tasks of MA-mujoco.
the famous KL-divergence empirically, particularly in MA-Mujoco tasks. This because the KL- divergence implicitly minimizes the entropy of the current policy when maximizing the divergence, which can be detrimental to exploration. In contrast, the CS policy divergence maximizes policies' novelty and, more importantly, the Rényi entropy for efficient exploration. Fig. 5bjdmmosststess that the aggregate evaluation, the CS-divergence outperforms other incentives with narrower confidence interval, indicating its better stability than KL-divergence.
Fig. 6 shows the experiments of parameter sensitivity. In this experiment, controls the influences of inter- and intra-agent policy divergence. When = O, the inter-agent policy divergence is disabled, and when À 1, the intra-agent policy divergence disabled. We can observe a consistent degradation in performance when any one aspect of the mutual policy divergence is turned off, thus confirming the significance of our method. When À 0.2, the proposed method achieves the highest reward, whereas excessive influence from inter-agent divergence with À = 0.5 is harmful. Parameter controls the kernel width in Cauchy-Schwer divergence, impacting the influence of the mutual PDM. We find that MADPO is slightly sensitive to a, behaving similarly to the entropy coefficient in MAPPO.
Conclusion
In this work, we present MADPO, sequential updating MARL method equipped with mutual policy divergence maximization for efficient exploration in heterogeneous tasks. By leveraging the sequential updating paradigm, MADPO maximizes intra-agent policy divergence to enhance exploration and inter-agent policy divergence to promote heterogeneity. However, maximizing traditional divergence measurements can lead to instability and lack of direction MARL. To tackle this issue, we propose conditional Cauchy-Schwerd policy divergence to quantify the distance between policies. The

---

(a)
ncentive
enmoay
estropy
pereeerrc
(Sdramgennce
Siea
Stga
(b)
Figure 5: Performance comparison against other exploration incentives.
Stedaad Open
1000
300
Figure 6: Parameter sensitivity studies for MADPO.
conditional Kauchy-Schware policy divergence possesses favorable properties and provides stable entropy-guided incentive for sequential exploration. We evaluate the performance of MADPO on two challenging heterogeneous tasks, MA-Mujoco and Bi-Dexhands. We observe that the proposed Mutual PDM outperforms entropy-based methods since it consider both previous and preceding information. Moreover, we verify the efficiency of the conditional Cauchy-Schwarc policy divergence in terms of stabilizing and guiding the exploration. Totally, the results demonstrate the effectivvnesss of MADPO, achieving state-of-the-art performance in complex multi-agent scenarios. The main limitation of this work is that when the number of agent increases, the proposed method may require more ram to restore previous information. We will investigate effective representation methods for previous information in the future.
Acknowledgments and Disclosure of Funding
This work is supported by the National Key R&D Program of China (2023YFB4704900) and National Natural Science Foundation of China A2044)) The authors declare that they have no conflict of interest.
10

---

References
R. Agarwal, M. Schwarzer, S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34:29304-29320, 2021.
D. Bertsekas. Multiagent reinforcement learning: Rollout and policy iteration. JEEF/CAA Journal of Automatica Sinica, 8(2):249-277 2021.
S. Bhattacharya, S. Kailas, S. Badyal, S. Gil, and D. Bertsekas. Multiagent reinforcement learning: Rollout and policy iteration for pomdp with application multi-robot problems. IEEE Transactions on Robotics, 40:2003-2023, 2023.
S. Cen, Y. Wei, and Y. Chi. Fast policy extragradient methods for competitive games with entropy regularization. Advances in Neural Information Processing Systems, (1227522279666 2021.
Chai, W. Li, Y. Zhu, D. Zhao, Z. Ma, K. Sun, and J. Ding. Unmas: Multiagent reinforcement learning for unshaped cooperative scenarios. IEEE Transactions on Neural Networks and Learning Systems, ((4))2093-21044 2021.
Y. Chen, T. Wu, S. Wang. Feng, Jiang. Z. Lu, S. McAleer, H. Dong, S.-C. Zhu, and Y. Yang. Towards human- level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems, 35:5150-5160 51600,55111000 2022.
C. S. de Witt, Peng, P.-. Kamienny, P. Torr, W. Böhmer, and S. Whiteson. Deep multi- agent reinforcement learning for decentralized continuous cooperative control. In CoRR, volume abs/2003.06709, 2020.
D. Ding, C.-Y. Wei, K. Zhang, and M. Jovanovic. Independent policy gradient for large-scale markov potential games: Sharper rates, function approximation, and game-agnostic convergence. In International Conference on Machine Learning, pages 5166-5220. PMLR, 2022.
S. Feng, H. Sun, X. Yan, H. Zhu, 7. Zou, S. Shen, and H. X. Liu. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 61 615(7953):600-677, 2023a.
Z. Feng, M. Huang, D. Wu, E. Q. Wu, and C. Yuen. Multi-agent reinforcement learning with policy clipping and average evaluation for uav-assisted communication markov game. IEEE Transactions on Intelligent Transportation Systems, 24(12): 14281-1292111 2023b.
J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
D. Guo, L. Tang, X. Zhang, and Y.-c. Liang. An off-policy multi-agent stochastic policy gradient algorithm for cooperative continuous control. Neural Networks, 170:610-621, 2024.
Haarnoja, Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018
A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, and Y. Narang. Dextreme: Transfer of agile in-hand ma- nipulation from simulation to reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5977-5984 2023.
Z. Hong. T.-.. Shann, S. Su, Y.H. Chang, T.J. Fu, and C.-Y. Lee. Diversity-driven exploration strategy for deep reinforcement learning. Advances in Neural Information Processing Systems, volume 31, 2018,
A. P. Jacob, J. J. Wu, G. Farina, A. Lerer, H. Hu, A. Bakhtin, J. Andreas, and N. Brown. Modeling strong and human-like gameplay with kl-regularized search. In International Conference on Machine Learning, pages 9695-9728. PMLR, 2022.
J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y. Geng, Y. Zhong, J. Dai, and Y. Yang. Safety gymnasium: unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, volume 36, pages 18964-18993, 2023.

---

D. Kim, J. Shin, P. Abbeel, and Y. Seo. Accelerating reinforcement learning with malue-conditional state entropy exploration. In Advances Neural Information Processing Systems, volume 36, pages 31811-31830, 2023.
Kuba, R. Chen, M. Wen, Wen, Sun, J. Wang, and Y. Yang. Trust region policy optimisation in multi-agent reinforcement learning. In International Conference on Learning Representations, page 1046, 2022.
H. Li and H. He. Multiagent trust region policy optimization. IEEE Transactions Neural Networks and Learning Systems, pages 1-15, 2023.
H. Li, Yu, V. Francois-Lavet, and J. C. Principe. Reward-free exploration by conditional divergence maximization.
J. Li, K. Kuang, B. Wang, F. Liu, L. Chen, F. Wu, and J. Xiao. Shapley counterfactual credits for multi-agent reinforcement learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 934-942 2021.
H. Liu and P. Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems, volume 34, pages 18459-18473, 2021.
X. Liu and K. Zhang. Partially observable multi-agent rl with (quasi-) efficiency: the blessing of information sharing. In International Conference on Machine Learning, pages 22370-22419, PMLR, 2023.
W. Mao, L. Yang, K. Zhang, and Basar. On improving model-free algorithms for decentralized multi-agent reinforcement learning. In International Conference on Machine Learning, pages 15007-15049. PMLR, 2022.
W. Qiu, X. Wang. R. Yu, R. Wang. X. He, B. An, S. Obraztsova, and Z. Rabinovich. Rmix: Learning sensitvve policies for cooperative reinforcement learning agents. Advances Neural Information Processing Systems, 34:23049-23062 2021.
Radosavovic, T. Xiao, B. Zhang, T. Darrell, Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 989))eadi9579, 2024.
A. Russo and A. Proutiere. Model-free active exploration in reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
K. Su and Z. Lu. Divergence-regulatio multi-agent actor-critic. Proceedings of the 39th International Conference Machine Learning, pages
PMLR, 2022.
J. Subramanian, A. Sinha, R. Seraj, and A. Mahajan. Approximate information state for approximate planning and reinforcement learning in partially observed systems. Journal of Machine Learning Research, 23(12):1 1-123 2022.
X. Wang, Z. Tian, Z. Wan, Y. Wen, J. Wang, and W. Zhang. Order matters: Agent-by-agent policy optimization. In International Conference on Learning Representations, pages 1-35, 2023a.
Z. Wang, K. Su, Zhang, H. Jia, Q. Ye, X. Xie, and Z. Lu. Multi-agent automated machine learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11960- 11969, 2023b.
Z. Wu, C. Yu, D. Ye, J. Zhang, H. H. Zhuo, et al. Coordinated proximal policy optimization. In Advances in Neural Information Processing Systems, volume 34, pages 26437-26448, 2021.
P. Xu, J. Zhang, and K. Huang. Exploration via joint policy diversity for sparse-reward multi-agen tasks. In Proceedings of the Thirty-Secome International Joint Conference on Artificial Intelligence, pages 326-334, 2023.
Yang, H. Qu, Fu, W. Hu, and Y. Zhao. A maximum divergence approach to optimal policy in deep reinforcement learning, IEEE Transactions Cybernetics, 53(3): 1499-11199 2021.
12

---

C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo cooperative multi-agent games. In Advances in Neural Information Processing Systems, volume 35, pages 24611 16611-200 1-24624, 2022.
Yu, H. Li, S. Løkse, Jenssen, and C. Príncipe. The conditional cauchy- schuhh- divergence with applications to time-series data and sequential decision making. arXiv preprint arXiv:2301.08970, 2023.
M. Yuan, M.-O. Pun, and D. Wang. Rényi state entropy maximization for exploration acceleration in reinforcement learning. IEEE Transactions on Artificial Intelligence, 4(5):1154-111 2023.
W. Zhan, S. Cen, B. Huang, Y. Chen, J. D. Lee, and Y. Chi. Policy mirror descent for regularized reinforcement learning: generalized framework with linear convergence. SIAM Journal on Optimization, 33(2): 1011)1106--191, 2023.
R. Zhang, Q. Liu, Wang, C. Xiong, N. Li, and Y. Bai. Policy optimization for markov games: Unified framework and faster convergence. Advances in Neural Information Processing Systems, 35:21886 21899, 2022a.
Y. Zhang, Q. Yang. D. An, D. Li, and Z. Wu. Multistep multiagent reinforcement learning for optimal energy schedule strategy of charging stations in smart grid. IEEE Transactions on Cybernetics, 53 (7):4292-434)) 4305, 2022b.
Y. Zhao, Z. Yang. Z. Wang, and J. Lee. Local optimization achieves global optimality in multi-agent reinforcement learning. In International Conference on Machine Learning, pages 22200-42226. PMLR, 2023.
A Theoretical Analysis
A.1 Proofs of the Proposition 1
Proposition 1. Given a policy be updated π and fixed policy and their a-order Rényi entropy (a( = I-a log / TT" (als)da = Euna log n." - (als), then we have:
1
H2((T) + H2( 2 Des (T|²).
(12)
Proof. At state 8, consider A as the action set, a is the random variable, we can rewrite the right side of Eq. 12 follows,
1 T(a= a al))da)
Dos (T| log (A T- a als)da)(( 72 als)da)
(13)
1 =- T(a als) als)da) H2( H2(
(14)
where the first term in Eq. i4 is the 2nd-order Rényi cross entropy between and Thus, using Gibbs inequality, we have
Hz( + H2( 2 log( / 7(a = al als)da),
(15)
A
2 H2(TT) + [2((7) 2 - -log(
Taa al als)da) (22(T) H2)(7),
(16)
A
2 H2(TT) 2 H2(5)) 2 Des (T|²),
(17)
which finish the proof.
13

---

A.2 Proofs of the Proposition 2
Proposition 2. Given a policy to be updated π and a fixed policy with a finite action set fsu...., S at state then the CS divergence of is lower bounded by:
(T 2 - log n.
(18)
Proof. For two policies represented by the trajectories, the CS divergence between them state is defined as follows,
~asa==ls) aas)
(cs( = log
(19)
V E A T(a als)) a(a als)
VAAT(a
Consider the quadratic mean of the r(a = =|ss),
Σ T(ass)
Σ n π(a|s)
(20)
n²
Hence,
Σ x [alss)² 2
(21)
We also have Σd T(== als)F(a = a|s) S 1. and then
Dcs(T]|F) 2 log
1/n = loog
(22)
which complete the proof.
B Experimental Results
B.1 Detailed experimental Settings
Table 1: Common hypermeters in MA-Mujoco
hyperparameters
MA-Mujoco
activation
ReLu
batch size
4000
gamma
0.99
gain PPO epoch
0.01
episode length
200
rollout threads
20
Table Different hyperparameters in MA-Mujoco
Tasks
hidden layer
actor
critic
clip
X
Ant-v2-2x4
[64,64]
5c-4
5e-4
0.2
0.5
lc3
Ant-v2-4x2
[64.64]
5e-4
5c-4
0.2
0.5
lc3
Ant-v2-8x1
[64.64]
5e-4
5e-4
0.2
0.5
le3
HalfCheetah-v222x3
[64,64]
Se-4
Se-4
0.2
0.2
1e3
HalfCheetah-v2-3x2
[64,64]
5e-4
5e-4
0.2
0.2
le3
HalfCheetah-v2-6xl
[64,64]
5e-4
5e-4
Walker2d-v2-223
[256,256]
le-3
le-3
0.05
0.1
2e3
Walker2d-v2-3x2
[256,256]
lc-3
le-3
0.05
0.1
2c3
Walker2d-v2-6xl
[256,256]
le-3
le-3
0.05
0.1
2e3
Hu2anoid-v2-17x
[256,256]
5c-4
5c-4
0.1
0.5
1c3
14

---

Table Common hypermeters Bi-DexHands
hyperparameters
BiDexHands
activation
ReLu
batch size
4000
gamma
0.99
gain
0.01
PPO epoch
episode length
75
n rollout threads
128
hidden layers
[256,256,256]
clip
0.2
actor Ir
5e-4
critic
5e-4
Table 4: Different hyperparameters in Bi-DexHand
Tasks
Shadow HanddoowaaBllkkSttckkk
0.2
5e2
Shad
0.2
5e2
0.2
5e2
Shadow Hadddaaoaadaooonnsseeeaaaaa
0.5
5e2
0.5
5e2
ShadooFFammCCccchoor2ZZmeeeez
0.2
1e3
ShadowWaandCncchUndeeaam
0.2
1e3
ShadowHanndCatchAbbrees
0.2
1e3
ShadowwEanddDoorrossseettttttt
0.5
1e3
Shadow/landDDorOQeeennvvvvwa
0.5
1e3
In this experiment, we follow the official implement and hyperparameter settings of HAPPO and HATRPO Kuba et al. 2022),, MAPPO Yuu al., 2022). and A2PC [Wang et al. 2023a]. We compare the proposed method with baselines Mujoco and Bi-DexHandy For MA-Mujoco, the commom hyperparameter are listed in Tab. [ and the diffferntt hyperparameters in each scenarios are listed in Tab. 2 For Bi- Bi-DexHands, the commom hyperparameter are listed in Tab. 3 and the different hyperparameters in each scenarios are listed in Tab. 4 The experiments were conducted PC with NVIDIA RTX3090 GPU, Intel Xeon 64-core CPU, and 64GB Ram.
popular heterogeneous environments, MA-
B.2 Additional Results
Full results of 10 scenarios in MA-Mujoco and 10 scenarios in Bi-DexHands are shown in Fig. 7 and We can make two observations on results MA-Mujoco tasks. First, MADPO demonstrates superiority in terms of both reward maximum and learning speed, highlighting its effectiveness in exploring novel policies. Second, MADPO exhibits the lowest variance compared to other methods in most scenarios, indicating its training stability. In the more challenging Bi- D--exHandd tasks, we find MADPO outperforms baselines most scenarios, confirming the effectiveness of MADPO in complex coordination tasks.
Fig. g
We also conduct the experiments of different updating orders in Ma-Mujoco, as indicated in Fig Here, each agent controls one joint of one leg, and the joints in the same position on the legs have the same specilization. The Rand. order represents updating agents randomly. The Def. order represents updating agents in the default order in Ma-Mujoco, where agents are grouped according to the legs
http::///giihbb.booo///MMMMMMMMP
15

---

Sacsud 40
A210
Figure Performance comparison on against baseline methods on Multi-Agent Mujoco scenarios.
they belong to. For example, the default updating order can be: right thigh joint, right leg joint, right foot joint, left thigh joint, .... The Spec. order represents updating agents according their specilization, and can be: right thigh joint, left thigh joint, right foot joins, foot joint, ... We can observe that the random updating order outperforms the other orders. We believe that this because our framework can benefit from various updating orders. For example, if the current agent share the same specilization the preceding agent, maximizing the inter-agent divergence enhance exploration. On the other hand, if the current agent differs from the preceding agent, maximizing the inter-agent divergence can promote heterogeneity.
In Fig 10 we compare the performance between different parameter settings using IQM aggregate test. We can observe that MADPO is little sensitive to parameter However, it outperforms HAPPO in a reasonable range of a. Additionally, we can also individually tune a for special task for further performance improvement. We also observe that MAPDO is little sensitive to the parameter X, yet consistently shows better performance than HAPPO.
Tab. 5 shows the running time of MADPO and other methods. Compared to MARL baselines, MADPO only introduces a negligible extra time cost.
C Social Impacts
We do not foresee an obvious negative impart by conducting the experiments included in this work, since we evaluate methods controlled simulation environment. However, We are also
aware that recent works (Radosavovie al. 2024 Handa et al. 2023 tested RL algorithms
16

---

Shide Tadeeeeleaaat
rapppo raudpo
8
5
2
2
Figure 8: Performance comparison on against baseline methods on 10 Bi-DexHands scenarios.
Figure Performance comparison of different updating orders on Ma-Mujoco scenarios.
20 sa in Corhande
23 taris in Marsaloos
1631
Figure 10: Aggregate parameter sensitivity study.
17

---

Table 5: Wall time comparison.
Task
training steps
A2PO
НАРРО
HATRPO
MAPPO
MADPO
Ant-v2-2x4
le7
1h2m
1h3m
1h16m
1h10m
1h17m
Walker2d-v2-6rs 1
le7
1h57m
2h1m
2h35m
1h49m
2h13m
HaliCheetah-vv26xx
le7
2h3m
1h56m
2h22m
1h40m
2h27m
Humanoid-v2-17xl
le7
6h20m
6h8m
6h5 Im
6h16m
7h3m
ShadoovaaandooooOpeeellaaad
2e7
1h48m
1h39m
2h27m
2h45m
ShabowwHaaddDooroppennOvaaa
2e7
1h50m
2h4m
2h7m
3h12m
real-world environments. This may raise concerns regarding the potential personal hazards caused by agent exploration in real world. To address this issue, one possible approach is to define safety behaviours to restrict the actions of agents [Feng et al. 2023al or perform evaluation safe simulation preliminarily.
environments [Ji et al. 2023]
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We confirm that we accurately and clearly claim the contributions and the scope this work in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the main limitation of this work in Conclusion 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
We have provided the detailed proofs in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have described the detailed steps of the proposed method in Algo. and the experiment settings in Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
The code is available at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee //nips.com / cassiicnncccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We have provided the experimental details in Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experiment results include the mean and the 95% trust interval of five random seeds, as indicated Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the compute resource required in the experiments in Ap- pendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We confirm that our research fully complies with the NeurIPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts of this work in Appendix C
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work does not poses risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have provided the URL of assets used in this work in B.I
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We do not release new assets in this work.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not include any crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not include any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "tULqsMfWlJ", "reviewer": "Reviewer_sHA3", "summary": "This paper introduces a novel multi-agent reinforcement learning (MARL) method called Multi-Agent Divergence Policy Optimization (MADPO), which enhances exploration and heterogeneity through a mutual policy divergence maximization framework. MADPO leverages a sequential updating scheme and quantifies discrepancies between episodes and agents, termed intra-agent divergence and inter-agent divergence, respectively. To address the instability and lack of directionality in traditional divergence measurements, the paper proposes using conditional Cauchy-Schwarz divergence to provide entropy-guided exploration incentives. Experiments demonstrate that the proposed method outperforms state-of-the-art sequential updating approaches in two challenging multi-agent tasks with various heterogeneous scenarios.", "questions": "1. Could you provide a detailed comparison between your proposed MADPO method and the approach presented in \"Maximum Entropy Heterogeneous-Agent Reinforcement Learning\" by Liu et al.? Specifically, how does MADPO improve upon or differ from this method in terms of handling heterogeneous sequential decision-making scenarios?\n\n2. I may have missed some details, but could you clarify whether the intrinsic reward method in MADPO ensures that the trained policies remain consistent with those optimized solely based on the original rewards?", "limitations": "The authors have addressed the limitations of their work and discussed potential negative societal impacts in accordance with the guidelines.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Innovation**: The paper introduces MADPO, a novel MARL method that enhances agent exploration and heterogeneity through mutual policy divergence maximization. \n   \n2. **Theoretical Foundation**: The use of conditional Cauchy-Schwarz divergence to address instability and directionality in traditional divergence measurements is a contribution.\n   \n3. **Experimental Validation**: The experiments conducted on two challenging multi-agent tasks with different heterogeneous scenarios convincingly demonstrate the effectiveness and superiority of MADPO in enhancing exploration and heterogeneity.", "weaknesses": "1.  The paper lacks analysis and comparison with relevant literature on sequential decision-making, such as:\n   - Liu J, Zhong Y, Hu S, et al. Maximum Entropy Heterogeneous-Agent Reinforcement Learning[C]//The Twelfth International Conference on Learning Representations.  (This paper extends SAC to heterogeneous sequential decision-making scenarios, and the relationship between this work and the current paper remains unclear.)\n\n2. It is unclear whether the intrinsic reward method proposed in this paper can ensure that the resulting trained policies are consistent with the original policies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive suggestions. We address your concerns as follows:\n## (1) Comparison with HASAC\nLiu et al. proposed Heterogeneous Agent SAC (HASAC) by extending maximum entropy RL into heterogeneous MARL [1]. However, we would like to clarify that our MADPO is an on-policy MARL method, while HASAC is based on an off-policy manner. Thus, HASAC naturally enjoys higher sample efficiency than other on-policy sequential updating methods, such as HAPPO, HATRPO and A2PO. However, as an off-policy method, HASAC incurs significant time costs. \n\nIn terms of handling heterogenous tasks, HASAC shares the similar idea in HAPPO, decomposing the joint Q function (joint advantage function in HAPPO) to individual ones conditioned by other agents’ policies. Nevertheless, HASAC lacks an active optimization objective to guide agents toward greater heterogeneity and diversity. In terms of enhancing exploration, HASAC adopts the strategy of SAC by incorporating a policy entropy term into the reward. In contrast, MADPO actively maximizes the stable mutual divergence of policies while implicitly maximizing the policy entropy, providing a stronger incentive for exploration and heterogeneity, as presented in Section 4 of the original manuscript. \n\nTo comprehensively compare MADPO with other sequential updating approaches, we conducted overall performance and wall time experiments for HASAC and our method. Please refer to the author rebuttal and the attached PDF for more information. We can observe from Table 1 and Figure 4 in the attached PDF that, although HASAC achieves a performance improvement, its huge time costs cannot be ignored. \n\nAdditionally, we recognize that the proposed mutual policy divergence maximization can be adapted to off-policy methods as well, which is our future research direction. \n\n## (2) Consistency between policies trained on MADPO and the original reward\n\nAs an intrinsic reward, mutual PDM in MADPO aims to apply incentives when the change of extrinsic reward extrinsic rewards does not lead to policy improvements. Therefore, in MADPO, when the episode reward is increasing, the original reward part is dominant, resulting in policies that keep consistency with those trained on extrinsic rewards. On the other hand, when the reward struggles to improve, indicating agents should explore the environment, the intrinsic reward provides guidance to encourage agents to escape from suboptimal policy. In such cases, the policy generated may deviate from the original policy. To tackle this issue, we use parameter $\\sigma$ to control the scale of the intrinsic reward, ensuring consistency between policies trained on the original and proposed rewards. \n\nWe have conducted the experimental comparison with the $no$ $incentive$ setting. We choose here the interquartile mean (IQM) across tasks as the metric. Please refer to Figure 2 of the attached PDF in author rebuttal above. The results indicate that mutual PDM in MADPO leads to significant policy improvements over the original reward. We have also conducted the parameter sensitivity experiments for $\\sigma$, revealing that the mutual PDM with a reasonable choice of $\\sigma$ will not cause performance degradation. \n\n[1]  Liu, Jiarong, et al., \"Maximum Entropy Heterogeneous-Agent Reinforcement Learning\", ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bgh8GeHSl3", "reviewer": "Reviewer_jcAK", "summary": "This paper is situated in the problem setting of heterogeneous cooperative agents, under the sequential update framework. The paper introduces the novel MADPO algorithm, in which agents maximize the Cauchy Schwarz divergence between agents and between episodes of data gathered by the same agent, to improve exploration. Empirical validation is performed on the Multi-Agent Mujoco and Bi-DexHands benchmark suites, demonstrating that the MADPO outperforms baselines.", "questions": "1. How sensitive is the algorithm to the scale of the divergence rewards? Have you done a study on this? \n2. On the intra-policy divergence: is the policy updated every episode? If not, then wouldn't the intra-policy divergence reward often be 0? \n3. Line 180 states that it would be challenging to define an inter-agent divergence in the simultaneous update scheme. Why not consider the divergence between $\\pi^i_k$ and $\\pi^j_{k-1}$? But this does not seem any more challenging to compute, and can be computed under  CTDE assumptions.\n4. Would it be possible to implement this exploration scheme in the CTDE setting? If so, it would be interesting to see how well the method performs. \n5.   Proposition 2 states that the CS divergence has a lower bound. Does it also have an upper bound?", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is clear, succinct, and the main idea is clear and easy to understand. The format, and figures are good, with all expected components included. The idea of maximizing the inter/intra agent divergences is intuitively appealing. Further, the authors address the pitfalls of naively maximizing intra-agent divergences by adopting the Cauchy Schwarz divergence. It's especially nice that maximizing the CS divergence implies maximizing the policy entropy as well. Experiments are done on a large number of tasks, with comparisons against expected baselines and parameter sensitivity analyses all present.", "weaknesses": "1. The motivation of the paper is not altogether clear to me. The paper seems to suggest that exploration is more challenging in the sequential update setting, necessitating devoted algorithms. Why would this be the case? \n2. In many of the presented domains, the improvement of MADPO over the next best method is not very large. Sometimes, confidence intervals of MADPO overlap those of the next best method. Can the authors provide statistical significance tests for the main results in Figures 2 and 3, comparing MADPO to the next best method?\n3. Some minor suggestions:\n- Please check your paper carefully for typos, as there are quite a few: \n    - Line 89: \"connecting link dimension curse\"? Not sure what this is\n    - No period after Figure 4\n    - Trust interval -> confidence interval \n    - Lacking 'and' at line 174\n    - Line 204: conditoned -> conditioned\n    - Line 216: extra \"of\" \n- Please be sure to state the number of trials in the main text. It is mentioned in the Neurips checklist, but I could not find it  in  the main text\n- Please make the colors of the methods the same for both domains (i.e. pick 1 color for MADPO and be consistent with it)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to express our gratitude for your careful review of our work, as well as for your positive comments and insightful suggestions.We address your concerns as follows: \n\n## (1) Motivation of our work\n\nThe sequential updating scheme offers a novel solution to heterogeneous MARL, enabling agents to access information from preceding ones. \n\nOur focus is on designing a sequential exploration strategy within this framework. Simply applying simultaneous exploration approaches, such as entropy or intra-agent divergence, to sequential methods may not fully leverage the available sequential information.\n\nBesides, in heterogeneous tasks, exploration should not only aim for novel state and policy discovery but also enhance heterogeneity. Existing sequential methods often disable parameter sharing to accommodate heterogeneity, which is a passive approach lacking active guidance. \n\nTherefore, in this work, we propose maximizing mutual policy divergence to actively enhance exploration and heterogeneity.\n\n## (2) Statistical significance tests\n\nWe believe that results presented with confidence intervals are relatively convincing in terms of statistical significance. To further evaluate the overall performance of MADPO, we utilize ***rliable***, a powerful statistical testing tool recommended by Reviewer 3yBc. We choose the aggregate interquartile mean (IQM) across multiple tasks as the metric for the sample efficiency test. Please refer to the author rebuttal and the attached PDF for more information. \n\nFigure (1) in the attached PDF shows that MADPO outperforms other baselines in terms of sample efficiency with a significant improvement gap in most tasks, and achieves the highest best episodic rewards in all tasks consistently. \n\n## (3) Minor suggestions\n\nThank you for pointing out the typos in our paper; we have corrected them accordingly. We have also adjusted the figures to highlight our method in red and included all experimental details in the main text.\n\nRegarding the statements in Line 89, we would like to clarify that in MATRPO [1], proposed by Li and He, a communication method using *connecting links* is proposed for sharing information among agents. However, as the number of agents increases, the additional cost of these connecting links cannot be ignored.\n\n## (4) Sensitivity to the scale of divergence reward\n\nIn CS divergence, the kernel width $\\sigma$ controls the scale of divergence. We conducted parameter sensitivity experiments, as presented in the original manuscript, which demonstrate that MADPO is somewhat sensitive to $\\sigma$. Additionally, we performed an IQM test to further investigate this sensitivity in the rebuttal. Please refer to the author rebuttal and the attached PDF for more information. \n\nWe can observe from Figure 2 (c) that, even though MADPO is slightly sensitive to $\\sigma$, it outperforms HAPPO in a reasonable range of $\\sigma$ across several tasks. Note that we show here the aggregate results of parameter sensitivity experiments, and we can tune $\\sigma$ in each tasks individually for further improvement. \n\n## (5) Policy updating in intra-agent divergence maximization\nThe policy is updated every episode. The intra-agent policy divergence measures the difference between one agent’s current policy and its policy in the last episode. Thus, except for the first episode, once the agent’s policy is improving, the intra-agent policy divergence would not be zero.  And the policy is updating according to the divergence. \n\n## (6) Concerns about CTDE\nFirst, we would like to clarify that our MADPO is based on CTDE. Because MADPO has a global V network and multiple agent policy networks and follows CTDE settings. The centralized V network generates a V function for training agent networks, while decentralized agents interact with the environment individually, as indicated in Algorithm 1 of the original manuscript.\n\nComputing the divergence between $\\pi_k^i$ and $\\pi_{k-1}^j$ is not practically challenging indeed. However, diversifying agents in simultaneous updating methods, such as MAPPO, requires the non-parameter sharing setting, under which MAPPO will lose the monotonic improvement guarantee, even though it may bring better performance [2]. On the contrary, in sequential methods, such as HAPPO and our MADPO, the objective naturally takes the preceding information into account, and maintains the monotonic improvement guarantee in heterogeneous tasks. \n\nTherefore, we made the statement that adapting the inter-agent divergence maximization to simultaneous updating methods is theoretically challenging. Additionally, we are also aware that adapting the proposed model to value-based methods, such as QMIX, is feasible, since it does not have the monotonic improvement issue. \n\n## (7) Upper bound of CS divergence\nThe CS divergence has an upper bound. According to Proposition 1 in the original manuscript, the upper bound of CS divergence $D_{CS} (\\pi||\\overline{\\pi})$ is the sum of two 2-order Rényi entropies: $$D_{CS} (\\pi||\\overline{\\pi}) \\leq  \\frac{1}{2}\\mathcal{H}_2(\\overline{\\pi})+\\frac{1}{2}\\mathcal{H}_2(\\pi).$$ Furthermore, given a finite action set $\\pmb{A}=\\\\{a_0,...,a_n\\\\}$, the entropy $\\mathcal{H}_2(\\pi)$ has an upper bound $\\log(n)$ when $\\pi$ is a uniform distribution. Thus, the CS divergence is upper bounded by $\\log(n)$, where $n$ is the number of the actions.\n\n[1] Li, Hepeng, and Haibo He., \"Multiagent trust region policy optimization.\", IEEE TNNLS 2023. \n\n[2] Kuba, J. G., et al., \"Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning.\", ICLR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hWFW28UsTG", "reviewer": "Reviewer_3yBc", "summary": "The paper proposes a novel training objective where it encourages the policies to diverge from each other and from the previous policy under heterogeneous multi-agent tasks based on sequential recently proposed sequential policy update. It utilizes CS divergence for calculation of \"distance\" between policies for tractable and stable optimization compared to KL divergence. The evaluation is done in high-dimensional multi-agent mujoco and and bi-dexhands environments, outperforming existing state-of-the-art sequential algorithms.", "questions": "- Is it possible to have a \"cyclic\" problem where the 1st, 3rd, 5th, ... (and also 2nd, 4th, 6th, ...) policies have the same behavior despite optimizing the proposed training objective?\n- Can the authors explain why CS is chosen over Jensen–Shannon divergence (JSD)?\n- Is there a guideline for tuning the coefficients for the intrinsic rewards?", "limitations": "Minor comments\n- line 36, wrong citation format\n- line 193 and line 217, Ep. 5 should be Eq. 4", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to understand; Fig. 1 is very informative.\n- The problem of exploration under agent heterogeneity is an important problem in multi-agent learning\n- The proposed method is sound and is backed by theory", "weaknesses": "- The evaluation is hard to judge whether the proposed method is actually performs better than the baselines, this is a deal breaker. I suggest the authors also incorporate aggregate quantities from https://agarwl.github.io/rliable/\n\nI'm willing to increase the score if the authors show that the improvement is statistically significant", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments. We address your concerns as follows:\n## (1) The aggregate evaluation metrics\nThanks for your constructive suggestion. We have re-evaluated our method by using this powerful toolbox ***rliable***. Please refer to the author rebuttal and the attached pdf for more information. Since the environments used in this study does not have a round end score, we choose the aggregate interquartile mean (IQM) across tasks as the evaluation metric. The results in Figure 1 of the PDF demonstrates that, in terms of aggregate IQM, MADPO achieves better performance and has smaller confidence intervals in most tasks. We believe these results has the statistical significance. \n## (2) A cyclic similarity problem in learned policies\nIt is an interesting question, and we believe the probability of such a situation occurring is extremely low. In our MADPO, agents are encouraged to learn different policies from the preceding ones, yet are not to be guided to behave like another one. We admit that there is a chance that one agent can autonomously imitate the one before the preceding one by a divergence incentive, as you mentioned. However, the probability is extremely low due to (a) there is no direct guidance for imitation, (b) agents are also goaded to diversify based on their previous policies (the intra-agent policy maximization), and (c) the CS divergence incentive can bring further stochasticity to agents by implicitly maximizing the policy entropy.\n\n## (3) Why choose CS divergence over JS divergence\nJS divergence fixes the problem that CS divergence may explode in MARL scenarios, since it has a constant lower bound and upper bound. However, the JS divergence is not the best choice. The JS divergence between the current policy and a fixed policy is defined as follows,\n$$\\begin{aligned}\nD_{JS}(\\pi||\\overline{\\pi}) &= \\frac{1}{2}D_{KL}(\\pi||\\frac{\\pi+\\overline{\\pi}}{2}) + \\frac{1}{2}D_{KL}(\\frac{\\pi+\\overline{\\pi}}{2}||\\overline{\\pi}) \\\\\\\\\n&=\\frac{1}{2}[\\mathcal{H}(\\pi,\\frac{\\pi+\\overline{\\pi}}{2})-\\mathcal{H}(\\pi) + \\mathcal{H}(\\frac{\\pi+\\overline{\\pi}}{2},\\overline{\\pi})-\\mathcal{H}(\\overline{\\pi})]. \n\\end{aligned}$$\nIt still has the same drawback as KL divergence, i.e. minimizing the policy entropy $\\mathcal{H}(\\pi)$ when maximizing the divergence, which is harmful to exploration. \n\nOn the contrary, as indicated in Proposition 1, our method implicitly maximizes the policy when maximizing the divergence, which can bring more stochasticity to policies and benefit exploration. Hence, our method is able to provide agents with entropy-guided divergence incentives for exploration. \n\n## (4) Guideline for tuning parameters\nWe have evaluated MADPO under different parameter settings by using the aggregate IQM test of ***rliable***, and thanks again for recommending this method. Please refer to the author rebuttal and the attached PDF above for the experimental results. The results show that our MADPO achieves better performance than HAPPO in a reasonable range of $\\sigma$. \n\n Based on the empirical results, here we can give a guideline for tuning $\\sigma$ and $\\lambda$: (a) set $\\sigma$ as commonly used values, such as 1e2 or 5e2; (b) set $\\lambda$ based on the task type, but not more than 0.5.\n\n## (5) Minor suggestions\nThanks for pointing out the wrong reference and citation formats, and we have corrected them accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZRJJUP3zkJ", "reviewer": "Reviewer_gpQ4", "summary": "The authors study MARL in heterogeneous settings, where agents are not allowed to share their parameters, and make use of the sequential updating scheme under the CTDE schema. They propose a method which exploits the preceding information to improve exploration and heterogeneity sequentially. This method is equipped with a mutual policy divergence maximization framework, which utilizes the discrepancies between episodes to enhance exploration and between agents to heterogenize agents. Interestingly, the authors propose the conditional Cauchy-Schwarz divergence to provide entropy-guided exploration incentives.", "questions": "- The authors mention: \"To the best of our knowledge, there is no exploration method that can adapt to both heterogeneous scenarios with sequential updating and homogeneous scenarios with simultaneous updating\". But can the proposed method adapt to homogeneous scenarios with simultaneous updating? No experiments in such settings have been provided. Could the proposed intrinsic rewards be used to improve exploration in MARL settings with homogeneous agents?\n\n- Why do the authors use $\\lambda$ and $1 - \\lambda$ for weighting the intrinsic rewards, instead of arbitrary weights (not in a convex combination)? How important is it to the performance?", "limitations": "The authors provide limitations of their work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The problem of exploration in settings with heterogeneous agents is important in MARL and not well-explored in literature. \n\n- The paper is the first to study the effectiveness of policy divergence maximization in the sequential updating schema, upon which important related work has been built.\n\n- The paper proposed the conditional Cauchy-Schwarz (CS) divergence as an alternative to the popular KL-divergence in MARL. Such an alternation may be interesting to the broader RL community. Interestingly, unlike KL-divergence which can expload for small values of the denominator, CS divergence has a provable good lower bound ($-\\log(n)$) only dependent on the number of finite actions.\n\n- The proposed method displays good performance, in comparison to strong SOTA methods (including MAPPO, HAPPO), on benchmarks with heterogeneous agents.\n\n- The proposed framework is simple and easy-to-implement.\n\n- The paper is generally well-written and easy-to-follow.", "weaknesses": "- The improvement over the baselines (standard KL-divergence, entropy term, no incentive) does not seem to be quite consistent in the ablation study, due to (a) high variance in the results of the no incentive, and (b) very close improvement over the KL divergence baseline in terms of best episodic reward in 2 out 3 tasks.\n\n- Since the CS divergence is new in MARL and RL, a table containing the running times of the evaluated algorithms is missing. How costly is the CS divergence?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments. We address your concerns as follows:\n## (1) The improvement over baselines \nWe have re-evaluated our method against the baselines using an aggregate statistical test. We have quantified the interquartile mean (IQM) across tasks of our method and baselines. Please refer to the author rebuttal and the attached PDF for more details. \n\nIn the PDF of the author rebuttal above, Figure 2 (a) and (b)  show the performance comparison with other exploration strategies. We can observe that in terms of IQM reward across multiple tasks, the proposed method consistently outperforms other methods in MA-Mujoco, with small overlap. Besides, the CS-divergence shows more stability and has smaller confidence intervals in MA-Mujoco compared to other methods. In the Bi-Dexhands environment, it shows a significant improvement gap in the last 10m steps and achieves the highest best episodic reward. \n\n## (2) The running time comparison\nWe have compared the running time of our methods and baselines. Please refer to the author rebuttal and the attached PDF for more details. \n\nFrom Table 1 in the attached PDF in the author rebuttal above, we observe that compared to sequential MARL baselines, MADPO only introduce a negligible extra time cost. \n\n## (3) Adapt to homogeneous tasks\nHomogeneous scenarios can be regarded as a special case of heterogeneous scenarios. Thus, yes, our method can be adapted to homogeneous scenarios. Existing methods ensure homogeneous agents by enabling parameter sharing. Under such a setting, the intra-agent divergence maximization is unnecessary, since agents should receive a homogeneous intrinsic reward. Consequently, when setting $\\lambda$ to $0$, the Mutual PDM can be adapted to homogeneous tasks with simultaneous updating. \n\nDue to the page limit of the attached PDF, we present here the following table, which shows the aggregate IQM best episodic rewards comparison across $10$ tasks and $5$ random seeds in MA-Mujoco, between different exploration strategies when adapted to MAPPO. Here, we disenable the parameter sharing. The results show that as an exploration incentive in MAPPO, the Mutual PDM achieves better performance than the entropy, indicating its effectiveness when adapted to homogeneous agents with simultaneous updating. \n| Steps| MAPPO (entropy)| MAPPO (Mutual PDM with $\\lambda =0$)|\n| :---- | :-----------: |:--------:|\n| 2e6| $\\pmb{1567.87(\\pm138.48)}$|  $1431.23(\\pm89.18)$|\n| 4e6| $2139.04(\\pm248.98)$|  $\\pmb{2494.59(\\pm201.06)}$|\n| 6e6| $2594.42(\\pm356.38)$|  $\\pmb{2954.39(\\pm389.27)}$|\n| 8e6| $3032.69(\\pm387.55)$|  $\\pmb{3228.39(\\pm231.90)}$|\n| 1e7| $3577.72(\\pm145.59)$|  $\\pmb{4155.31(\\pm283.54)}$|\n\n## (4) Why $\\lambda$ and $1-\\lambda$ for weighting\nWe use $\\lambda$ and $1-\\lambda$ here to balance the two components of mutual policy divergence maximization for the following reasons. \n\nFirst, it is necessary to scale the two types of divergences to the same range since they share the same range of values. In different tasks with different requirements for heterogeneity and exploration ability, adjusting the degree of the two components is necessary. \n\nEven in tasks requiring heterogeneity and exploration ability both highly, we can tune the other parameter $\\sigma$ to adjust the scale of the mutual divergence. Therefore, jointly tuning $\\lambda$ and $\\sigma$ can achieve the desired parameter combinations, and is sufficient for MADPO to adapt to different tasks. Compared to arbitrary weights, this parameter setting facilitates parameter tuning, since we can use $\\sigma$ to control the sacle of mutual divergence and within use $\\lambda$ to weighting the two parts. \n\nAdditionally, using other weighting methods, such as no-linear weighted mean or boosting learning, may introduce extra parameters and computational costs. Due to the limited time for rebuttal, we will investigate the performance of different weighting methods in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their encouraging comments and constructive feedback. We are glad to note that the reviewers recognized our work as innovative, appealing and easy-to-follow *[sHA3, jcAK, 3yBc, gpQ4]*, theoretically nice and interesting to RL community *[sHA3, jcAK, 3yBc, gpQ4]*, well-organized and well-written *[jcAK, 3yBc, gpQ4]*, and experimentally effective *[sHA3, gpQ4]*. 

We report here the additional comparison to address several concerns regarding the experimental results.
We re-evaluate our method with baselines by using ***rliable*** [1], a powerful statistical testing toolkit suggested by Reviewer 3yBc. Since the environments we used in this work do not have a round end score, we choose the aggregate interquartile mean (IQM) sample efficiency test of ***rliable*** for evaluation. 

First, we compare the IQM rewards of our method across multiple tasks against other MARL baselines *[jcAK, 3yBc]*, against other exploration incentives *[gpQ4]*, and under different parameter settings *[jcAK, 3yBc]*. Then, we compare our method with the SOTA off-policy sequential method HASAC, in terms of overall performance and running time *[sHA3]*. Lastly, we present the running time results of MADPO and other baselines *[gpQ4]*. All the additional results are included in the attached PDF. 

## Aggregate IQM sample efficiency test 
The interquartile mean (IQM) computes the mean scores of the middle 50% runs, while discarding the bottom and top 25%. Here, we evaluate the performance across multiple tasks, and the total number of runs is $n \times m$, where $n$ is the number of trials for one task ($n=5$ in this paper), $m$ is the number of tasks. IQM test is more robust than the mean and has less bias than the median. The lines in the figures represent the IQM, while the shaded areas indicate the confidence intervals. 

## Aggregate IQM comparison with MARL baselines *[jcAK, 3yBc]*
Figure 1 shows the IQM rewards comparison against baselines across $10$ tasks in Bi-Dexhands and $9$ tasks in MA-Mujoco. The $10$ tasks in Bi-Dexhands and include all the Bi-Dexhands tasks used, and the $10$ tasks in MA-Mujoco include all the MA-Mujoco tasks used, in the original manuscripts. The $3$ tasks of MA-Mujoco Ant include *Ant-v2-2x4*, *Ant-v2-4x2*, and *Ant-v2-8x1*. The $3$ tasks of MA-Mujoco Halfcheetah include *Halfcheetah-v2-2x3*, *Halfcheetah-v2-3x2*, and *Halfcheetah-v2-6x1*. The $3$ tasks of MA-Mujoco Walker2d include *Walker2d-v2-2x3*, *Walker2d-v2-3x2*, and *Walker2d-v2-6x1*.

We observed that the proposed MADPO consistently outperforms the state-of-the-art MARL methods in terms of best episodic reward across multiple tasks. The results also show that, MADPO has higher sample efficiency compared to other methods and achieves an improvement gap in most tasks. 

## Aggregate IQM comparison with other exploration incentives *[gpQ4]*
Figure 2 (a) and (b) show the IQM rewards comparison against other exploration incentives across $10$ tasks in Bi-Dexhands and $10$ tasks in MA-Mujoco. We observed that in MA-Mujoco, the CS-divergence outperforms other incentives with a small overlap and narrow confidence interval, indicating its better stability than KL-divergence. In $10$ tasks of  Bi-Dexhands, Figure 2 (b) shows that MADPO is the only one that keeps increasing with a significant improvement gap after 10m steps. 

## Aggregate IQM comparison under different parameter settings *[jcAK, 3yBc]*
Figure 2 (c) and (d) indicate the parameter sensitivity of MADPO across $10$ tasks in Bi-Dexhands and $10$ tasks in MA-Mujoco. We can observe that MADPO is a little sensitive to parameter $\sigma$. However, in terms of IQM, it still outperforms HAPPO in a reasonable range of $\sigma$. Note that we can individually tune $\sigma$ for each task for further performance improvement. Figure 2 (d) indicates that MAPDO is also a little sensitive to the parameter $\lambda$, yet it consistently shows better performance than HAPPO. In conclusion, we suggest that when tuning $\sigma$, frequently used values can be a good choice, and tuning $\lambda$ should consider the characteristics of the task, and do not set $\lambda$ more than 0.5 in most cases. 

## The performance and the running time comparison against on-policy baselines and HASAC *[sHA3, gpQ4]*
Here we compare MADPO with off-policy baseline HASAC on five random seeds and two tasks of Bi-Dexhands and two tasks of MA-Mujoco (due to the limited time), as indicated in Figure 3. We can observe that the off-policy HASAC has a performance improvement compared to on-policy methods. However, in Table 1, the running time of HASAC is much more than on-policy methods, which is a trade-off between time cost and performance. Table 1 also indicates that compared to baselines, MADPO only introduces a negligible extra time cost. 

[1]  Agarwal, Rishabh, et al., "Deep reinforcement learning at the edge of the statistical precipice.", NeurIPS 2021.

Meta-review: The paper presents Multi-Agent Divergence Policy Optimization (MADPO), which introduces a mutual policy divergence maximization framework aimed at enhancing exploration and heterogeneity in MARL. The method leverages a sequential updating scheme, allowing agents to learn from preceding ones, and proposes the use of conditional Cauchy-Schwarz (CS) divergence to address the stability and directionality issues in traditional divergence measurements. The paper is well-grounded in theory, with clear explanations of how the proposed method addresses these stability and directionality challenges in policy divergence measurements. The results in heterogeneous multi-agent environments demonstrate that MADPO outperforms existing methods, providing strong empirical evidence for the effectiveness of the proposed approach. The reviewers are satisfied with the addition of statistical significance tests, as well as comparisons to related works and other baselines. Based on this, the paper is recommended for acceptance.
Decision: Accept (poster)