Title: Measuring Mutual Policy Divergence for Multi-Agent Sequential Exploration
Venue: NeurIPS 2024
Accepted: yes
Keywords: multi-agent reinforcement learning

sequential updating

exploration

Cauchy-Schwarz divergence
Abstract: Despite the success of Multi-Agent Reinforcement Learning (MARL) algorithms in cooperative tasks, previous works, unfortunately, face challenges in heterogeneous scenarios since they simply disable parameter sharing for agent specialization. Sequential updating scheme was thus proposed, naturally diversifying agents by encouraging agents to learn from preceding ones. However, the exploration strategy in sequential scheme has not been investigated. Benefiting from updating one-by-one, agents have the access to the information from preceding agents. Thus, in this work, we propose to exploit the preceding information to enhance exploration and heterogeneity sequentially. We present Multi-Agent Divergence Policy Optimization (MADPO), equipped with mutual policy divergence maximization framework. We quantify the policy discrepancies between episodes to enhance exploration and between agents to heterogenize agents, termed intra-agent and inter-agent policy divergence. To address the issue that traditional divergence measurements lack stability and directionality, we propose to employ the conditional Cauchy-Schwarz divergence to provide entropy-guided exploration incentives. Extensive experiments show that the proposed method outperforms state-of-the-art sequential updating approaches in two challenging multi-agent tasks with various heterogeneous scenarios. Source code is available at \url{https://github.com/hwdou6677/MADPO}.

Measuring Mutual Policy Divergence for Multi-Agent Sequential Exploration

Abstract

Despite the success of Multi-Agent Reinforcement Learning (MARL) algorithms in cooperative tasks, previous works, unfortunately, face challenges in heterogeneous scenarios since they simply disable parameter sharing for agent specialization. Sequential updating scheme was thus proposed, naturally diversifying agents by encouraging agents to learn from preceding ones. However, the exploration strategy in sequential scheme has not been investigated. Benefiting from updating one-by-one, agents have the access to the information from preceding agents. Thus, in this work, we propose to exploit the preceding information to enhance exploration and heterogeneity sequentially. We present Multi-Agent Divergence Policy Optimization (MADPO), equipped with mutual policy divergence maximization framework. We quantify the discrepancies between episodes to enhance exploration and between agents to heterogenize agents, termed intra-agent divergence and inter-agent divergence. To address the issue that traditional divergence measurements lack stability and directionality, we propose to employ the conditional Cauchy-Schwarz divergence to provide entropy-guided exploration incentives. Extensive experiments show that the proposed method outperforms state-of-the-art sequential updating approaches in two challenging multi-agent tasks with various heterogeneous scenarios. Source code is available at https://github.com/hwdou6677/MADPO .

1 Introduction

Multi-Agent Reinforcement Learning (MARL) plays an increasingly important role in numerous real-world cooperative problems, such as smart grid management [Zhang et al., 2022b], autonomous driving [Wang et al., 2023b], unmanned system control [Feng et al., 2023b] and games [Zhang et al., 2022a]. Centralized and decentralized MARL methods have been investigated as the first two extensions from single-agent to multi-agent systems. However, challenges have arisen regarding the curse of dimensionality and non-stationary training as the number of agents increasing [Mao et al., 2022]. To address this issue, Centralized Training with Decentralized Execution (CTDE) was developed to disentangle training and execution phases [Foerster et al., 2018]. In the CTDE scheme,

the centralized critic provides global information, guiding agents during training but not during execution. CTDE significantly simplifies and stabilizes the training process, providing an effective and efficient paradigm for policy-gradient cooperative MARL.

In CTDE, agents share parameters for homogeneous tasks, such as multi-particle coordination, and then take actions sampled from the same policies. For heterogeneous tasks, such as multijoint coordination in robotic control, they learn distinct policies without sharing parameters and exhibit different behaviors. However, in these scenarios, relying solely on the non-parameter-sharing setting to achieve cooperation is an oversimplification [Bhattacharya et al., 2023]. This is because agents can never learn optimal policies that depend on trajectories from other agents when updating simultaneously. To tackle this problem, sequential updating [Bertsekas, 2021] has been proposed to improve heterogeneity and collaboration. This updating scheme originates from the insight that agents in one rollout update their policies one-by-one, rather than simultaneously, to retain preceding agent information. Several sequential methods have been proposed by leveraging the multi-agent advantage decomposition lemma[Kuba et al., 2022], the multi-agent performance difference lemma [Wang et al., 2023a], and rollout policy iteration [Bertsekas, 2021], to not only adapt the sequential updating scheme but also maintain the monotonic improvement property.

Despite the success of sequential policy updating, the exploration towards further heterogeneity improvement remains unexplored and challenging [Zhang et al., 2022a]. In MARL, agents struggle to learn globally optimal policy due to the huge exploration space complexity, which is, unfortunately, further amplified in heterogeneous tasks. Existing multi-agent exploration strategies typically require parameter sharing in homogeneous scenarios. However, when applied to heterogeneous scenarios, they suffer from performance degeneration despite employing the non-parameter sharing setting. This is because these methods fail to fully leverage the main advantage of sequential updating, i.e. the preceding information. To the best of our knowledge, there is no exploration method that can adapt to both heterogeneous scenarios with sequential updating and homogeneous scenarios with simultaneous updating.

To this end, this paper presents a novel sequential MARL framework, termed M ultiA gent D ivergence P olicy O ptimization (MADPO), where a simple yet efficient exploration strategy is equipped to enhance sample efficiency, particularly in heterogeneous scenarios. In MADPO, we first propose a Mutual Policy Divergence Maximization (Mutual PDM) strategy to heterogenize agents. Specifically, mutual PDM consists of the intra-agent divergence and the inter-agent divergence. The intra-agent divergence measures the policy discrepancy between episodes, encouraging agents to learn diversified policies. The inter-agent divergence measures the policy discrepancy between agents, enhancing heterogeneity and promoting greater diversity. However, simply applying classical divergence measures to the proposed framework may trap the exploration in local optima due to the lack of positive incentives. To address this issue, we propose to employ conditional Cauchy-Schwarz (CS) divergence to provide entropy-guided incentives. Compared to the famous Kullback-Leibler (KL) divergence, the conditional CS divergence implicitly maximizes the entropy of current policy and is more stable. The main contributions can be summarized as follows:

We develop a novel multi-agent divergence reinforcement learning model equipped with mutual policy divergence maximization, termed MADPO, to enhance exploration and heterogenize agents in heterogeneous scenarios. To the best of our knowledge, we are the first to demonstrate the efficacy of policy divergence maximization in sequential MARL.

We propose to maximize conditional Cauchy-Shwarz policy divergence to provide entropyguided incentive and stabilize multi-agent sequential exploration.

We evaluate the proposed method through extensive experiments. The results show that MADPO outperforms state-of-the-art sequential methods in two multi-joint coordination tasks with various heterogeneous scenarios.

2 Related Works

2.1 Multi-Agent Reinforcement Learning with CTDE

Benefiting from the CTDE framework, multi-agent policy gradient algorithms have paved a promising path for cooperative games [Chai et al., 2021, Qiu et al., 2021, Li et al., 2021]. For example, Wu et al. [2021] proposed CoPPO which guarantees the joint policy improvement by adapting the step size

dynamically. Yu et al. [2022] proposed Multi-Agent Proximal Policy Optimization (MAPPO) which applies PPO to multi-agent scenarios without violating the guarantee of monotonic improvement in the individual policy level. Policy entropy incentive in MAPPO is one of the most related parts to our method, providing diversified policy learning from an information theory perspective. Li and He [2023] proposed MATRPO to extend Trust Region Policy Optimization (TRPO) to multi-agent tasks through a fully decentralized setting and distributed optimization. However, when the number of agents becomes large, MATRPO may encounter the challenge of the connecting link dimension curse. This is because it relies on communication rather than global information to facilitate cooperation. Guo et al. [2024] proposed MASPG, a trust region-based MARL algorithm in the off-policy manner, to enhance the sample efficiency of trust region methods. However, these methods require homogeneity of agents, i.e. parameter sharing, to ensure monotonic improvement. This homogeneity assumption can impose significant restrictions on agents, limiting their ability to explore the joint policy space adequately [Ding et al., 2022]. Consequently, if the sharing of parameters is canceled, it can lead to violations of the monotonicity guarantee and result in performance degradation [Zhan et al., 2023].

2.2 Sequential Updating MARL

The sequential updating scheme originates from single-agent rollout and policy iteration [Bertsekas, 2021], aiming to update policies of agents one by one, as shown in Fig. 1a. This structure encourages agents to learn different policies based on information from preceding ones, thereby naturally generalizing the homogeneous MARL to heterogeneous MARL. To build the multi-agent sequential updating scheme, attempts have been addressed from both joint and individual policy perspectives. For example, Bertsekas [2021] proposed rollout and policy iteration method, which was the first to consider sequential updating in MARL. Kuba et al. [2022] observed the multi-agent advantage decomposition lemma and proposed HAPPO. Leveraging this powerful lemma, HAPPO estimates and decomposes the joint advantage function to implement sequential updating and ensure the joint monotonic improvement. On the other hand, A2PO proposed by Wang et al. [2023a] focuses on individual policy improvement by leveraging the multi-agent policy difference lemma. A2PO maintains distribution invariance during each agent's advantage estimation process and consider a more refined updating order. Zhao et al. [2023] introduced a localized action value function as the surrogate optimization objective, offering a provable convergence guarantee for multi-agent PPO.

2.3 Information Theory Induced RL

Information-theoretic principles serve as a powerful regularization technique for providing valuable guidance in intrinsic reward-driven RL [Liu and Zhang, 2023, Subramanian et al., 2022, Russo and Proutiere, 2024], including both policy and state exploration [Cen et al., 2021, Jacob et al., 2022]. For example, the Soft Actor-Critic (SAC) is the first to maximize the Shannon entropy of policies, promoting randomness and encouraging exploration [Haarnoja et al., 2018]. On-policy methods, such as PPO, MAPPO and HAPPO, embrace the same concept by incorporating entropy regularization into the optimization objective. Additionally, recent advancements have explored the utilization of various entropy forms, such as encoder estimated stable entropy [Liu and Abbeel, 2021], value conditional entropy [Kim et al., 2023] and Rényi entropy [Yuan et al., 2023] to model environmental dynamics and accelerate novel state discovery. However, maximizing entropy only introduces stochasticity for measuring uncertain dynamics. To address this limitation, policy divergence regularization between episodes [Su and Lu, 2022, Xu et al., 2023] has been proposed. This regularization method calculates the policy divergence based on a fixed policy and offers more directed guidance compared to entropy alone. Furthermore, the efficacy of state divergence in combating local optima and fostering state novelty has been demonstrated [Hong et al., 2018, Yang et al., 2021]. However, the existing divergence RL methods cannot be effectively extended to the sequential updating paradigm.

In this work, we pursue an on-policy method to enhance exploration and heterogenize agents in a sequential updating paradigm, termed M ultiA gent D ivergence P olicy O ptimization (MADPO). In contrast to the aforementioned policy divergence-based methods, we introduce a novel approach that maximizes inter- and intra-agent policy divergence, thereby incorporating policy information. To further address the deficiency of exploration direction in the traditional divergence RL, we propose to employ the conditional Cauchy-Schwarz divergence to provide an entropy-guided incentive.

T

X

2

TT

3

一

Upd

元

π

2

×

T

3

1

π

二

π

π

3

1

一

Update

元

元

元

元3

3

T

九

Mutual PDM

Update

元

MutualPDM

元

Update

Agent

Agent

元元

×

元

×

k

Episode

元

i-1

元

1

Mutual PDM

Episode

k

Update

元

3

Div(πkπk-1) +Div(πkllπk-1)

(b) Multi-Agent Divergence Policy Optimization

3 Preliminaries

3.1 MARLPorblem Formulation

In this paper, we consider a multi-agent sequential decision-making problem, which can be described as a decentralized Markov decision process (DEC-MDP). A DEC-MDP with n agents can be formulated as the tuple: ⟨S , A , r, T , γ ⟩ , where S represents the state space. We denote N = { 1 , . . . , n } as the set of finite agents. A = A 1 ×··· × A n is the joint action space by taking the product of actions spaces of n agents. T : S × A × S ↦→ [0 , 1] is the state transition function of the environment dynamics. r : S × A ↦→ R is the reward function. γ is the discount factor. At time step t , to interact with the environment, each agent at state s t ∈ S takes an action a i t from its own policy π i ( ·| s t ) to form a joint action a t = { a 1 t , . . . , a n t } and a joint policy π ( ·| s t ) = π 1 × . . . × π n . The agents then receive a joint reward r ( s t , a t ) and step to the new state s t +1 with the probability T ( s t +1 | s t , a t ) . The objective is to learn an optimal joint policy by maximizing the expected cumulative reward: ¯ π = arg max ∑ ∞ t =0 E s t ∼ ρ π , a t ∼ π [ γ t r ( s t , a t )] , where ρ π is the

π

marginal state distribution. Following Bellman Equations, the state-action value function and the state function of state s t are defined as Q π ( s t , a t ) = r ( s t , a t ) + ∑ ∞ i>t E s i ∼ ρ π , a i ∼ π [ γ i -t r ( s i , a i ) ] , and V π ( s t ) = ∑ ∞ i>t E s i ∼ ρ π , a i ∼ π [ γ i -t r ( s i , a i ) | s 0 = s t ] . And the advantage function is defined as A π ( s t , a t ) = Q π ( s t , a t ) -V π ( s t ) .

3.2 Multi-Agent Sequential Policy Updating Paradigm

Sequential updating paradigm was introduced to alleviate homogeneity in multi-agent reinforcement learning. The overview of sequential updating with a three-agent setting is shown in Fig. 1a. For instance, Heterogeneous-Agent Proximal Policy Optimisation (HAPPO) takes preceding agent information into account by employing the multi-agent advantage decomposition lemma and the joint advantage estimator [Kuba et al., 2022]. At episode k , agent m in HAPPO maximizes the extrinsic

multi-agent clipping objective as formulated in Eq. 1,

where ¯ π i m k ( a i | s ) is the policy of the m th agent updated at episode k -1 , the superscript of r E represents Extrinsic , and M i 1: m ( s, a ) is the joint advantage estimator of the first to m th agents, which is defined as follows,

where π i 1: m -1 = m -1 ∏ π i p is the joint policy of the first to m th agents, and ˆ A ( s, a ) is an individual p =1

advantage estimator, such as Generalized Advantage Estimation (GAE). Additionally, Eq. 1 is also incorporated with an intrinsic reward term, i.e. the policy entropy, defined as r I = H ( π i m ( a i | s )) .

4 Method

4.1 Mutual Policy Divergence Maximization

Most existing divergence RL methods only consider state or policy divergence between episodes in a simultaneous updating scheme, which lacks practicality in heterogeneous scenarios. To address this issue, We introduce the main framework of MADPO in this section, i.e. Mutual Policy Divergence Maximization (Mutual PDM), and the framework is shown in Fig 1b. Specifically, we consider a mutual intrinsic reward which consists of two types of policy divergence: inter-agent and intra-agent policy divergence. At episode k , agent i maximize mutual policy divergence as follows,

where Div ( · ) is one divergence measurement, λ is the coefficient to control the influence of the two divergence, and the superscript I represents Intrinsic .

The first term in Eq. 3 is the inter-agent policy divergence, quantifying the discrepancy between policies of the current agent and the preceding agent. In heterogeneous tasks, such as multi-joint control in robotics, each agent has its own specialization. Hence, learning diversified policies for different agents is more desirable in these scenarios. By maximizing the inter-agent divergence, agents are provided with a novel optimization direction towards heterogeneity, resulting in significant diversification. Note that in the simultaneous updating manner, the inter-agent divergence maximization becomes theoretically challenging, since these methods lack access to the information from the preceding agents.

The second term in Eq. 3 is the intra-agent policy divergence, which measures the difference between the current policy and the former policy of an agent. The intra-agent policy divergence encourages agents to learn new and diverse policies based on their previous policies. Consequently, we provide agents an optimization direction towards policy novelty, greatly enhancing exploration.

4.2 Conditional Cauchy-Schwarz Policy Divergence

To measure the discrepancy between policies, a natural idea is to use the KL-divergence. At episode k , agent i optimizes the KL-divergence between the current policy π i k and a fixed policy ¯ π , which can be defined as follows,

where H ( π i k , ¯ π ) is the cross entropy between π i k and ¯ π , H ( π i k ) is the policy entropy. However, optimizing KL-divergence between policies raises problems regarding instability and inhibition of exploration in MARL. Specifically, when approaching 0 , the fixed policy ¯ π ( a j | s j ) in Eq. 4 may

lead to uncontrollability and unreliability of the log term, which is common in initialization and converged phrase of MARL. Moreover, the second term in Eq. 5 minimizes the entropy of the current policy, which brings an opposite optimization direction, thus adversely affecting exploration. To address these issues, we introduce Conditional Cauchy-Schwarz Divergence for policy divergence maximization.

The Conditional CS divergence, recently proposed by Yu et al. [2023], is an extension from classic CS divergence for quantifying the discrepancy between two conditional distributions. Formally, given random variable A and S with a finite data set, the CS inequality is defined as follows,

where p ( a ) and q ( a ) are the probability density functions. By leveraging Eq. 6, we can obtain the CS divergence, defined as D CS = -1 2 log ( ∫ p ( a ) q ( a )) 2 ∫ p 2 ( a ) ∫ q 2 ( a ) . Similarly, we can derive the conditional CS divergence of two policies, i.e. the action distributions conditioned by the states, defined as follows,

We then present some desirable properties of Eq 7.

Proposition 1. Given a policy to be updated π and a fixed policy ¯ π , and α -order Rényi policy entropy H α ( π ) = 1 1 -α log ∫ A π α ( a | s ) da = 1 1 -α E a ∼ π log π α -1 ( a | s ) , then we have:

Proofs can be found in Appendix A.1. Proposition. 1 is motivated by [Li et al.] and indicates that the CS divergence between distributions is a lower bound of the sum of 2nd-Rényi entropy of distributions. Consequently, in MARL, maximizing the CS divergence between a target policy and a fixed policy behaves like maximizing the 2nd-Rényi entropy of the target policy, which is a generalized form of Shannon policy entropy [Yuan et al., 2023]. In this way, by taking the conditional CS divergence into account, agents are encouraged to enhance their policy entropy while diversifying their policies. Thus, maximizing the CS policy divergence can provide agents with 2nd-Rényi entropy-guided exploration incentives.

Proposition 2. Given a policy to be updated π and a fixed policy ¯ π with a finite action set A = { a 0 , ..., a n } at state s , then the CS divergence is lower bounded by:

Proofs can be found in Appendix A.2. Recall that in Eq. 4, it is obvious that the KL-divergence is unstable when the probability of one action approaching 0 , a common occurrence in MARL. In contrast, Proposition. 2 demonstrates that the CS divergence has a deterministic lower bound unless the number of actions approaches infinity, which is not feasible in practical MARL. Even if in continuous action tasks, the trajectories sampled from policies have finite actions. Thus, maximizing the CS divergence can provide a more stable guidance for policy optimization.

4.3 Multi-Agent Divergence Policy Optimization

We first present the overall optimization objective of MADPO in this section. At episode k , agent i in MADPO maximizes the practical objective as follows,

where ˆ D CS ( ·||· ) is the estimator of the conditional CS divergence, and σ is the parameter of the estimator. Given trajectories τ ¯ π = { s ¯ π 1 , a ¯ π 1 , ..., s ¯ π n , a ¯ π n } and τ π = { s π 0 , a π 0 , ..., s π n , a π n } sampled from a fixed policy ¯ π and the current policy π . The empirical estimator of Eq. 7 can be formulated by using kernel density estimation:

where S π and A π represent the Gram matrices of states and actions sampled from the policy π : S π ij = κ ( s π i -s π j ) , A π ij = κ ( a π i -a π j ) , where κ ( · ) is a Gaussian kernel denoted as κ ( · ) = exp( -||·|| 2 2 σ 2 ) , and σ is the parameter of κ ( · ) . Moreover, S π → ¯ π and A π → ¯ π represent the Gram matrices from distribution ( i.e. policy) π to distribution ¯ π , formulated as S π → ¯ π ij = κ ( s π i -s ¯ π j ) . Detailed proofs can be found in Yu et al. [2023].

In contrast to existing sequential methods, starting from the second agent in the first episode, MADPO maintains the buffer data for more time. Specifically, for mutual policy divergence maximization, when finished training in episode k , MADPO maintains the minibatch of the updated k -th policies for one more episode. We summarize the whole algorithm in Algo. 1.

5 Experiments

We evaluate the proposed MADPO on two challenging multi-agent heterogeneous environments, Multi-Agent Mujoco (MA-Mujoco) [de Witt et al., 2020] and Bi-DexHands [Chen et al., 2022]. Multi-Agent Mujoco is a complex and widely used task which necessitates up to 17 different joints of one robot to coordinate for human-like behavior imitation, such as running and walking. Bi-DexHands is a bimanual dexterous manipulation environment, where agents are in control of fingers, hands or joints. Sub-scenarios in Bi-DexHands require agents to collaborate for more complex bimanual tasks, such as opening a door inward and outward, passing an item from one hand to another. We compare MADPO with state-of-the-art MARL algorithms, including one simultaneous method MAPPO [Yu et al., 2022] and sequential methods, such as HATRPO [Kuba et al., 2022], HAPPO [Kuba et al., 2022] and A2PO[Wang et al., 2023a]. Clearly, different agents in the two benchmarks should learn diversified policies. Hence, we switch off the parameter sharing setting for HAPPO, HATRPO, A2PO and our MADPO, and keep sharing parameter in MAPPO. In this work, we conduct experiments of 5 random seeds on 10 scenarios of MA-Mujoco and 10 scenarios of Bi-DexHands.

We also conduct statistical testing experiments by using rliable [Agarwal et al., 2021]. Since the environments we used in this work do not have a round end score, we choose the aggregate interquartile mean (IQM) sample efficiency test of rliable for evaluation. The interquartile mean (IQM) computes the mean scores of the middle 50% runs, while discarding the bottom and top 25% . Here, we evaluate the performance across multiple tasks, and the total number of runs is n × m , where n is the number of trials for one task, and n = 5 in this paper. m is the number of tasks. IQM test is more robust than the mean and has less bias than the median. The experimental details can be found in Appendix B.

5.1 Results on MA-Mujoco and Bi-DexHands

Fig. 2 and Fig. 3 show the results on MA-Mujoco and Bi-DexHands. The shaded areas represent the 95% confidence interval. we observe that MADPO consistently outperforms all baselines in MA-Mujoco, especially when the number of agents is large, indicating its efficiency in highly complex scenarios. Additionally, MADPO shows superior in challenging bimanual coordination tasks in Bi-DexHands, while other methods like HAPPO suffer from local optima due to insufficient exploration.

HalfCheetah-v2-3x2

HalfCheetah-v2-6x1

6000

A2PO

7000

0009

happo

5000

hatrpo

6000

5000

4000

mappo

madpo

Episode rewards

5000

4000

000E

4000

Episode

3000

3000

2000

2000

2000

1000

1000

1000

0.6

0.0

02

06

02

0.o

0.6

10

Steps

1e7

Steps

1e7

Steps

1e7

Humanoid-v2-17x1

8000

6000

6000

7000

5000

5000

6000

spuema aposid3

4000

rewards

4000

3000

I aposidg

3000

apo

4000

Id=

3000

2000

2000

2000

1000

1000

1000

0.2

0.6

0.8

6.0

02

0.o

02

0.0

Steps

1e7

Steps

le7

Steps

le7

Fig. 4 shows the IQM rewards comparison against other baselines. The lines in the figures represent the IQM, while the shaded areas indicate the confidence intervals. The 10 tasks in MA-Mujoco include all the tasks of MA-Mujoco used and 10 tasks in Bi-Dexhands include all the tasks of Bi-Dexhands used. The 3 tasks of MA-Mujoco Ant include Ant-v2-2x4, 4x2, and 8x1. The 3 tasks of MA-Mujoco Halfcheetah include Halfcheetah-v2-2x3, 3x2, and 6x1. The 3 tasks of MA-Mujoco Walker2d include Walker2d-v2-2x3, 3x2, and 6x1. The results in Fig 4 indicates that the proposed MADPO consistently outperforms the state-of-the-art MARL methods in terms of best episodic reward across multiple tasks. The results also show that, MADPO has higher sample efficiency compared to other methods and achieves an improvement gap in most tasks.

5.2 Ablation Study

We also investigate the efficiency of conditional CS policy divergence compared to other widely used exploration incentives as shown in Fig. 5a and Fig 5b. Here, no incentive presents disabling the intrinsic reward for training. We can clearly observe in Fig. 5a that the conditional CS policy divergence and KL-divergence achieve significant improvements compared to the popular policy entropy. These results indicate the effectiveness of mutual PDM framework, which takes the information from preceding agent into account. Additionally, the conditional CS policy divergence outperforms

P

ShadowHandCatchOver2Underarm

ShadowHandBlockStack

ShadowHandCatchUnderarm

25

happo

hatrpo

mappo

70

80

3.0

spemalaposda

20

madpo

Episode rewards

2.5

60

2.0

15

aposidg

10

50

1.5

40

1.0

2.00

0205091095

2.00

200

Steps

1e7

Steps

1e7

Steps

1e7

ShadowHandDoorCloselnward

ShadowHandDoorOpenlnward

ShadowHandOver

100

25

80

80

20

spuemau aposid3

70

Episode rewards

Episode rewards

60

15

60

50

10

20

40

0.0

025

050

0.5

100

150

175

200

5.0

0.25

050

0.5

100

123

150

175

2.00

00505

1.00

23

150

75

2,00

Steps

1e7

Steps

1e7

Steps

1e7

10 tasks in Dexhands

3 tasks in Mamujoco Ant-v2

3 tasks in Mamujoco HalfCheetah-v2

3 tasks in Mamujoco Walker2d-v2

09

6000

6000

6000

50

000S

5000

5000

IQM Episode Reward

40

1 Episode Reward

4000

1 Episode Reward

4000

1 Episode Reward

4000

OE

3000

3000

3000

20

happo

WOI

2000

+

happo

A2PO

WOI

2000

happo

A2PO

WOI

2000

A2PO

happo

10

0.000.250.500.751.00

Steps

1.25

1.50

1.75

hatrpo

mappo

madpo

2.00

le7

1000

0.0

0.2

0.4

Steps

0.6

+

0.8

madpo

mappo

hatrpo

1.0

1000

0

0.0

0.2

0.4

Steps

9'0

8'0

mappo

hatrpo

madpo

1.0

le7

1000

Ol

0.0

0.2

0.4

Steps

0.6

0.8

hatrpo

mappo

odpeu

1.6

le7

the famous KL-divergence empirically, particularly in MA-Mujoco tasks. This is because the KLdivergence implicitly minimizes the entropy of the current policy when maximizing the divergence, which can be detrimental to exploration. In contrast, the CS policy divergence maximizes policies' novelty and, more importantly, the Rényi entropy for efficient exploration. Fig. 5b demonstrates that in the aggregate evaluation, the CS-divergence outperforms other incentives with narrower confidence interval, indicating its better stability than KL-divergence.

Fig. 6 shows the experiments of parameter sensitivity. In this experiment, λ controls the influences of inter- and intra-agent policy divergence. When λ = 0 , the inter-agent policy divergence is disabled, and when λ = 1 , the intra-agent policy divergence is disabled. We can observe a consistent degradation in performance when any one aspect of the mutual policy divergence is turned off, thus confirming the significance of our method. When λ = 0 . 2 , the proposed method achieves the highest reward, whereas excessive influence from inter-agent divergence with λ = 0 . 5 is harmful. Parameter σ controls the kernel width in Cauchy-Schwarz divergence, impacting the influence of the mutual PDM. We find that MADPO is slightly sensitive to σ , behaving similarly to the entropy coefficient in MAPPO.

6 Conclusion

In this work, we present MADPO, a sequential updating MARL method equipped with mutual policy divergence maximization for efficient exploration in heterogeneous tasks. By leveraging the sequential updating paradigm, MADPO maximizes intra-agent policy divergence to enhance exploration and inter-agent policy divergence to promote heterogeneity. However, maximizing traditional divergence measurements can lead to instability and lack of direction in MARL. To tackle this issue, we propose conditional Cauchy-Schwarz policy divergence to quantify the distance between policies. The

(a)

(b)

HalfCheetah-v2-6x1

ShadowHandDoorOpenlnward

6000

7000

KL-divergence

no incentive

CS-divergence

entropy

100

90

6000

KL-divergence

no incentive

entropy

CS-divergence

80

5000

speemau

5000

Episode rewards

Episode rewards

70

4000

Episode r

4000

60

3000

3000

50

2000

2000

1000

40

30

CS-divergence

KL-divergence

no incentive

entropy

1000

Steps

1e7

0.00

D

Steps

Lo0

00

1e7

D

Steps

1e7

10 tasks in Dexhands

10 tasks in Mamujoco

80

5000

70

IQM Episode Reward

IQM Episode Reward

4000

60

50

3000

40

2000

30

no incentive

entropy

noincentive

entropy

20

KL-divergence

1000

KL-divergence

CS-divergence

CS-divergence

0.00

0.250.500.751.00

1.25

1.50

1.75

2.00

0.0

0.2

0.4

0.6

0.8

1.0

Steps

le7

Steps

1e7

ShadowHandDoorOpenlnward

ShadowHandDoorOpenlnward

100

7000

A =0

A = 0.1

100

90

6000

A = 0.2

A = 0.5

90

Episode Reward

80

Episode Reward

5000

A =1

Episode Reward

80

70

4000

70

60

3000

60

50

50

A = 0

2000

40

30

A = 0.1

A = 0.2

A = 1

A = 0.5

1000

40

30

g=100

0=1000

=500

g=50

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.0

0.2

0.4

0.6

0.8

1.0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Steps

1e7

Steps

1e7

Steps

1e7

conditional Cauchy-Schwarz policy divergence possesses favorable properties and provides a stable entropy-guided incentive for sequential exploration. We evaluate the performance of MADPO on two challenging heterogeneous tasks, MA-Mujoco and Bi-Dexhands. We observe that the proposed Mutual PDM outperforms entropy-based methods since it consider both previous and preceding information. Moreover, we verify the efficiency of the conditional Cauchy-Schwarz policy divergence in terms of stabilizing and guiding the exploration. Totally, the results demonstrate the effectiveness of MADPO, achieving state-of-the-art performance in complex multi-agent scenarios. The main limitation of this work is that when the number of agent increases, the proposed method may require more ram to restore previous information. We will investigate effective representation methods for previous information in the future.

Acknowledgments and Disclosure of Funding

This work is supported by the National Key R&D Program of China (2023YFB4704900) and National Natural Science Foundation of China (U21A20485). The authors declare that they have no conflict of interest.

References

Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems , 34:29304-29320, 2021.

Bertsekas. Multiagent reinforcement learning: Rollout and policy iteration. IEEE/CAA Journal of Automatica Sinica , 8(2):249-272, 2021.

Bhattacharya, S. Kailas, S. Badyal, S. Gil, and D. Bertsekas. Multiagent reinforcement learning: Rollout and policy iteration for pomdp with application to multi-robot problems. IEEE Transactions on Robotics , 40:2003-2023, 2023.

Cen, Y. Wei, and Y. Chi. Fast policy extragradient methods for competitive games with entropy regularization. Advances in Neural Information Processing Systems , 34:27952-27964, 2021.

Chai, W. Li, Y. Zhu, D. Zhao, Z. Ma, K. Sun, and J. Ding. Unmas: Multiagent reinforcement learning for unshaped cooperative scenarios. IEEE Transactions on Neural Networks and Learning Systems , 34(4):2093-2104, 2021.

Chen, T. Wu, S. Wang, X. Feng, J. Jiang, Z. Lu, S. McAleer, H. Dong, S.-C. Zhu, and Y. Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems , 35:5150-5163, 2022.

S. de Witt, B. Peng, P.-A. Kamienny, P. Torr, W. Böhmer, and S. Whiteson. Deep multiagent reinforcement learning for decentralized continuous cooperative control. In CoRR , volume abs/2003.06709, 2020.

Ding, C.-Y. Wei, K. Zhang, and M. Jovanovic. Independent policy gradient for large-scale markov potential games: Sharper rates, function approximation, and game-agnostic convergence. In International Conference on Machine Learning , pages 5166-5220. PMLR, 2022.

Feng, H. Sun, X. Yan, H. Zhu, Z. Zou, S. Shen, and H. X. Liu. Dense reinforcement learning for safety validation of autonomous vehicles. Nature , 615(7953):620-627, 2023a.

Feng, M. Huang, D. Wu, E. Q. Wu, and C. Yuen. Multi-agent reinforcement learning with policy clipping and average evaluation for uav-assisted communication markov game. IEEE Transactions on Intelligent Transportation Systems , 24(12):14281-14293, 2023b.

Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Guo, L. Tang, X. Zhang, and Y.-c. Liang. An off-policy multi-agent stochastic policy gradient algorithm for cooperative continuous control. Neural Networks , 170:610-621, 2024.

Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning , pages 1861-1870. PMLR, 2018.

Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, and Y. Narang. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 5977-5984, 2023.

Z.-W. Hong, T.-Y. Shann, S.-Y. Su, Y.-H. Chang, T.-J. Fu, and C.-Y. Lee. Diversity-driven exploration strategy for deep reinforcement learning. In Advances in Neural Information Processing Systems , volume 31, 2018.

P. Jacob, D. J. Wu, G. Farina, A. Lerer, H. Hu, A. Bakhtin, J. Andreas, and N. Brown. Modeling strong and human-like gameplay with kl-regularized search. In International Conference on Machine Learning , pages 9695-9728. PMLR, 2022.

Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y. Geng, Y. Zhong, J. Dai, and Y. Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In Advances in Neural Information Processing Systems , volume 36, pages 18964-18993, 2023.

Kim, J. Shin, P. Abbeel, and Y. Seo. Accelerating reinforcement learning with value-conditional state entropy exploration. In Advances in Neural Information Processing Systems , volume 36, pages 31811-31830, 2023.

Kuba, R. Chen, M. Wen, Y. Wen, F. Sun, J. Wang, and Y. Yang. Trust region policy optimisation in multi-agent reinforcement learning. In International Conference on Learning Representations , page 1046, 2022.

Li and H. He. Multiagent trust region policy optimization. IEEE Transactions on Neural Networks and Learning Systems , pages 1-15, 2023.

Li, S. Yu, V . Francois-Lavet, and J. C. Principe. Reward-free exploration by conditional divergence maximization.

Li, K. Kuang, B. Wang, F. Liu, L. Chen, F. Wu, and J. Xiao. Shapley counterfactual credits for multi-agent reinforcement learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 934-942, 2021.

Liu and P. Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems , volume 34, pages 18459-18473, 2021.

Liu and K. Zhang. Partially observable multi-agent rl with (quasi-) efficiency: the blessing of information sharing. In International Conference on Machine Learning , pages 22370-22419. PMLR, 2023.

Mao, L. Yang, K. Zhang, and T. Basar. On improving model-free algorithms for decentralized multi-agent reinforcement learning. In International Conference on Machine Learning , pages 15007-15049. PMLR, 2022.

Qiu, X. Wang, R. Yu, R. Wang, X. He, B. An, S. Obraztsova, and Z. Rabinovich. Rmix: Learning risk-sensitive policies for cooperative reinforcement learning agents. Advances in Neural Information Processing Systems , 34:23049-23062, 2021.

Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics , 9(89):eadi9579, 2024.

Russo and A. Proutiere. Model-free active exploration in reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Su and Z. Lu. Divergence-regularized multi-agent actor-critic. In Proceedings of the 39th International Conference on Machine Learning , pages 20580-20603. PMLR, 2022.

Subramanian, A. Sinha, R. Seraj, and A. Mahajan. Approximate information state for approximate planning and reinforcement learning in partially observed systems. Journal of Machine Learning Research , 23(12):1-83, 2022.

Wang, Z. Tian, Z. Wan, Y. Wen, J. Wang, and W. Zhang. Order matters: Agent-by-agent policy optimization. In International Conference on Learning Representations , pages 1-35, 2023a.

Wang, K. Su, J. Zhang, H. Jia, Q. Ye, X. Xie, and Z. Lu. Multi-agent automated machine learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11960-11969, 2023b.

Wu, C. Yu, D. Ye, J. Zhang, H. H. Zhuo, et al. Coordinated proximal policy optimization. In Advances in Neural Information Processing Systems , volume 34, pages 26437-26448, 2021.

Xu, J. Zhang, and K. Huang. Exploration via joint policy diversity for sparse-reward multi-agent tasks. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence , pages 326-334, 2023.

Yang, H. Qu, M. Fu, W. Hu, and Y. Zhao. A maximum divergence approach to optimal policy in deep reinforcement learning. IEEE Transactions on Cybernetics , 53(3):1499-1510, 2021.

Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo in cooperative multi-agent games. In Advances in Neural Information Processing Systems , volume 35, pages 24611-24624, 2022.

Yu, H. Li, S. Løkse, R. Jenssen, and J. C. Príncipe. The conditional cauchy-schwarz divergence with applications to time-series data and sequential decision making. arXiv preprint arXiv:2301.08970 , 2023.

Yuan, M.-O. Pun, and D. Wang. Rényi state entropy maximization for exploration acceleration in reinforcement learning. IEEE Transactions on Artificial Intelligence , 4(5):1154-1164, 2023.

Zhan, S. Cen, B. Huang, Y. Chen, J. D. Lee, and Y. Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. SIAM Journal on Optimization , 33(2):1061-1091, 2023.

Zhang, Q. Liu, H. Wang, C. Xiong, N. Li, and Y. Bai. Policy optimization for markov games: Unified framework and faster convergence. Advances in Neural Information Processing Systems , 35:21886-21899, 2022a.

Zhang, Q. Yang, D. An, D. Li, and Z. Wu. Multistep multiagent reinforcement learning for optimal energy schedule strategy of charging stations in smart grid. IEEE Transactions on Cybernetics , 53 (7):4292-4305, 2022b.

Zhao, Z. Yang, Z. Wang, and J. D. Lee. Local optimization achieves global optimality in multi-agent reinforcement learning. In International Conference on Machine Learning , pages 42200-42226. PMLR, 2023.

A Theoretical Analysis

A.1 Proofs of the Proposition 1

Proposition 1. Given a policy to be updated π and a fixed policy ¯ π , and their α -order Rényi entropy H α ( π ) = 1 1 -α log ∫ A π α ( a | s ) da = 1 1 -α E a ∼ π log π α -1 ( a | s ) , then we have:

Proof. At state s , consider A as the action set, a is the random variable, we can rewrite the right side of Eq. 12 as follows,

where the first term in Eq. 14 is the 2nd-order Rényi cross entropy between π and ¯ π . Thus, using Gibbs inequality, we have

which finish the proof.

A.2 Proofs of the Proposition 2

Proposition 2. Given a policy to be updated π and a fixed policy ¯ π with a finite action set A = { s 0 , ..., s n } at state s , then the CS divergence of is lower bounded by:

Proof. For two policies represented by the trajectories, the CS divergence between them at state s is defined as follows,

Consider the quadratic mean of the π ( a = a | s ) ,

We also have ∑ A π ( a = a | s )¯ π ( a = a | s ) ≤ 1 , and then

which complete the proof.

B Experimental Results

B.1 Detailed experimental Settings

Hence,

In this experiment, we follow the official implement and hyperparameter settings of HAPPO and HATRPO 1 [Kuba et al., 2022], MAPPO 2 [Yu et al., 2022], and A2PO 3 [Wang et al., 2023a]. We compare the proposed method with baselines on two popular heterogeneous environments, MAMujoco 4 , and Bi-DexHands 5 . For MA-Mujoco, the commom hyperparameter are listed in Tab. 1, and the different hyperparameters in each scenarios are listed in Tab. 2. For Bi-DexHands, the commom hyperparameter are listed in Tab. 3, and the different hyperparameters in each scenarios are listed in Tab. 4. The experiments were conducted on a PC with NVIDIA RTX3090 GPU, Intel Xeon 64-core CPU, and 64GB Ram.

B.2 Additional Results

Full results of 10 scenarios in MA-Mujoco and 10 scenarios in Bi-DexHands are shown in Fig. 7 and Fig. 8. We can make two observations on results of MA-Mujoco tasks. First, MADPO demonstrates superiority in terms of both reward maximum and learning speed, highlighting its effectiveness in exploring novel policies. Second, MADPO exhibits the lowest variance compared to other methods in most scenarios, indicating its training stability. In the more challenging Bi-DexHands tasks, we find MADPO outperforms baselines in most scenarios, confirming the effectiveness of MADPO in complex coordination tasks.

We also conduct the experiments of different updating orders in Ma-Mujoco, as indicated in Fig 9. Here, each agent controls one joint of one leg, and the joints in the same position on the legs have the same specilization. The Rand. order represents updating agents randomly. The Def. order represents updating agents in the default order in Ma-Mujoco, where agents are grouped according to the legs

5000

A2PO

5000

happo

hatrpo

5000

5000

mappo

madpo

000

4000

8000

apos

3000

2000

2000

1000

Steps

0.6

0.8

10

1e7

Steps

0.8

1e7

Steps

~

1e7

HalfCheetah-v2-2x3

HalfCheetah-v2-3x2

HalfCheetah-v2-6x1

5000

5000

5000

000

apos

4000

8000

3000

2000

2000

2000

1000

0001

0.0

0.8

10

0.0

0.8

08

Steps

1e7

Steps

1e7

Steps

1e7

5000

5000

5000

4000

8000

3000

2000

2000

2000

1000

L00C

1000

02

0.6

0.8

0.

0.

0.8

Steps

1e7

Steps

Steps

le7

Humanoid-v2-17x1

7000

6000

000E

2000

1000

they belong to. For example, the default updating order can be: right thigh joint, right leg joint, right foot joint, left thigh joint, · · · . The Spec. order represents updating agents according to their specilization, and can be: right thigh joint, left thigh joint, right foot joint, left foot joint, · · · . We can observe that the random updating order outperforms the other orders. We believe that this is because our framework can benefit from various updating orders. For example, if the current agent share the same specilization as the preceding agent, maximizing the inter-agent divergence can enhance exploration. On the other hand, if the current agent differs from the preceding agent, maximizing the inter-agent divergence can promote heterogeneity.

In Fig 10, we compare the performance between different parameter settings using IQM aggregate test. We can observe that MADPO is a little sensitive to parameter σ . However, it outperforms HAPPO in a reasonable range of σ . Additionally, we can also individually tune σ for special task for further performance improvement. We also observe that MAPDO is a little sensitive to the parameter λ , yet it consistently shows better performance than HAPPO.

Tab. 5 shows the running time of MADPO and other methods. Compared to MARL baselines, MADPO only introduces a negligible extra time cost.

C Social Impacts

We do not foresee an obvious negative impart by conducting the experiments included in this work, since we evaluate methods in a controlled simulation environment. However, We are also aware that recent works [Radosavovic et al., 2024, Handa et al., 2023] tested RL algorithms on

ShadowHandCatchOver2Underarm

ShadowHandBlockStack

ShadowHandCatchAbreast

happo

hatrpo

10

25

mappo

70

madpo

15

10

5o0

050015

2.00

50005

100

125

150

25

050

075

Lo0

7

Steps

le7

Steps

le7

Steps

ShadowHandCatchUnderarm

ShadowHandDoorCloselnward

ShadowHandDoorCloseOutward

3.0

175

150

sode rewards

125

60

200

75

25

2

050

1o0

1.25

2.00

050

2.00

Soo

050

23

2.00

Steps

le7

Steps

1e7

sdaas

le7

ShadowHandDoorOpenlnward

ShadowHandDoorOpenOutward

ShadowHandOver

25

10

100

125150

2.00

0.0

1.00

125

50

175

2.00

6.0

2.00

steps

1e7

Steps

steps

ShadowHandPen

Episode rewards

40

30

20

10

022

050

0.73

1o0

23

150

Steps

HalfCheetah-v2-6x1

6000

Rand.

Def.

6000

Rand.

Def.

6000

Rand.

Def.

Spec.

Spec.

Spec.

5000

5000

5000

Episode Reward

4000

Episode Reward

4000

Episode Reward

4000

3000

3000

3000

2000

2000

2000

1000

1000

1000

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0°0

0.2

0.4

0.6

0.8

1.0

Steps

1e7

Steps

1e7

Steps

1e7

10 tasks in Dexhands

10 tasks in Mamujoco

60

5000

50

IQM Episode Reward

IQM Episode Reward

4000

40

OE

3000

20

madpo(α = 1e2)

2000

madpo(入 = 0.1)

madpo(α = 5e2)

madpo(入 = 0.2)

10

madpo(α = 1e3)

happo

1000

madpo(入 = 0.5)

happo

0.00

0.25

0.500.75 1.001.25

Steps

1.50

1.752.00

le7

0.0

0.2

0.4

Steps

0.6

0.8

1.0

1e7

real-world environments. This may raise concerns regarding the potential personal hazards caused by agent exploration in real world. To address this issue, one possible approach is to define safety behaviours to restrict the actions of agents [Feng et al., 2023a] or perform evaluation in safe simulation environments [Ji et al., 2023] preliminarily.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We confirm that we accurately and clearly claim the contributions and the scope of this work in the abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the main limitation of this work in Conclusion 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided the detailed proofs in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have described the detailed steps of the proposed method in Algo. 1, and the experiment settings in Appendix B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available at https://github.com/hwdou6677/MADPO .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided the experimental details in Appendix B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our experiment results include the mean and the 95% trust interval of five random seeds, as indicated in Section 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided the compute resource required in the experiments in Appendix B.1

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm that our research fully complies with the NeurIPS Code of Ethics in all aspects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the societal impacts of this work in Appendix C.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work does not poses a risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have provided the URL of assets used in this work in B.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We do not release new assets in this work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not include any crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not include any crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study MARL in heterogeneous settings, where agents are not allowed to share their parameters, and make use of the sequential updating scheme under the CTDE schema. They propose a method which exploits the preceding information to improve exploration and heterogeneity sequentially. This method is equipped with a mutual policy divergence maximization framework, which utilizes the discrepancies between episodes to enhance exploration and between agents to heterogenize agents. Interestingly, the authors propose the conditional Cauchy-Schwarz divergence to provide entropy-guided exploration incentives.\n\nSTRENGTHS:\n- The problem of exploration in settings with heterogeneous agents is important in MARL and not well-explored in literature. \n\n- The paper is the first to study the effectiveness of policy divergence maximization in the sequential updating schema, upon which important related work has been built.\n\n- The paper proposed the conditional Cauchy-Schwarz (CS) divergence as an alternative to the popular KL-divergence in MARL. Such an alternation may be interesting to the broader RL community. Interestingly, unlike KL-divergence which can expload for small values of the denominator, CS divergence has a provable good lower bound ($-\\log(n)$) only dependent on the number of finite actions.\n\n- The proposed method displays good performance, in comparison to strong SOTA methods (including MAPPO, HAPPO), on benchmarks with heterogeneous agents.\n\n- The proposed framework is simple and easy-to-implement.\n\n- The paper is generally well-written and easy-to-follow.\n\nWEAKNESSES:\n- The improvement over the baselines (standard KL-divergence, entropy term, no incentive) does not seem to be quite consistent in the ablation study, due to (a) high variance in the results of the no incentive, and (b) very close improvement over the KL divergence baseline in terms of best episodic reward in 2 out 3 tasks.\n\n- Since the CS divergence is new in MARL and RL, a table containing the running times of the evaluated algorithms is missing. How costly is the CS divergence?\n\nQUESTIONS:\n- The authors mention: \"To the best of our knowledge, there is no exploration method that can adapt to both heterogeneous scenarios with sequential updating and homogeneous scenarios with simultaneous updating\". But can the proposed method adapt to homogeneous scenarios with simultaneous updating? No experiments in such settings have been provided. Could the proposed intrinsic rewards be used to improve exploration in MARL settings with homogeneous agents?\n\n- Why do the authors use $\\lambda$ and $1 - \\lambda$ for weighting the intrinsic rewards, instead of arbitrary weights (not in a convex combination)? How important is it to the performance?\n\nLIMITATIONS:\nThe authors provide limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper proposes a novel training objective where it encourages the policies to diverge from each other and from the previous policy under heterogeneous multi-agent tasks based on sequential recently proposed sequential policy update. It utilizes CS divergence for calculation of "distance" between policies for tractable and stable optimization compared to KL divergence. The evaluation is done in high-dimensional multi-agent mujoco and and bi-dexhands environments, outperforming existing state-of-the-art sequential algorithms.

STRENGTHS:
- The paper is well written and easy to understand; Fig. 1 is very informative.
- The problem of exploration under agent heterogeneity is an important problem in multi-agent learning
- The proposed method is sound and is backed by theory

WEAKNESSES:
- The evaluation is hard to judge whether the proposed method is actually performs better than the baselines, this is a deal breaker. I suggest the authors also incorporate aggregate quantities from https://agarwl.github.io/rliable/

I'm willing to increase the score if the authors show that the improvement is statistically significant

QUESTIONS:
- Is it possible to have a "cyclic" problem where the 1st, 3rd, 5th, ... (and also 2nd, 4th, 6th, ...) policies have the same behavior despite optimizing the proposed training objective?
- Can the authors explain why CS is chosen over Jensen–Shannon divergence (JSD)?
- Is there a guideline for tuning the coefficients for the intrinsic rewards?

LIMITATIONS:
Minor comments
- line 36, wrong citation format
- line 193 and line 217, Ep. 5 should be Eq. 4
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is situated in the problem setting of heterogeneous cooperative agents, under the sequential update framework. The paper introduces the novel MADPO algorithm, in which agents maximize the Cauchy Schwarz divergence between agents and between episodes of data gathered by the same agent, to improve exploration. Empirical validation is performed on the Multi-Agent Mujoco and Bi-DexHands benchmark suites, demonstrating that the MADPO outperforms baselines.\n\nSTRENGTHS:\nOverall, the paper is clear, succinct, and the main idea is clear and easy to understand. The format, and figures are good, with all expected components included. The idea of maximizing the inter/intra agent divergences is intuitively appealing. Further, the authors address the pitfalls of naively maximizing intra-agent divergences by adopting the Cauchy Schwarz divergence. It's especially nice that maximizing the CS divergence implies maximizing the policy entropy as well. Experiments are done on a large number of tasks, with comparisons against expected baselines and parameter sensitivity analyses all present.\n\nWEAKNESSES:\n1. The motivation of the paper is not altogether clear to me. The paper seems to suggest that exploration is more challenging in the sequential update setting, necessitating devoted algorithms. Why would this be the case? \n2. In many of the presented domains, the improvement of MADPO over the next best method is not very large. Sometimes, confidence intervals of MADPO overlap those of the next best method. Can the authors provide statistical significance tests for the main results in Figures 2 and 3, comparing MADPO to the next best method?\n3. Some minor suggestions:\n- Please check your paper carefully for typos, as there are quite a few: \n    - Line 89: \"connecting link dimension curse\"? Not sure what this is\n    - No period after Figure 4\n    - Trust interval -> confidence interval \n    - Lacking 'and' at line 174\n    - Line 204: conditoned -> conditioned\n    - Line 216: extra \"of\" \n- Please be sure to state the number of trials in the main text. It is mentioned in the Neurips checklist, but I could not find it  in  the main text\n- Please make the colors of the methods the same for both domains (i.e. pick 1 color for MADPO and be consistent with it)\n\nQUESTIONS:\n1. How sensitive is the algorithm to the scale of the divergence rewards? Have you done a study on this? \n2. On the intra-policy divergence: is the policy updated every episode? If not, then wouldn't the intra-policy divergence reward often be 0? \n3. Line 180 states that it would be challenging to define an inter-agent divergence in the simultaneous update scheme. Why not consider the divergence between $\\pi^i_k$ and $\\pi^j_{k-1}$? But this does not seem any more challenging to compute, and can be computed under  CTDE assumptions.\n4. Would it be possible to implement this exploration scheme in the CTDE setting? If so, it would be interesting to see how well the method performs. \n5.   Proposition 2 states that the CS divergence has a lower bound. Does it also have an upper bound?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel multi-agent reinforcement learning (MARL) method called Multi-Agent Divergence Policy Optimization (MADPO), which enhances exploration and heterogeneity through a mutual policy divergence maximization framework. MADPO leverages a sequential updating scheme and quantifies discrepancies between episodes and agents, termed intra-agent divergence and inter-agent divergence, respectively. To address the instability and lack of directionality in traditional divergence measurements, the paper proposes using conditional Cauchy-Schwarz divergence to provide entropy-guided exploration incentives. Experiments demonstrate that the proposed method outperforms state-of-the-art sequential updating approaches in two challenging multi-agent tasks with various heterogeneous scenarios.\n\nSTRENGTHS:\n1. **Innovation**: The paper introduces MADPO, a novel MARL method that enhances agent exploration and heterogeneity through mutual policy divergence maximization. \n   \n2. **Theoretical Foundation**: The use of conditional Cauchy-Schwarz divergence to address instability and directionality in traditional divergence measurements is a contribution.\n   \n3. **Experimental Validation**: The experiments conducted on two challenging multi-agent tasks with different heterogeneous scenarios convincingly demonstrate the effectiveness and superiority of MADPO in enhancing exploration and heterogeneity.\n\nWEAKNESSES:\n1.  The paper lacks analysis and comparison with relevant literature on sequential decision-making, such as:\n   - Liu J, Zhong Y, Hu S, et al. Maximum Entropy Heterogeneous-Agent Reinforcement Learning[C]//The Twelfth International Conference on Learning Representations.  (This paper extends SAC to heterogeneous sequential decision-making scenarios, and the relationship between this work and the current paper remains unclear.)\n\n2. It is unclear whether the intrinsic reward method proposed in this paper can ensure that the resulting trained policies are consistent with the original policies.\n\nQUESTIONS:\n1. Could you provide a detailed comparison between your proposed MADPO method and the approach presented in \"Maximum Entropy Heterogeneous-Agent Reinforcement Learning\" by Liu et al.? Specifically, how does MADPO improve upon or differ from this method in terms of handling heterogeneous sequential decision-making scenarios?\n\n2. I may have missed some details, but could you clarify whether the intrinsic reward method in MADPO ensures that the trained policies remain consistent with those optimized solely based on the original rewards?\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work and discussed potential negative societal impacts in accordance with the guidelines.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}