Title: Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: multi-agent reinforcement learning; parameter sharing
Abstract: In multi-agent reinforcement learning (MARL), parameter sharing is commonly employed to enhance sample efficiency. However, the popular approach of full parameter sharing often leads to homogeneous policies among agents, potentially limiting the performance benefits that could be derived from policy diversity. To address this critical limitation, we introduce \emph{Kaleidoscope}, a novel adaptive partial parameter sharing scheme that fosters policy heterogeneity while still maintaining high sample efficiency. Specifically, Kaleidoscope maintains one set of common parameters alongside multiple sets of distinct, learnable masks for different agents, dictating the sharing of parameters. It promotes diversity among policy networks by encouraging discrepancy among these masks, without sacrificing the efficiencies of parameter sharing. This design allows Kaleidoscope to dynamically balance high sample efficiency with a broad policy representational capacity, effectively bridging the gap between full parameter sharing and non-parameter sharing across various environments. We further extend Kaleidoscope to critic ensembles in the context of actor-critic algorithms, which could help improve value estimations. Our empirical evaluations across extensive environments, including multi-agent particle environment, multi-agent MuJoCo and StarCraft multi-agent challenge v2, demonstrate the superior performance of Kaleidoscope compared with existing parameter sharing approaches, showcasing its potential for performance enhancement in MARL. The code is publicly available at \url{https://github.com/LXXXXR/Kaleidoscope}.

Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning

Abstract

In multi-agent reinforcement learning (MARL), parameter sharing is commonly employed to enhance sample efficiency. However, the popular approach of full parameter sharing often leads to homogeneous policies among agents, potentially limiting the performance benefits that could be derived from policy diversity. To address this critical limitation, we introduce Kaleidoscope , a novel adaptive partial parameter sharing scheme that fosters policy heterogeneity while still maintaining high sample efficiency. Specifically, Kaleidoscope maintains one set of common parameters alongside multiple sets of distinct, learnable masks for different agents, dictating the sharing of parameters. It promotes diversity among policy networks by encouraging discrepancy among these masks, without sacrificing the efficiencies of parameter sharing. This design allows Kaleidoscope to dynamically balance high sample efficiency with a broad policy representational capacity, effectively bridging the gap between full parameter sharing and non-parameter sharing across various environments. We further extend Kaleidoscope to critic ensembles in the context of actor-critic algorithms, which could help improve value estimations. Our empirical evaluations across extensive environments, including multi-agent particle environment, multi-agent MuJoCo and StarCraft multi-agent challenge v2, demonstrate the superior performance of Kaleidoscope compared with existing parameter sharing approaches, showcasing its potential for performance enhancement in MARL. The code is publicly available at https://github.com/LXXXXR/Kaleidoscope .

1 Introduction

Cooperative multi-agent reinforcement learning (MARL) has demonstrated remarkable effectiveness in solving complex real-world decision-making problems across various domains, such as resource allocation (Ying and Dayong, 2005), package delivery (Seuken and Zilberstein, 2007), autonomous driving (Zhou et al., 2021), and robot control (Swamy et al., 2020). To mitigate the challenges posed by the non-stationary and partially observable environments typical of MARL (Yuan et al., 2023), the centralized training with decentralized execution (CTDE) paradigm (Foerster et al., 2016) has become prevalent, inspiring many influential MARL algorithms such as MADDPG (Lowe et al., 2017), COMA (Foerster et al., 2018), MATD3 (Ackermann et al., 2019), QMIX (Rashid et al., 2020), and MAPPO (Yu et al., 2022).

gains. An alternative approach is the non-parameter sharing scheme, where each agent possesses its own unique parameters. Nevertheless, while this method naturally supports heterogeneous policies, it suffers from reduced sample efficiency, leading to significant training costs. This is particularly problematic given the current trend towards increasingly large model sizes, with some scaling to trillions of parameters (Zhao et al., 2023; Achiam et al., 2023). Therefore, it is imperative to develop a parameter sharing strategy that enjoys both high sample efficiency and broad policy representational capacity, potentially achieving significantly enhanced performance. While several efforts (Christianos et al., 2021; Kim and Sung, 2023) have explored partial parameter sharing initiated at the start of training, such initializations can be challenging to design without detailed knowledge of agent-specific environmental transitions or reward functions (Christianos et al., 2021).

In this work, we build upon insights from previous studies (Christianos et al., 2021; Fu et al., 2022; Kim and Sung, 2023) and introduce Kaleidoscope , a novel adaptive partial parameter sharing scheme. It maintains a single set of policy parameters and employs multiple learnable masks to designate the shared parameters. Unlike earlier methods that depend on fixed initializations, Kaleidoscope dynamically learns these masks alongside MARL parameters throughout the training process. This end-to-end training approach inherently integrates environmental information, and its adaptive nature enables Kaleidoscope to dynamically adjust the level of parameter sharing based on the demands of the environment and the learning progress of the agents. The learnable masks facilitate a dynamic balance between full parameter sharing and nonparameter sharing, offering a flexible trade-off between sample efficiency and policy representational capacity

Food

Forest

Obstacle

Predator

through enhanced heterogeneity. Initially, we build Kaleidoscope upon agent networks, where it achieves diverse policies. Following this success, we extend it to multi-agent actor-critic algorithms to encourage heterogeneity among the central critic ensembles for further performance enhancement.

Just like a kaleidoscope uses the reflective properties of rotating mirrors to transform simple shapes into beautiful patterns, our proposed method leverages learnable masks to map a single set of parameters into diverse policies, thereby enhancing task performance.

We summarize our contributions as follows:

To enable policy heterogeneity among agents for better training flexibility, we adapt the soft threshold reparameterization (STR) technique to learn distinct masks for different agent networks while only maintaining one set of common parameters, effectively balancing between full parameter sharing and non-parameter sharing mechanisms.

To enhance policy diversity among agents, we introduce a novel regularization term that encourages the pairwise discrepancy between masks. Additionally, we design resetting mechanisms that recycle masked parameters to preserve the representational capacity of the joint networks.

Through extensive experiments on MARL benchmarks, including multi-agent particle environment (MPE) (Lowe et al., 2017), multi-agent MuJoCo (MAMuJoCo) (Peng et al., 2021) and StarCraft multi-agent challenge v2 (SMACv2) (Ellis et al., 2024), we demonstrate the superior performance of Kaleidoscope over existing parameter sharing approaches.

2 Background

Multi-agent reinforcement learning (MARL) In MARL, a fully cooperative partially observable multi-agent task is typically formulated as a decentralized partially observable Markov decision process (dec-POMDP) (Oliehoek and Amato, 2016), represented by a tuple M = ⟨S , A, P, R, Ω , O, N, γ ⟩ . Here, N denotes the number of agents, and γ ∈ (0 , 1] represents the discount factor. At each timestep t , with the environment state as s t ∈ S , agent i receives a local observation o t i ∈ Ω drawn from the observation function O ( s t , i ) and then follows its local policy

π i to select an action a t i ∈ A . Individual actions form a joint action a t ∈ A N , leading to a state transition to the next state s t +1 ∼ P ( s t +1 | s t , a t ) and inducing a global reward r t = R ( s t , a t ) . The overall team objective is to learn the joint policies π = ⟨ π 1 , . . . , π N ⟩ such that the expectation of discounted accumulated reward G t = ∑ t γ t r t is maximized.

To learn such policies π θ , various MARL algorithms (Lowe et al., 2017; Foerster et al., 2018; Rashid et al., 2020; Yu et al., 2022) have been developed. For instance, the off-policy actor-critic algorithm MATD3 (Ackermann et al., 2019) serves as an example method. Specifically, the critic networks are updated by minimizing the temporal difference (TD) error loss

with

where ϕ are the parameters for critics, θ are the parameters for actor policies and θ ′ are the parameters for target actor policies. And ϵ is the clipped Gaussian noise, given as clip ( N (0 , σ ) , -c, c ) .

The policy is updated by the deterministic policy gradient algorithm (Silver et al., 2014)

Soft threshold reparameterization (STR) Originally introduced in the context of model sparsification, STR (Kusupati et al., 2020) is an unstructured pruning method that achieves notable performance without requiring a predetermined sparsity level. Specifically, STR applies a transformation to the original parameters W as follows

where s is a learnable parameter, α = g ( s ) serves as the pruning threshold, and ReLU ( · ) = max( · , 0) . The original supervised learning problem modeled by

with D as the data is now transferred to

Overall, this approach optimizes the learnable pruning threshold alongside the model parameters, facilitating dynamic adjustment to the sparsity level during training.

3 Learnable Masks for Heterogenous MARL

In this section, we propose using learnable masks as a low-cost method to enable network heterogeneity in MARL. The core concept, illustrated in Figure 2, is to learn a single set of shared parameters complemented by multiple masks for distinct agents, specifying which parameters to share.

Specifically, in Section 3.1, we first adapt STR into a dynamic partially parameter sharing method, unlocking the joint policy network's capability to represent diverse policies among agents. In Section 3.2, we actively foster policy heterogeneity through a novel regularization term based on the masks. Given that the masking technique could excessively sparsify the network, potentially diminishing its representational capacity, in Section 3.3, we propose a straightforward remedy to periodically reset the parameters based on the outcomes of masking, which additionally mitigates primacy bias. Finally, in Section 3.4, we explore how to further extend this approach within the critic components of actor-critic algorithms to improve value estimations in MARL and further boost performance.

For the sake of clarity, we integrate the proposed Kaleidoscope with the MATD3 (Ackermann et al., 2019) algorithm to demonstrate the concept within this section. Nevertheless, as a versatile partial parameter-sharing technique, our method can readily be adapted to other MARL algorithms. We defer its integration with other MARL frameworks to Appendix A.1.2 and will evaluate them empirically in Section 4.

𝑄(#; 𝜙

!

)

…

…

Mean

…

…

Critic Ensembles

𝑄(#; 𝜙

"

)

𝑄(#; 𝜙

#

)

Actor

𝟏

𝜃

$

⊙

⊙

𝑴

!

𝜃

!

=

=

𝜋

!

(#; 𝜃

!

)

Actor

𝒊

𝜃

$

⊙

⊙

𝑴

%

𝜃

%

=

=

𝜋

%

(#; 𝜃

%

)

Actor

𝑵

𝜃

$

⊙

⊙

𝑴

&

𝜃

&

=

=

𝜋

&

(#; 𝜃

&

)

𝜙

$

⊙

⊙

𝑴

!

'

𝜙

!

=

=

𝑄(#; 𝜙

!

)

𝜙

$

⊙

⊙

𝑴

#

'

𝜙

#

=

=

𝑄(#; 𝜙

#

)

3.1 Adaptive partial parameter sharing ♦

The core idea of this work is to learn distinct binary masks M i for different agents to facilitate differentiated policies, ultimately aiming to improve MARL performance. To achieve this, we apply the STR (Kusupati et al., 2020) technique to the policy parameters with different thresholds dedicated to each agent:

where θ i parameterizes the policy for agent i , θ 0 is the set of learnable parameters shared by all agents, and M i is the learnable mask for agent i . Specifically, assume θ 0 = [ θ (1) 0 , . . . , θ ( N a ) 0 ] , θ i = [ θ (1) i , . . . , θ ( N a ) i ] and M i = [ m (1) i , . . . , m ( N a ) i ] with N a being the total parameter count of an agent's network. In line with STR, we compute each element m ( k ) i of M i as m ( k ) i = ✶ [ | θ ( k ) 0 | > σ ( s ( k ) i ) ] , where σ ( · ) denotes the Sigmoid function. The benefits of such a combination are summarized as follows:

Preservation of original MARL learning objectives: Unlike most of the methods in pruning literature, which primarily aim to minimize the discrepancies between pruned and unpruned networks in terms of weights, loss, or activations (Hoefler et al., 2021; Menghani, 2023; Deng et al., 2020), STR maintains the original goal of minimizing task-specific loss, aligning directly with our objectives to enhance MARL performance.

Flexibility in sparsity: Many classical pruning methods require predefined per-layer sparsity levels (Evci et al., 2020; Ramanujan et al., 2020). Such requirements can complicate our design, with the goal not to gain extreme sparsity but rather to promote heterogeneity through masking. The STR technique is ideal in our case as it does not require predefining sparsity levels, allowing for adaptive learning of the masks.

Enhanced network representational capacity: Utilizing learnable masks for adaptive partial parameter sharing enhances the network's representational capacity beyond traditional full parameter sharing. In full parameter sharing, agents' joint policies are parameterized as π ps ( ·| θ 0 ) = ⟨ π 1 ( ·| θ 0 ) , . . . , π N ( ·| θ 0 ) ⟩ . In contrast, our proposed adaptive partial parameter sharing mechanism parameterizes the joint policies as π Kaleidoscope ( ·| θ 0 , M ) = ⟨ π 1 ( ·| θ 0 ⊙ M 1 ) , . . . , π n ( ·| θ 0 ⊙ M N ) ⟩ . In the extreme case where all the values in M i are 1 s, the function set represented by π Kaleidoscope ( ·| θ 0 , M ) degrades to that of π ps ( ·| θ 0 ) . In other scenarios, it is a superset of that represented by π ps ( ·| θ 0 ) .

𝜃

!

⊙

𝑴

"

𝜃

"

=

𝜃

!

⊙

𝑴

#

𝜃

#

=

Actors

𝜃

!

⊙

𝑴

"

𝜃

"

=

𝜃

!

⊙

𝑴

#

𝜃

#

=

Actors

𝜙

!

⊙

𝑴

"

#

𝜙

"

=

𝜙

!

⊙

𝑴

$

#

𝜙

$

=

Critic Ensembles

𝜙

!

⊙

𝑴

"

#

𝜙

"

=

𝜙

!

⊙

𝑴

$

#

𝜙

$

=

Critic Ensembles

(a) Actors: reinitialize the weights that are masked by all agents with probability ρ .

(b) Critic ensembles: reset one set of masks at a time.

3.2 Policy diversity regularization ♣

While independently learned masks enable agents to develop distinct policies, without a specific incentive, these policies may still converge to being homogeneous. To this end, we propose to explicitly encourage agent policy heterogeneity by introducing a diversity regularization term maximizing the weighted pairwise distance between network masks, which is defined as

̸

This term is inherently non-differentiable due to the indicator function ✶ [ · ] inside M . To overcome this difficulty, following established practices in the literature (Bengio et al., 2013; Alizadeh et al., 2018), we utilize a surrogate function for gradient approximation:

We formally provide the overall training objective for actors in Appendix A.1.1.

3.3 Periodically reset ♠

As the training with masks proceeds, we observe an increasing sparsity in each agent's network, potentially reducing the overall network capacity. To remedy the issue, we propose a simple approach to periodically reset the parameters that are consistently masked across all M i with a certain probability ρ , which is illustrated in Figure 3a. At intervals defined by t mod reset_interval == 0 , if the parameter index k satisfies ∀ i, m ( k ) i == 0 , we apply the following resetting rule

This resetting mechanism recycles the weights masked as zeros by all the masks, preventing the networks from becoming overly sparse. A side benefit of this resetting mechanism is the enhancement of neural plasticity (Lyle et al., 2023; Nikishin et al., 2024), which helps alleviate the primacy bias (Nikishin et al., 2022) in reinforcement learning. Unlike methods that reinitialize entire layers resulting in abrupt performance drops (Nikishin et al., 2022), our resetting approach selectively targets weights as indicated by the learnable masks, thus avoiding significant performance disruptions, as shown in Section 4.

3.4 Critic ensembles with learnable masks

In actor-critic algorithm frameworks, we further apply Kaleidoscope to central critics as an efficient way to implement ensemble-like critics. By facilitating dynamic partial parameter sharing, Kaleidoscope enables heterogeneity among critic ensembles. Furthermore, by regularizing the diversity

among critic functions, we can control ensemble variances. This approach is elaborated in subsequent paragraphs.

♦ Adaptive partial parameter sharing for critic ensembles In the standard MATD3 algorithm (Ackermann et al., 2019), two critics with independent parameters are maintained to mitigate overestimation risks. However, using separate parameters typically results in a low update-to-data (UTD) ratio (Hiraoka et al., 2022). To address this issue, we propose to enhance the UTD ratio by employing Kaleidoscope parameter sharing among ensembles of critics. Specifically, we maintain a single set of parameters ϕ 0 and K masks [ M c j ] K j =1 to distinguish the critic functions, resulting in K ensembles [ Q ( · ; ϕ j )] K j =1 with ϕ j = ϕ 0 ⊙ M c j .

To be specific, we update the critic networks by minimizing the temporal difference (TD) error loss

with

And the policies are updated by the mean estimation of the ensembles as

♣ Critic ensembles diversity regularization As in Section 3.2, we also apply diversity regularization to critic masks to prevent critics functions from collapsing to identical ones. The diversity regularization to maximize for the critic ensembles is expressed as

̸

Intuitively, as training progresses, this term encourages divergence among the critic masks, leading to increased model estimation uncertainty. This process fosters a gradual shift from overestimation to underestimation. As discussed in prior research (Hiraoka et al., 2022; Lan et al., 2020; Chen et al., 2021; Wang et al., 2021b), overestimation can encourage exploration, beneficial in early training stages, whereas underestimation alleviates error accumulation (Fujimoto et al., 2018), which is preferred in the late training stage. We formally provide the overall training objective for critic ensembles in Appendix A.1.1.

♠ Periodically reset To further promote diversity among critic ensembles and counteract the reduction in network capacity caused by masking, we implement a resetting mechanism similar to that described in Section 3.3. In particular, we sequentially reinitialize the masks M c j following a cyclic pattern, as illustrated in Figure 3b. In this way, each critic function's mask is trained on distinct data segments, leading to different biases.

In summary, by adopting Kaleidoscope parameter sharing with learnable masks, we establish a cost-effective implementation for critic ensembles that enjoy a high UTD ratio. Through enforcing distinctiveness among the masks, we subtly control the differences among critic functions, thereby improving the value estimations in MARL.

4 Experimental Results

In this section, we integrate Kaleidoscope with the value-based MARL algorithm QMIX and the actor-critic MARL algorithm MATD3, and evaluate them across eleven scenarios in three benchmark tasks.

4.1 Experimental Setups

Environment descriptions We test our proposed Kaleidoscope on three benchmark tasks: MPE (Lowe et al., 2017), MaMuJoCo (Peng et al., 2021) and SMACv2 (Ellis et al., 2024). For the discrete tasks MPE and SMACv2, we integrate Kaleidoscope and baselines with QMIX (Rashid et al., 2020) and assess the performance. For the continuous task MaMuJoCo, we employ MATD3 (Ackermann et al., 2019). We use five random seeds for MPE and MaMuJoCo and three random seeds for SMACv2, reporting averaged results and displaying the 95% confidence interval with shaded areas. The chosen benchmark tasks reflect a mix of discrete and continuous action spaces and both homogeneous and heterogeneous agent types, detailed further in Appendix A.2.

Baselines In the following, we compare our proposed Kaleidoscope with baselines (Christianos et al., 2021; Kim and Sung, 2023), as listed in Table 1. For both Kaleidoscope and the baselines, in scenarios with fixed agent types (MPE and MaMuJoCo), we assign one mask per agent. For SMACv2, where agent types vary, we assign one mask per agent type. We use official implementations of the baselines where available; otherwise, we closely follow the descriptions from their respective papers, integrating them into QMIX or MATD3. Hyperparameters and further details are provided in Appendix A.1.3.

4.2 Results

Performance We present the comparative performance of Kaleidoscope and baselines in Figure 4 and Figure 5. Overall, Kaleidoscope demonstrates superior performance, attributable to the flexibility of the learnable masks and the effectiveness of diversity regularization. Additionally, we observe that FuPS + ID generally outperforms NoPS, except for the Ant-v2-4x2 scenario (Figure 4c). This advantage is largely due to FuPS's higher sample efficiency; a single transition data sample updates the model parameters N times in FuPS + ID, once for each agent, compared to just once in NoPS. Consequently, FuPS + ID models learn faster from the same number of transitions. Similarly, Kaleidoscope benefits from this mechanism as it shares weights among agents, allowing a single transition to update the model parameters multiple times. Furthermore, by integrating policy heterogeneity through learnable masks, Kaleidoscope enables diverse agent behaviors, as illustrated in the visualization results in Figure 8. Ultimately, Kaleidoscope effectively balances parameter sharing and diversity, outperforming both full parameter sharing and non-parameter sharing approaches.

Cost analysis Despite its superior performance, Kaleidoscope does not increase computational complexity at test time compared to the baselines. We report the test time averaged FLOPs comparison of Kaleidoscope and baselines in Table 2. We see that due to the masking technique, Kaleidoscope has lower FLOPs compared to baselines, thereby enjoying a faster inference speed when being deployed.

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e7

0

2000

4000

6000

Test Return

Kaleidoscope

SNP

MultiH

SePS

FuPS + ID

FuPS

NoPS

0

1

2

3

Timesteps

1e6

0

100

200

300

Test Return

(a) World

0

1

2

3

Timesteps

1e6

-20

-10

0

10

Test Return

(b) Push

0.00

0.25

0.50

0.75

1.00

Timesteps

1e7

0

2000

4000

6000

Test Return

0.00

0.25

0.50

0.75

1.00

Timesteps

1e7

0

1000

2000

3000

4000

Test Return

0.00

0.25

0.50

0.75

1.00

Timesteps

1e7

0

2000

4000

6000

Test Return

0.00

0.25

0.50

0.75

1.00

Timesteps

1e7

0

2000

4000

6000

8000

Test Return

0.00

0.25

0.50

0.75

1.00

Timesteps

1e7

0

2000

4000

6000

8000

Test Return

(g) HalfCheetah-v2-2x3

0.00

0.25

0.50

0.75

1.00

Timesteps

1e7

0

100

200

300

400

500

Test Return

(h) Swimmer-v2-10x2

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e7

0

20

40

60

Test Win Rate %

(a) Terran_5_vs_5

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e7

0

20

40

60

Test Win Rate %

(b) Protoss_5_vs_5

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e7

0

10

20

30

40

50

Test Win Rate %

(c) Zerg_5_vs_5

0.00

0.25

0.50

0.75

Timesteps

0

5000

Test Return

Kaleidoscope

SNP

MultiH

SePS

FuPS + ID

FuPS

NoPS

1e7

Ablation studies We conduct ablation studies to assess the impact of key components in Kaleidoscope, with results presented in Figure 6. Specifically, we compare Kaleidoscope with three ablations: 1) Kaleidoscope w/o reg , which lacks the regularization term in Equation (8) that encourages the masks to be distinct. 2) Kaleidoscope w/o reset , which does not reset parameters. 3) Kaleidoscope w/o ce , which does not use Kaleidoscope parameter sharing in critic ensembles and instead maintains two independent sets of parameters for critics. From the results, we observe that diversity regularization contributes the most to the performance of Kaleidoscope. Without it, masking degrades the performance due to the reduced number of parameters in each policy network. Resetting primarily aids learning in the late stages of training when needed, which aligns with the observation made by Nikishin et al. (2022). Notably, even with resetting, the performance does not experience abrupt drops thanks to the guidance provided by the masks on where to reset. When ablating the critic ensembles with Kaleidoscope parameter sharing, we observe inferior performance from the beginning of the training. This is because the critic ensembles with Kaleidoscope parameter sharing enable a higher UTD ratio of the critics, as discussed in Section 3.4.

Furthermore, we conduct experiments to study the impact of mask designs. The results are shown in Figure Figure 7. Specifically, we compare original Kaleidoscope with two alternative mask design choices: 1) Kaleidoscope w/ neuron masks , where adaptive masking techniques are applied to neurons rather than weights. 2) Kaleidoscope w/ fixed masks , where the masks are initialized at the beginning of training and kept fixed throughout the learning process. The results show that performance drops with either alternative design choice, demonstrating that Kaleidoscope's superior performance originates from the flexibility of the learnable masks on weights.

More results on hyperparameter analysis are included in Appendix B.2.

1.00

Methods

NoPS

FuPS

FuPS +ID

SePS

MultiH

SNP

Kaleidoscope

MPE

1.0x

0.992x

1.0x

1.0x

1.0x

0.988x

0.901x

MaMuJoCo

1.0x

0.985x

1.0x

1.0x

1.0x

0.900x

0.680x

SMACv2

1.0x

0.992x

1.0x

1.0x

1.0x

0.988x

0

1

2

3

Timesteps

1e6

0

100

200

300

Test Return

Kaleidoscope

Kaleidoscope w/o reg

Kaleidoscope w/o reset

(a) World

0.00

0.25

0.50

0.75

1.00

Timesteps

1e7

0

2000

4000

6000

Test Return

Kaleidoscope

Kaleidoscope w/o reg

Kaleidoscope w/o reset

Kaleidoscope w/o ce

0

1

2

3

Timesteps

1e6

0

100

200

300

Test Return

Kaleidoscope

Kaleidoscope w/ neuron masks

Kaleidoscope w/ fixed masks

(a) World

0.00

0.25

0.50

0.75

1.00

Timesteps

1e7

0

2000

4000

6000

Test Return

Kaleidoscope

Kaleidoscope w/ neuron masks

Kaleidoscope w/ fixed masks

Figure 6: Ablation studies.

Figure 7: Comparison on mask designs.

Visualization We visualize the trained policies of Kaleidoscope on World , as shown in Figure 8a. The agents exhibit cooperative divide-and-conquer strategies (four red agents divide into two teams and surround the preys), contrasting with the homogeneous policies depicted in Figure 1. We further examine the distinctions in the agents' masks and present the results in Figure 8b. First, we observe that by the end of the training, each agent has developed a unique mask, revealing that distinct masks facilitate diverse policies by selectively activating different segments of the neural network weights. Second, throughout the training process, we note that the differences among the agents' masks evolve dynamically. This observation confirms that Kaleidoscope effectively enables dynamic parameter sharing among the agents based on the learning progress, empowered by the adaptability of the learnable masks. More visualization results are provided in Appendix B.3.

5 Related Work

Parameter sharing First introduced by Tan (1993), parameter sharing has been widely adopted in MARL algorithms (Foerster et al., 2018; Rashid et al., 2020; Yu et al., 2022), due to its simplicity and high sample efficiency (Grammel et al., 2020). However, schemes without parameter sharing typically offer greater flexibility for policy representation. To balance sample efficiency with policy representational capacity, some research efforts aim to find effective partial parameter sharing schemes. Notably, SePS (Christianos et al., 2021) first clusters agents based on their transitions at the start of training and restricts parameter sharing within these clusters. Subsequently, SNP (Kim and Sung, 2023) enables partial parameter sharing by utilizing the lottery ticket hypothesis (Su et al., 2020) to initialize heterogeneous network structures. Concurrent to our work, AdaPS (Li et al., 2024) combines SNP and SePS by proposing a cluster-based partial parameter sharing scheme. While these methods have shown promise in certain domains, their performance potential is often limited by the static nature of the parameter sharing schemes set early in training. Our proposed Kaleidoscope distinguishes itself by dynamically learning specific parameter sharing configurations alongside the development of MARL policies, thereby offering enhanced training flexibility.

Agent heterogeneity in MARL To incorporate agent heterogeneity in MARL and enable diverse behaviors among agents, previous methods have explored concepts such as diversity and roles. Specifically, diversity-based approaches aim to enhance pairwise distinguishability among agents based on identities (Jiang and Lu, 2021), trajectories (Li et al., 2021), or credits assignment (Liu et al., 2023; Hu et al., 2023) through contrastive learning techniques. Concurrently, role-based strategies, sometimes referred to as skills (Yang et al., 2020) or subtasks (Yuan et al., 2022), employ conditional policies to differentiate agents by assigning them to various conditions. These conditions may be based on agent identities (Yang et al., 2022), local observations (Yang et al., 2020), local

1

2

3

4

Agent

4

3

2

1

Agent

Timesteps = 1M

1

2

3

4

Agent

4

3

2

1

Timesteps = 2M

1

2

3

4

Agent

4

3

2

1

Timesteps = 3M

0.00

0.02

0.04

0.06

0.08

Differences

(a) Trained policies.

(b) Pairwise mask differences among agents at different training timesteps.

histories (Wang et al., 2020, 2021a; Yuan et al., 2022) or joint histories (Liu et al., 2021; Iqbal et al., 2022; Zeng et al., 2023). This line of researches mainly focus on module design and operate separately from parameter-level adjustments, making them orthogonal to our approach. Nevertheless, integrating these methods with our work could potentially enhance performance further.

Sparse networks in deep reinforcement learning (RL) Although relatively few, there are some noteworthy recent attempts to find sparse networks for deep RL. In particular, PoPS (Livne and Cohen, 2020) prunes the dense networks post-training, achieving significantly reduced execution time complexity. Additionally, (Yu et al., 2020) validate the lottery ticket hypothesis within the RL domain, producing high-performance models even under extreme pruning rates. Subsequent efforts, including DST (Sokar et al., 2022), TE-RL* (Graesser et al., 2022) and RLx2 (Tan et al., 2023) employ topology evolution (TE) techniques to further decrease the training costs. While these developments utilize sparse training techniques, which are similar to the methods we employ, their primary focus is on reducing training and execution costs in single-agent settings. In contrast, our work leverages sparse network strategies as a means to enhance parameter sharing techniques, aiming to improve MARL performance.

6 Conclusions and Future Work

In this work, we introduced Kaleidoscope , a novel adaptive partial parameter sharing mechanism for MARL. It leverages distinct learnable masks to facilitate network heterogeneity, applicable to both agent policies and critic ensembles. Specifically, Kaleidoscope is built on three technical components: STR-empowered learnable masks, network diversity regularization, and a periodic resetting mechanism. When applied to agent policy networks, Kaleidoscope balances sample efficiency and network representational capacities. In the context of critic ensembles, it improves value estimations. By combining our proposed Kaleidoscope with QMIX and MATD3, we have empirically demonstrated its effectiveness across various MARL benchmarks. This study shows great promises in developing adaptive partial parameter sharing mechanisms to enhance the performance of MARL. For future work, it is interesting to further extend Kaleidoscope to other domains such as offline MARL or meta-RL.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023).

Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. 2019. Reducing overestimation bias in multi-agent domains using double centralized critics. arXiv preprint arXiv:1910.01465 (2019).

Milad Alizadeh, Javier Fernández-Marqués, Nicholas D Lane, and Yarin Gal. 2018. An empirical study of binary neural networks' optimisation. In Proceedings of the International Conference on Learning Representations .

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013).

Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. 2021. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model. In Proceedings of the 9th International Conference on Learning Representations .

Filippos Christianos, Georgios Papoudakis, Arrasy Rahman, and Stefano V. Albrecht. 2021. Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing. In Proceedings of the 38th International Conference on Machine Learning , Vol. 139. PMLR, 1989-1998.

Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. 2020. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proc. IEEE 108, 4 (2020), 485-532.

Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. 2024. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems , Vol. 36.

Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. 2020. Rigging the lottery: Making all tickets winners. In Proceedings of the International Conference on Machine Learning . PMLR, 2943-2952.

Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. 2016. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems , Vol. 29.

Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 32.

Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. 2022. Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning. In Proceedings of the International Conference on Machine Learning . PMLR, 6863-6877.

Scott Fujimoto, Herke van Hoof, and David Meger. 2018. Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the 35th International Conference on Machine Learning , Vol. 80. PMLR, 1582-1591.

Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. 2022. The state of sparse training in deep reinforcement learning. In Proceedings of the International Conference on Machine Learning . PMLR, 7766-7792.

Nathaniel Grammel, Sanghyun Son, Benjamin Black, and Aakriti Agrawal. 2020. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005.13625 (2020).

Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. 2022. Dropout Q-Functions for Doubly Efficient Reinforcement Learning. In The Tenth International Conference on Learning Representations .

Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research 22, 241 (2021), 1-124.

Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih wei Liao. 2021. Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning. (2021). arXiv:2102.03479 [cs.LG]

Zican Hu, Zongzhang Zhang, Huaxiong Li, Chunlin Chen, Hongyu Ding, and Zhi Wang. 2023. Attention-Guided Contrastive Role Representations for Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2312.04819 (2023).

Shariq Iqbal, Robby Costales, and Fei Sha. 2022. ALMA: Hierarchical learning for composite multi-agent tasks. In Advances in Neural Information Processing Systems , Vol. 35. 7155-7166.

Jiechuan Jiang and Zongqing Lu. 2021. The emergence of individuality. In Proceedings of the International Conference on Machine Learning . PMLR, 4992-5001.

Woojun Kim and Youngchul Sung. 2023. Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems . 1942-1950.

Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. 2020. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the International Conference on Machine Learning . PMLR, 5544-5555.

Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. 2020. Maxmin Q-learning: Controlling the Estimation Bias of Q-learning. In Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, April 26-30, 2020 .

Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. 2021. Celebrating diversity in shared multi-agent reinforcement learning. In Advances in Neural Information Processing Systems , Vol. 34. 3991-4002.

Dapeng Li, Na Lou, Bin Zhang, Zhiwei Xu, and Guoliang Fan. 2024. Adaptive parameter sharing for multi-agent reinforcement learning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 6035-6039.

Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandkumar. 2021. Coachplayer multi-agent reinforcement learning for dynamic team composition. In Proceedings of the International Conference on Machine Learning . PMLR, 6860-6870.

Shunyu Liu, Yihe Zhou, Jie Song, Tongya Zheng, Kaixuan Chen, Tongtian Zhu, Zunlei Feng, and Mingli Song. 2023. Contrastive identity-aware learning for multi-agent value decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37. 11595-11603.

Dor Livne and Kobi Cohen. 2020. Pops: Policy pruning and shrinking for deep reinforcement learning. IEEE Journal of Selected Topics in Signal Processing 14, 4 (2020), 789-801.

Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems , Vol. 30.

Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. 2023. Understanding plasticity in neural networks. In Proceedings of the International Conference on Machine Learning . PMLR, 23190-23211.

Gaurav Menghani. 2023. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. Comput. Surveys 55, 12 (2023), 1-37.

Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. 2024. Deep reinforcement learning with plasticity injection. In Advances in Neural Information Processing Systems , Vol. 36.

Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. 2022. The primacy bias in deep reinforcement learning. In Proceedings of the International Conference on Machine Learning . PMLR, 16828-16847.

Frans A Oliehoek and Christopher Amato. 2016. A concise introduction to decentralized POMDPs . Springer.

Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. 2021. Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks . http://arxiv.org/abs/2006.07869

Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. 2021. Facmac: Factored multi-agent centralised policy gradients. In Advances in Neural Information Processing Systems , Vol. 34. 12208-12221.

Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. 2020. What's hidden in a randomly weighted neural network?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11893-11902.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research 21, 1 (2020), 7234-7284.

Sven Seuken and Shlomo Zilberstein. 2007. Improved memory-bounded dynamic programming for decentralized POMDPs. In Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence . 344-351.

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. 2014. Deterministic policy gradient algorithms. In Proceedings of the International Conference on Machine Learning . Pmlr, 387-395.

Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. 2022. Dynamic Sparse Training for Deep Reinforcement Learning. In Proceedings of the ThirtyFirst International Joint Conference on Artificial Intelligence . ijcai.org, 3437-3443.

Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason D Lee. 2020. Sanity-checking pruning methods: Random tickets can win the jackpot. In Advances in Neural Information Processing Systems , Vol. 33. 20390-20401.

Gokul Swamy, Siddharth Reddy, Sergey Levine, and Anca D Dragan. 2020. Scaled autonomy: Enabling human operators to control robot fleets. In 2020 IEEE International Conference on Robotics and Automation . IEEE, 5942-5948.

Ming Tan. 1993. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the 10th International Conference on Machine Learning . 330-337.

Yiqin Tan, Pihe Hu, Ling Pan, Jiatai Huang, and Longbo Huang. 2023. RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch. In Proceedings of the Eleventh International Conference on Learning Representations .

Hang Wang, Sen Lin, and Junshan Zhang. 2021b. Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback. In Advances in Neural Information Processing Systems . 24778-24790.

Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. 2020. ROMA: multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning . 9876-9886.

Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. 2021a. RODE: Learning Roles to Decompose Multi-Agent Tasks. In Proceedings of the 9th International Conference on Learning Representations .

Jiachen Yang, Igor Borovikov, and Hongyuan Zha. 2020. Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems . 1566-1574.

Mingyu Yang, Jian Zhao, Xunhan Hu, Wengang Zhou, Jiangcheng Zhu, and Houqiang Li. 2022. LDSA: Learning dynamic subtask assignment in cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems , Vol. 35. 1698-1710.

Wang Ying and Sang Dayong. 2005. Multi-agent framework for third party logistics in E-commerce. Expert Systems with Applications 29, 2 (2005), 431-436.

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of ppo in cooperative multi-agent games. In Advances in Neural Information Processing Systems , Vol. 35. 24611-24624.

Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S. Morcos. 2020. Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP. In Proceedings of the 8th International Conference on Learning Representations .

Lei Yuan, Chenghe Wang, Jianhao Wang, Fuxiang Zhang, Feng Chen, Cong Guan, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. 2022. Multi-Agent Concentrative Coordination with Decentralized Task Representation.. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence . 599-605.

Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. 2023. A Survey of Progress on Cooperative Multi-agent Reinforcement Learning in Open Environment. arXiv preprint arXiv:2312.01058 (2023).

Xianghua Zeng, Hao Peng, and Angsheng Li. 2023. Effective and stable role-based multi-agent collaboration by structural information principles. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37. 11772-11780.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023).

Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, and Yaodong Yang. 2024. Heterogeneous-agent reinforcement learning. Journal of Machine Learning Research 25 (2024), 1-67.

Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, et al. 2021. Smarts: An open-source scalable multi-agent rl training school for autonomous driving. In Proceedings of the Conference on Robot Learning . PMLR, 264-285.

A Experimental details

A.1 Implementation details

A.1.1 Kaleidoscope with MATD3

Critic ensembles When incorporating Kaleidoscope into the MATD3 algorithm, the overall training loss for the critic ensembles becomes:

with L c ( ϕ 0 , s c j ) being the original MATD3 loss given in Equation (11), J div c ( s c ) being the diversity regularization given in Equation (14), and α d being a coefficient balancing the original MARL objective and the proposed diversity regularization. Note that although J div c ( s c ) contains parameters ϕ 0 , we stop the gradients for ϕ 0 in J div c ( s c ) .

In the implementation, we apply layer-wise weights to the diversity regularization term J div c ( s c ) , which is defined as

̸

where l denotes the layer index of the neuron networks, L represents the total number of layers, M c i,l is the mask for agent i at layer l , and the layer-wise weights are set as w l = 2 l . The intuition behind this choice is that features closer to the output tend to be more compact (Kusupati et al., 2020); consequently, assigning larger regularization weights to these layers may have a more significant impact on the output action decisions. Our initial experiments empirically demonstrate that setting w l = 2 l improves performance compared to the case where w l = 1 . Based on these findings, we maintain this design choice throughout all our experiments.

In practice, we adaptively adjust α d while maintaining a constant ratio between the MATD3 loss and the diversity loss, which is treated as a hyperparameter:

where α is a hyperparameter, and the gradients for | ∑ j =1 ,...,K L c ( ϕ 0 , s c j ) | |J div c ( s c ) | are stopped.

Actors For the actors, the training objective is to maximize the following term

where J ( θ 0 , s i ) is the original actor objective defined in Equation (3), J div ( s ) is the diversity regularization given in Equation (8) with layer-wise weights as in Equation (16), β d is the regularization coefficient. The value of β d is determined by

where β is a constant hyperparameter, similar to the approach used in Equation (17) for the critic ensembles.

A.1.2 Kaleidoscope with QMIX

When incorporating Kaleidoscope into the QMIX algorithm (Rashid et al., 2020), we apply Kaleidoscope parameter sharing only to the local Q networks. Consequently, the training loss is defined as:

where

with y tot = r + γ max a Q tot ( s t +1 , o t +1 , a ; θ -) and θ -representing the parameters of a target network as in DQN.

A.1.3 Network architecture and hyperparameters

Codebase Our implementation of Kaleidoscope and baseline algorithms are based on the following codebase:

HARL (Zhong et al., 2024) (MATD3 implementation): https://github.com/PKU-MARL/ HARL

EPyMARL (Papoudakis et al., 2021) (QMIX implementation for MPE): https://github.com/uoe-agents/epymarl

PyMARL2 (Hu et al., 2021) (QMIX implementation for SMACv2):

https://github.com/benellis3/pymarl2

SePS (Christianos et al., 2021): https://github.com/uoe-agents/seps

The code for Kaleidoscope is publicly available at https://github.com/LXXXXR/Kaleidoscope .

Network architecture In line with prior works (Zhong et al., 2024; Papoudakis et al., 2021; Hu et al., 2021), we employ deep neural networks consisting of multilayer perceptrons (MLPs) with rectified linear unit (ReLU) activation functions and gated recurrent units (GRUs) to parameterize the actor and critic networks. Moreover, when the masking technique is applied to critic ensembles, we incorporate layer normalization between the MLP layers and ReLU activations (Hiraoka et al., 2022). In Kaleidoscope, the masking technique is applied to the MLP layers, and the resetting mechanisms described in Sections 3.3 and 3.4 are applied to the last three layers of the respective neural networks, following Nikishin et al. (2022).

Hyperparameters To ensure a fair comparison, we implement our method and all the baselines using the same codebase with the same set of hyperparameters, with the exception of methodspecific ones. The common hyperparameters are listed in Tables 3 to 5. The Kaleidoscope-specific hyperparameters are provided in Table 6.

A.2 Environmental details

Codebase The environments used in this work are listed below with descriptions in Table 7.

MaMuJoCo (Peng et al., 2021): https://github.com/schroederdewitt/multiagent_ mujoco

MPE (Lowe et al., 2017; Papoudakis et al., 2021): https://github.com/semitable/ multiagent-particle-envs

SMACv2 (Ellis et al., 2024): https://github.com/oxwhirl/smacv2

MPE We extend the scenario settings provided in the original codebase to increase the complexity and challenge of the tasks. In World , we set the number of predators (agents) to 4, the number of prey to 2, the number of obstacles to 1, and the number of forests to 2. The objective of the game is for the predators to approach the prey while avoiding collisions with obstacles. The prey is attracted to the food and can hide from the predators in the forests. In Push , we set the number of agents to 5, the number of adversaries to 2, and the number of landmarks to 2. The goal of the game is for the agents to push the adversaries away from the landmarks. In both scenarios, we pretrain the adversary (prey) policies using the MADDPG algorithm Lowe et al. (2017) and use these pretrained policies to test the performance of different algorithms

A.3 FLOPs calculation

To calculate the number of floating-point operations (FLOPs) for a single forward pass of a sparse model, we sum the total number of multiplications and additions layer by layer, following the approach in Evci et al. (2020). For a fully-connected layer, the FLOPs are computed as follows:

For a GRU cell, the FLOPs are computed as:

A.4 Experimental Infrastructure

The experiments on the SMACv2 benchmark were conducted using NVIDIA GeForce RTX 3090 GPUs, while the experiments on other benchmarks were performed using NVIDIA GeForce RTX 3080 GPUs. Each experimental run required less than 2 days to complete.

B More results and discussion

B.1 Detailed Costs

We provide per-scenario FLOPs across different methods in Table 8 as a supplement for Table 2.

0.00

0.25

0.50

0.75

1.00

Timesteps

1e7

0

2000

4000

6000

8000

Test Return

α

=10

α

=1

α

=0.1

α

=0.01

0.00

0.25

0.50

0.75

1.00

Timesteps

1e7

0

2000

4000

6000

8000

Test Return

β

=10

β

=1

β

=0.1

β

=0.01

(a) Critic ensembles diversity coefficient α .

(b) Actors diversify coefficient

β

.

B.2 Hyperparameter Analysis

We conduct further analysis on the hyperparameters α and β , and present the results in Figure 9. The hyperparameter α controls the variance of the critic ensembles. As shown in Figure 9a, we observe that an excessively small α results in degraded performance because it reduces the critic ensembles to a single critic network, causing the value estimation to suffer from severe overestimation. Conversely, an excessively large α also deteriorates performance, possibly due to increased estimation bias. For β , as illustrated in Figure 9b, an overly small β leads to degraded performance because it reduces the Kaleidoscope parameter sharing to full parameter sharing, confining the policies to be identical. An overly large β also negatively impacts performance, as it may cause the training objective to deviate too much from minimizing the original MARL loss.

In general, we recommend setting both hyperparameters between 0 . 1 and 1 . However, the optimal hyperparameter values may vary across different scenarios. For fair comparisons, we maintain the same set of hyperparameters across all scenarios in our experiments. Nevertheless, further tuning of these hyperparameters has the potential to enhance performance.

B.3 Further Visualization Results

To better understand how learnable masks in Kaleidoscope affect the performance through policies, we visualize the pairwise mask differences among agents and the agent trajectories at different training stages in Figure 10. As training progresses, the test return increases and diversity loss decreases, indicating better performance and greater diversity among agent policies. Correspondingly, mask differences among agents increase, and the agent trajectory distribution becomes more diverse.

B.4 Limitations

Here we discuss some limitations of Kaleidoscope.

First, as suggested by results in Appendix B.2, the optimal hyperparameters vary from scenario to scenario. Therefore, using the same hyperparameters across all scenarios may not yield the best performance for Kaleidoscope. Developing an automatic scheme that utilizes environmental information to determine these hyperparameters would be beneficial.

Second, as the environments used in this work contain no more than 10 agents, we assign a distinct mask for each agent. However, when the problem scales to hundreds of agents, this vanilla implementation may fail. In such cases, a possible approach is to cluster N agents into K ( K < N ) groups and train K masks with Kaleidoscope. This would reduce computational costs and achieve a better trade-off between sample efficiency and diversity. Within the same group, agents share all parameters, while agents from different groups share only partial parameters. Techniques for clustering agents based on experience, as proposed by Christianos et al. (2021), could be useful.

GLYPH<c=5,font=/AAAAAB+DejaVuSans>GLYPH<c=4,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>

GLYPH<c=5,font=/AAAAAB+DejaVuSans>GLYPH<c=4,font=/AAAAAB+DejaVuSans>GLYPH<c=9,font=/AAAAAB+DejaVuSans>

GLYPH<c=6,font=/AAAAAB+DejaVuSans>GLYPH<c=4,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>

GLYPH<c=6,font=/AAAAAB+DejaVuSans>GLYPH<c=4,font=/AAAAAB+DejaVuSans>GLYPH<c=9,font=/AAAAAB+DejaVuSans>

GLYPH<c=7,font=/AAAAAB+DejaVuSans>GLYPH<c=4,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>

GLYPH<c=7,font=/AAAAAB+DejaVuSans>GLYPH<c=4,font=/AAAAAB+DejaVuSans>GLYPH<c=9,font=/AAAAAB+DejaVuSans>

GLYPH<c=8,font=/AAAAAB+DejaVuSans>GLYPH<c=4,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>

GLYPH<c=14,font=/AAAAAB+DejaVuSans>

GLYPH<c=16,font=/AAAAAB+DejaVuSans>GLYPH<c=17,font=/AAAAAB+DejaVuSans>GLYPH<c=15,font=/AAAAAB+DejaVuSans>GLYPH<c=22,font=/AAAAAB+DejaVuSans>GLYPH<c=23,font=/AAAAAB+DejaVuSans>GLYPH<c=15,font=/AAAAAB+DejaVuSans>GLYPH<c=20,font=/AAAAAB+DejaVuSans>GLYPH<c=22,font=/AAAAAB+DejaVuSans>

GLYPH<c=6,font=/AAAAAB+DejaVuSans>GLYPH<c=15,font=/AAAAAB+DejaVuSans>GLYPH<c=10,font=/AAAAAB+DejaVuSans>

GLYPH<c=5,font=/AAAAAB+DejaVuSans>

GLYPH<c=7,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>

GLYPH<c=14,font=/AAAAAB+DejaVuSans>

GLYPH<c=15,font=/AAAAAB+DejaVuSans>GLYPH<c=22,font=/AAAAAB+DejaVuSans>GLYPH<c=23,font=/AAAAAB+DejaVuSans>GLYPH<c=3,font=/AAAAAB+DejaVuSans>GLYPH<c=13,font=/AAAAAB+DejaVuSans>

GLYPH<c=15,font=/AAAAAB+DejaVuSans>GLYPH<c=23,font=/AAAAAB+DejaVuSans>GLYPH<c=24,font=/AAAAAB+DejaVuSans>GLYPH<c=21,font=/AAAAAB+DejaVuSans> GLYPH<c=18,font=/AAAAAB+DejaVuSans>

GLYPH<c=14,font=/AAAAAB+DejaVuSans>

GLYPH<c=15,font=/AAAAAB+DejaVuSans>GLYPH<c=22,font=/AAAAAB+DejaVuSans>GLYPH<c=23,font=/AAAAAB+DejaVuSans>GLYPH<c=3,font=/AAAAAB+DejaVuSans>GLYPH<c=13,font=/AAAAAB+DejaVuSans>

GLYPH<c=15,font=/AAAAAB+DejaVuSans>GLYPH<c=23,font=/AAAAAB+DejaVuSans>GLYPH<c=24,font=/AAAAAB+DejaVuSans>GLYPH<c=21,font=/AAAAAB+DejaVuSans> GLYPH<c=18,font=/AAAAAB+DejaVuSans>

GLYPH<c=11,font=/AAAAAB+DejaVuSans>GLYPH<c=16,font=/AAAAAB+DejaVuSans>GLYPH<c=25,font=/AAAAAB+DejaVuSans>GLYPH<c=15,font=/AAAAAB+DejaVuSans>GLYPH<c=21,font=/AAAAAB+DejaVuSans>GLYPH<c=22,font=/AAAAAB+DejaVuSans>GLYPH<c=16,font=/AAAAAB+DejaVuSans>GLYPH<c=23,font=/AAAAAB+DejaVuSans>GLYPH<c=26,font=/AAAAAB+DejaVuSans>GLYPH<c=3,font=/AAAAAB+DejaVuSans>GLYPH<c=12,font=/AAAAAB+DejaVuSans>

GLYPH<c=19,font=/AAAAAB+DejaVuSans>GLYPH<c=22,font=/AAAAAB+DejaVuSans>GLYPH<c=22,font=/AAAAAB+DejaVuSans>

GLYPH<c=27,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>GLYPH<c=4,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>GLYPH<c=6,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>

GLYPH<c=27,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>GLYPH<c=4,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>GLYPH<c=9,font=/AAAAAB+DejaVuSans>

GLYPH<c=5,font=/AAAAAB+DejaVuSans>GLYPH<c=4,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>GLYPH<c=5,font=/AAAAAB+DejaVuSans>

GLYPH<c=11,font=/AAAAAB+DejaVuSans>GLYPH<c=16,font=/AAAAAB+DejaVuSans>GLYPH<c=25,font=/AAAAAB+DejaVuSans>GLYPH<c=15,font=/AAAAAB+DejaVuSans>GLYPH<c=21,font=/AAAAAB+DejaVuSans>GLYPH<c=22,font=/AAAAAB+DejaVuSans>GLYPH<c=16,font=/AAAAAB+DejaVuSans>GLYPH<c=23,font=/AAAAAB+DejaVuSans>GLYPH<c=26,font=/AAAAAB+DejaVuSans>GLYPH<c=3,font=/AAAAAB+DejaVuSans>GLYPH<c=12,font=/AAAAAB+DejaVuSans>

GLYPH<c=19,font=/AAAAAB+DejaVuSans>GLYPH<c=22,font=/AAAAAB+DejaVuSans>GLYPH<c=22,font=/AAAAAB+DejaVuSans>

Pairwise Mask Differences

Agent 1

Agent 3

Agent 2

Agent 4

T= 0

T= 1M

T= 2M

T= 3M

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the paper's contributions by proposing a novel parameter sharing technique in MARL (Section 3) and claiming its superior performance compared to existing methods, which is supported by the experimental results (Section 4).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The method and implementation details are provided in Section 3 and Appendix A.1, and the experiment settings and environment details are described in Section 4 and Appendix A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available at https://github.com/LXXXXR/Kaleidoscope .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiment settings are provided in Section 4 with details elaborated in Appendices A.1 and A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The main results are reported with 95% confidence error bars in Figures 4 to 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see Appendix A.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Reinforcement Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The experiments in this work are conducted in simulated game environments, thereby presenting minimal risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The papers corresponding to the environments used are cited in Section 4, and the codebases utilized are listed in Appendices A.1.3 and A.2.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for adaptive parameter sharing in Multi-Agent Reinforcement Learning (MARL), by using learnable weight masks for each agent. They combine this with a regularization method to encourage diversity in the masks and a resetting mechanism to reuse masked parameters with a certain probability. They showed that their method can outperform parameter sharing baselines on a set of MARL benchmarks.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow. Their methodology and context are explained well. Furthermore, the authors provide a clear motivation for their work, to allow heterogeneous behaviour in MARL, while still leveraging the sample efficiency of parameter sharing, which is an important problem in MARL. \n- Applying soft threshold reparameterization (STR) [1] in MARL appears to be novel. \n- The authors provide a range of experiments on common MARL benchmarks. They also provide a detailed ablation study to show the importance of the different components of their method and measure the computation cost (FLOPs) of their method during test time.  They also provide a hyperparameter analysis in the appendix, which is useful.\n\n1] Kusupati, A., Ramanujan, V., Somani, R., Wortsman, M., Jain, P., Kakade, S., & Farhadi, A. (2020, November). Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning (pp. 5544-5555). PMLR.\n\nWEAKNESSES:\n- The authors do not include FuPS+ID (full parameter sharing, conditioning on one-hot encoded agent ids) as a baseline in their experiments. This was used as a baseline in prior work [1,2] and is the most common parameter sharing implementation in MARL, therefore it would be a useful comparison to have. \n- The authors appear to have missed a relevant related work [3] that also uses pruning masks for adaptive parameter sharing in MARL. This should be included in the related work section and discussed in the context of the proposed method.\n- The motivation for using a method like STR, which learns different use different weights for each agent, versus a method that uses different activations per agent (like SNP [2]) is not clear to me. I understand that STR learns the pruning schedule itself and this is useful, but it is not clear that learning weights is the best way. A discussion on this might be relevant if future works build on these methods. \n- From the experimental results, it is not clear that the proposed method is significantly better than the baselines. A lot of the results of the proposed method are within 95% confidence intervals of other methods (and vice versa). This makes it hard to say that the proposed method is significantly better. I recommend using the Interquartile Mean (IQM) as suggested by [4], with at least 5 seeds (as you have done for all environments except SMAC V2) to help clarify if the proposed method is better in a statically significant way. Furthermore, I am unable to find the details of the evaluation procedure in the paper, such as the evaluation interval and how many evaluation steps are used. This should be included in the paper.\n- An analysis of the computation cost (speed and memory) during training time might be relevant. Parameter sharing generally reduces the number of parameters required, but with learnable masks, this might not be the case. This should be discussed in the paper.\n\n1] Christianos, F., Papoudakis, G., Rahman, M. A., & Albrecht, S. V. (2021, July). Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning (pp. 1989-1998). PMLR.\n\n2] Kim, W., & Sung, Y. (2023). Parameter sharing with network pruning for scalable multi-agent deep reinforcement learning. arXiv preprint arXiv:2303.00912.\n\n3] Li, D., Lou, N., Zhang, B., Xu, Z., & Fan, G. (2024, April). Adaptive parameter sharing for multi-agent reinforcement learning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6035-6039). IEEE.\n\n4] Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., & Bellemare, M. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, 29304-29320.\n\nQUESTIONS:\n1. Is there a motivation for using STR over other methods that learn different activations per agent, like SNP?\n2. Can you provide more details on the evaluation procedure, such as the evaluation interval and how many evaluation steps are used?\n3. Was there a reason for not including FuPS+ID as a baseline in the experiments?\n\nLIMITATIONS:\nThe authors address the limitations in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach to multi-agent reinforcement learning.  In this approach, the agent network has an overall set of parameters.  These parameters are transformed by agent-specific masks.  The masks are learnable.  Agent policies are encouraged to be diverse through a diversity-regularisation term.  Masks and diversity regularisations are also applied to the critic network parameters.  Experimental results compare the performance of Kaleidoscope with other methods.\n\nSTRENGTHS:\nThe introduction of masking as part of the learnable parameters for MARL agents is interesting.  It removes a previous limitation of parameter sharing, and allows this to be flexible.  The contributions are made clear by the authors. Figure 2 illustrates the concept of masking well, which is important to the core of the work.  The manuscript is generally easy to read, barring some minor comments pointed out in the Questions section.\n\nWEAKNESSES:\n- The current limitations of the work are presented in an Appendix.  It would be good to see this fleshed out with more details and presented as part of the main text.  For example, it would be interesting if the authors provide a guideline for how their method would scale with the number of agents. \n- While the presentation of the material is clear, it is repetitive in places.  It would be nice to make the text more concise and provide some more interpretation of the results. \n- In general, the Figure captions can be made more self-contained. \n- Minor comments:\n   * In Section 2, after line 78, it appears that the meaning of $O$ is not explained in the MDP $\\mathcal{M}$. \n   * Line 89 appears to have either grammatical errors or a confusing choice of words between singular and plural. \n   * Line 277 has a typo in MARL.\n\nQUESTIONS:\nIt would be nice if the authors are able to address the points in the Weaknesses section.\n\nLIMITATIONS:\nNo.  The current limitations of the work are presented in an Appendix.  It would be good to see this fleshed out with more details and presented as part of the main text.  For example, it would be interesting if the authors provide a guideline for how their method would scale with the number of agents.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel adaptive partial parameter sharing scheme in multi-agent reinforcement learning (MARL) to enhance policy heterogeneity while maintaining high sample efficiency. The key innovation, Kaleidoscope, employs a set of common parameters and multiple sets of distinct, learnable masks for different agents, encouraging policy diversity without sacrificing the benefits of parameter sharing. This approach dynamically balances the trade-off between full parameter sharing and non-parameter sharing, thereby bridging the gap between the two. Furthermore, Kaleidoscope is extended to critic ensembles within actor-critic algorithms to improve value estimations. Empirical evaluations across various environments, including multi-agent particle environment, multi-agent MuJoCo, and StarCraft multi-agent challenge v2, demonstrate that Kaleidoscope outperforms existing parameter sharing methods.\n\nSTRENGTHS:\nThis paper investigates the limits of parameter sharing in MARL and conducts conprehensive experiments.\n\nWEAKNESSES:\n### Methodology\n1. The proposed method is similar to SNP, and the difference is not fully discussed.\n2. The learned masks can not be explained. The proposed method can be regarded as enlarging the network into a Mix-of-Expert network with a dense router, which is expected to improve the performance. However, the proposed method does not leverage the relationships among agents.\n3. Pseudocode is lacking. The proposed method has several components, it is better to include a pseudocode in the main paper.\n4. The proposed method maintains N masks whose sizes are similar to those of the actor and critic networks, which may result in low training efficiency.\n\n### Experiments\n1. The authors should consider implementing a simple and intuitive partial parameter sharing network as a baseline, such as a shared trunk with multiple action heads.\n2. Apart from Table 2, the authors should compare the training costs among those algorithms.\n3. The ablation study is incomplete. The main new idea behind the proposed method is to share parameters using learnable masks. Because of this, the authors should show how this main new idea improves performance instead of getting rid of secondary methods in Figure 6.\n\nQUESTIONS:\n1. What is the purpose of citing LLM works in lines 35 and 36?\n2. Are there any other works discussing parameter sharing apart from references [12,13,14] in this paper?\n3. How to distinguish `Neurons` and `Weights` in Table 1?\n4. In the experiments, are algorithms using similar sizes of networks? For a fair comparison, the network sizes for agent policies and critics should be similar.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Kaleidoscope, an adaptive partial parameter sharing method for multi-agent reinforcement learning (MARL). Kaleidoscope aims to balance the benefits of full parameter sharing (sample efficiency) with the flexibility of non-parameter sharing (policy diversity). It achieves this by using learnable binary masks to control which parameters are shared between agent networks and within critic ensembles. The authors demonstrate Kaleidoscope's effectiveness on various MARL benchmarks, showcasing superior performance compared to existing parameter sharing approaches.\n\nSTRENGTHS:\n*Originality*: The application of STR for dynamic partial parameter sharing in MARL is novel and well-motivated. The paper clearly distinguishes itself from prior work on partial parameter sharing that relies on fixed initializations.\n\n*Quality*:  The proposed method is technically sound. The combination of STR with diversity regularization and periodic resetting is well-reasoned and contributes to the effectiveness of Kaleidoscope. Experimental results across diverse MARL benchmarks provide strong support for the claims. The ablation is also helpful to understand the contribution each component of Kaleidoscope has on final performance.\n\n*Clarity*: The paper is generally well-written and organized. The core idea of using learnable masks is clearly conveyed through figures and explanations. The technical details are adequately provided, allowing for reproducibility.\n\n*Significance*: Addressing the trade-off between sample efficiency and policy diversity is a crucial challenge in MARL. Kaleidoscope's adaptive and dynamic nature makes it a potentially valuable contribution to the field. The improved performance on diverse benchmarks suggests that the proposed method can be broadly applicable.\n\nWEAKNESSES:\nThe paper primarily focuses on environments with a relatively small number of agents. The scalability of Kaleidoscope to environments with  tens or hundreds of agents. In that case it is not clear if a large number of masks could have a negative performance compared with the parameter sharing baseline.\n\nQUESTIONS:\nHow does the computational overhead of Kaleidoscope scale with the number of agents?\n\nImpact of Mask Sparsity: Does the sparsity level of the masks induced by STR vary across different tasks and environments? Are there any insights into how the sparsity level affects the policy diversity and overall performance?\n\nIntegration with other MARL Algorithms: The paper demonstrates Kaleidoscope with MATD3 and QMIX. How straightforward would it be to adapt the method to other MARL algorithms such MAPPO? Are there any specific challenges or considerations?\n\nReal-world Applications: Can you envision any specific real-world applications where Kaleidoscope's ability to promote agent heterogeneity would be particularly beneficial?\n\nLIMITATIONS:\nThe authors have properly addressed the limitations of their paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}