Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning
Authors: Xinran Li
Ling Pan
Jun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-agent reinforcement learning; parameter sharing
Primary area: reinforcement_learning
TL;DR: A learnable partial parameter sharing mechanism for multi-agent reinforcement learning

Abstract: In multi-agent reinforcement learning (MARL), parameter sharing is commonly employed to enhance sample efficiency. However, the popular approach of full parameter sharing often leads to homogeneous policies among agents, potentially limiting the performance benefits that could be derived from policy diversity. To address this critical limitation, we introduce \emph{Kaleidoscope}, a novel adaptive partial parameter sharing scheme that fosters policy heterogeneity while still maintaining high sample efficiency. Specifically, Kaleidoscope maintains one set of common parameters alongside multiple sets of distinct, learnable masks for different agents, dictating the sharing of parameters. It promotes diversity among policy networks by encouraging discrepancy among these masks, without sacrificing the efficiencies of parameter sharing. This design allows Kaleidoscope to dynamically balance high sample efficiency with a broad policy representational capacity, effectively bridging the gap between full parameter sharing and non-parameter sharing across various environments. We further extend Kaleidoscope to critic ensembles in the context of actor-critic algorithms, which could help improve value estimations. Our empirical evaluations across extensive environments, including multi-agent particle environment, multi-agent MuJoCo and StarCraft multi-agent challenge v2, demonstrate the superior performance of Kaleidoscope compared with existing parameter sharing approaches, showcasing its potential for performance enhancement in MARL. The code is publicly available at \url{https://github.com/LXXXXR/Kaleidoscope}.

Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning
Xinran Li
Ling Pan
Jun Zhang
Department of Electronic and Computer Engineering
The Hong Kong University of Science and Technology
xinran li@connect......omooomoo..o....e ust.hk, lingpandust.hie eejzhangdust.
Abstract
In multi-agent reinforcement learning (MARL), parameter sharing is commonly employed to enhance sample efficiency. However, the popular approach of full parameter sharing often leads to homogeneous policies among agents, potentially limiting the performance benefits that could be derived from policy diversity. To address this critical limitation, we introduce Kaleidoscope, a novel adaptive partial parameter sharing scheme that fosters policy heterogeneity while still maintaining high sample efficiency. Specifically, Kaleidoscope maintains one set of common parameters alongside multiple of distinct, learnable masks for different agents, dictating the sharing of parameters. It promotes diversity among policy networks by encouraging discrepancy among these masks, without sacrificing the efficiencies of parameter sharing. This design allows Kaleidoscope to dynamically balance high sample efficiency with broad policy representational capacity, effectively bridging the gap between full parameter sharing and non parameter sharing across various environments. We further extend Kaleidoscope to critic ensembles the context actor-critic algorithms, which could help improve value estimations. Our empirical evaluations across extensive environments, including multi-agent particle environ- ment, multi-agent MuloCo and StarCraft multi-agen challenge v2, demonstrate the superior performance of Kaleidoscope compared with existing parameter sharing approaches, showcasing its potential for performance enhancement in MARL, The code is publicly available at [hhttss //ggthubb com//XXXX///aaaeisssssrr..
Introduction
Cooperative multi-agent reinforcement learning (MARL) has demonstrated remarkable effectiveness in solving complex real-world decision-making problems across various domains, such as resource allocation (Ying and Dayong 2005), package delivery (Seuken and Zilberstein 2007). autonomous driving (Zhou et al. 2021), and robot control (Swamy et al. 2020). To mitigate the challenges posed by the non-stationary and partially observable environments typical of MARL (Yuan al. 2023). the centralized training with decentralized execution (CTDE) paradigm (Foerster et 2016) has become prevalent, inspiring many influential MARL algorithms such as MADDPG (Lowe et al. 2017). COMA (Foerster et al. 2018). MATD3 (Ackermann al. 2019), QMIX (Rashid et al. 20200)
and MAPPO (Yu et al. 2022).
Under the CTDE paradigm, parameter sharing among agents is commonly adopted practice to improve sample efficiency. However, identical network parameters across agents often lead to homogencous policies, restricting diversity in behaviors and the overall joint policy representational capacity. This limitation can result undesired outcomes in certain situations (Christianos et al.)
2021
Fu et al. 2022 Kim and Sung 2023), as shown in Figure I impeding further performance
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

gains. An alternative approach is the non por-ammeteer sharing scheme, where each agent possesses its own unique parameters. Nevertheless, while this method naturally supports heterogeneous policies, it suffers from reduced sample efficiency, leading to significant training costs. This is particularly problematic given the current trend towards increasingly large model sizes, with some scaling to trillions of parameters (Zhao et al. 2023 Acchamm et al. 2023). Therefore, it is imperative to develop a parameter sharing strategy that enjoys both high sample efficiency and broad policy representational capacity, potentially achieving significantly enhanced performance. While several efforts (Christianos ct al. 202T; Kim and Sung, 2023) have explored partial parameter sharing initiated at the start of training, such initializations can be challenging design without detailed knowledge of agent-specific environmental transitions or reward functions (Christianos et al. 2021
In this work, we build upon insights from previous studies (Christianos et al., 2021 et al., 2022 Kim and Sung 2023) and introduce Kaleidoscope, novel adaptive partial parameter sharing scheme. It maintains single set of policy parameters and employs multiple learnable masks to designate the shared parameters. Unlike earlier methods that depend on fixed initializations, Kaleidoscope dynamically learns these masks alongside MARL param- eters throughout the training process. This end-to-end training approach inherently integrates environmental information, and its adaptive nature enables Kaleidoscope to dynamically adjust the level of parameter sharing based the demands of the environment and the learning progress of the agents. The learnable masks facilitate a dynamic balance between full parameter sharing and non- parameter sharing, offering a flexible trade-off between sample efficiency and policy representational capacity through enhanced heterogeneity. Initially, we build Kaleidoscope upon agent networks, where it achieves diverse policies. Following this success, we extend it to multi-agent actor-critic algorithms to encourage heterogeneity among the central critic ensembles for further performance enhancement.
Food
Forest
Obstacle
Prey
Predator
Full parameter sharing confines the policies to be homogeneous, In this example, all predators pursue the same prey, neglecting another prey in the game World. Further game details are in Appendix A.2
Figure 1:
Just like kaleidoscope the reflective properties of rotating mirrors transform simple shapes into beautiful patterns, our proposed method leverages learnable masks to map single set of parameters into diverse policies, thereby enhancing task performance.
We summarize our contributions as follows:
To enable policy heterogeneity among agents for better training flexibility, we adapt the soft threshold reparameterization (STR) technique learn distinct masks for different agent net- works while only maintaining one set of common parameters, effectively balancing between full parameter sharing and non-parameter sharing mechanisms.
To enhance policy diversity among agents. introduce novel regularization term that encour- ages the pairwise discrepancy between masks. Additionally, we design resetting mechanisms that recycle masked parameters to preserve the representational capacity of the joint networks.
Through extensive experiments MARL benchmarks, including multi-agent particle environ- ment (MPE) (Lowe et al.) 2017), multi-agent MuJoCo (MAMuloCo) (Peng et al.] (021) and StarCraft multi-agent challenge v2 (SMACv2) (Ellis al. 2024), demonstrate the superior performance of Kaleidoscope over existing parameter sharing approaches.
2 Background
Multi-agent reinforcement learning (MARL) In MARL, a fully cooperative partially observable multi-agent task is typically formulated as decentralized partially observable Markov decision process (dec-POMDP) (Oliehoek and Amato] 2016). represented by a tuple
M = (S, A, P. R, Q2 O, 7))). Here, N denotes the number of agents, and 7 E (0. represents the discount factor. each timestep I, with the environment state as s E S, agent receives local observation of e Ω drawn from the observation function O(s i) and then follows its local policy

---

Ty to select an action of E A. Individual actions form a joint action a' E AN leading to state a') and inducing global reward r' R(s', a²). The
transition the next state sh+1 P(s 1 |s", overall team objective is to learn the joint policies π (- NN) such that the expectation of discounted accumulated reward G = E. 7'r" maximized.
To learn such policies Te. various MARL algorithms (Lowe et al. 2017 Foerster et al. 2018 Rashid al. 2020, Yu et al. (2022) have been developed. For instance, the off-policy actor-critic algorithm MATD3 (Ackermann et al. 2019) serves as an example method. Specifically, the critic networks are updated by minimizing the temporal difference (TD) error loss
Le(#) Eieet
a' o' o'+1)D (v' Q(st,
(1)
with
y'=r' + 1 Q(s TI (oitt + TN, :0w) + +;;
(2)
where & are the parameters for critics, are the parameters for actor policies and 6' are the parameters for target actor policies. And e is the clipped Gaussian noise, given as clip(N -c,
The policy is updated by the deterministic policy gradient algorithm (Silver al. 2014)
(..) of at
V Te [: V,) 01, aI,...,aylaa...... (olamm b1) (3)
Soft threshold reparameterizatiio (STR) Originally introduced in the context of model sparsifica-
tion, STR (Kusupati et al. (020) is an unstructured pruning method that achieves notable performance without requiring predetermined sparsity level. Specifically, STR applies transformation to the original parameters W as follows
SW,, s) = sign(W) ReLU ([W) - -g()),
(4)
where is learnable parameter, a = g(s) serves as the pruning threshold, and ReLU(.) = max(. ; 0).
The original supervised learning problem modeled by
min L(W;D)
(5)
with D as the data now transferred to
min C(S,(W. s): W.a
(6)
Overall, this approach optimizes the learnable pruning threshold alongside the model parameters, facilitating dynamic adjustment to the sparsity level during training.
3 Learnable Masks for Heterogenous MARL
In this section, we propose using learnable masks as low-cost method to enable network heterogene- ity in MARL. The core concept, illustrated in Figure 2 is to learn a single set of shared parameters complemented by multiple masks for distinct agents, specifying which parameters to share.
Specifically, in Section 3.1. we first adapt STR into dynamic partially parameter sharing method, unlocking the joint policy network's capability to represent diverse policies among agents. In Section 3.2 we actively foster policy heterogeneity through a novel regularization term based on the masks. Given that the masking technique could excessively sparsify the network, potentially diminish- ing its representational capacity, in Section 3.3 propose straightforward remedy to periodically reset the parameters based on the outcomes of masking, which additionally mitigates primacy bias. Finally, in Section 3.4 we explore how to further extend this approach within the critic components of actor-critic algorithms to improve value estimations MARL and further boost performance.
For the sake of clarity, we integrate the proposed Kaleidoscope with the MATD3 (Ackermann jet al., 2019) algorithm to demonstrate the concept within this section. Nevertheless, as a versatile partial parameter-sharing technique, our method can readily be adapted to other MARL algorithms. We defer its integration with other MARI. frameworks Appendix A.1.2 and will evaluate them empirically in Section 4
3

---

Q ps)
Critic Ensembles
( Px)
Q v1)
26:+) 9,3
(=:4x)
6
O
Mi
Mean
60
M
4x
Actor1
8.)
Actor
m((:6)
Actor ky(:0w)
0
M
Pp
M
6n
Figure 2: Overall network architecture of Kaleidoscope. It maintains one set of parameters dg with N sets of masks [M.]]= for actor networks, and one set of parameters so with K sets of masks [Mq]] j=1 for critic ensemble networks, where N is the number of agents, K is the number of ensembles, and @ denotes the Hadamard product.
3.1 Adaptive partial parameter sharing
The core idea of this work is to learn distinct binary masks M, for different agents to facilitate differentiated policies, ultimately aiming improve MARL performance. To achieve this, we apply the STR (Kusupati et al. 2020) technique to the policy parameters with different thresholds dedicated to each agent:
0, 0o M,,
where 0, parameterizes the policy for agent i, 0g is the set of learnable parameters shared by all agents, and M, is the learnable mask for agent i. Specifically, assume 6g = (000) (N,)'
ofWaJ]] and M, [mm" "m."......!.... with N. being the total parameter count of an agent's network. In line with STR, we compute each element (k) of M; as m
0: (0()) "o!")...,!!!""
mNaa)
1 (s("))) where of) denotes the Sigmoid function.
The benefits of such combination are summarized as follows:
Unlike most of the methods in pruning literature, which primarily aim minimize the discrepancies between pruned and unpruned networks in terms of weights, loss, activations (Hoefler et al.) [021 Menghani 2023 Deng (et al.] 2020), STR maintains the original goal of minimizing task- task-specific loss, aligning directly with objectives enhance MARL performance.
Preservation of original MARL learning objectives:
Flexibility sparsity: Many classical pruning methods require predefined per-layer sparsity levels (Evei et al.) 2020) Ramanujan al. 2020). Such requirements complicate design, with the goal not to gain extreme sparsity but rather to promote heterogeneity through masking. does not require predefining sparsity levels, allowing
The STR technique ideal in our
for adaptive learning of the masks.
Enhanced network representational capacity: Utilizing learnable masks for adap- tive partial parameter sharing enhances the network's representational capacity beyond traditional full parameter sharing. In full parameter sharing. agents' joint policies are parameterized as (-1(-))))
(TI(-180),... x (-16u)). In contrast, our pro- posed adaptive partial parameter sharing mechanism parameterizes the joint policies as I Kaleidoscope (-10g. M) (T (-10p @ Mi).... ..... (:00 @ Mx)). . In the extreme case where all the values in M, are 1s, the function set represented by π Kaleiiosccope-1ff M) degrades to that of PT8 (-100)) In other scenarios, is superset of that represented by T (-10%)

---

Actors
Actors
Critie Ensembles
Critic Ensembles
0g
M
M,
d
O
M
O
8
9
4u
Mf
(a) Actors:
reinitialize the weights that are masked by all agents with probability
(b) Critic ensembles: reset one set of masks at a time.
Figure 3: Illustration on resetting mechanisms.
3.2 Policy diversity regularization +
While independently learned masks enable agents to develop distinct policies, without a specific in- centive, these policies may still converge being homogeneous. To this end, we propose explicitly encourage agent policy heterogeneity by introducing a diversity regularization term maximizing the weighted pairwise distance between network masks, which is defined
Johl Σ
Σ
110 @ (M, - M1)111
(8)
This term is inherently non-differentiable due to the indicator function [ inside M. To overcome this difficulty, following established practices in the literature (Bengio et al. (2033 Alizadeh et al.) 2018), utilize surrogate function for gradient approximation:
a tanh
(9)
ag(s:)
iM,
We formally provide the overall training objective for actors in Appendix A.1.1
3.3 Periodically reset
As the training with masks proceeds, we observe an increasing sparsity each agent's network, potentially reducing the overall network capacity. To remedy the issue, we propose simple approach to periodically reset the parameters that are consistently masked across all M, with certain
probability p, which is illustrated in Figure 3a
At intervals defined by t mod reset interval ==
the parameter index satisfies Vi, m (k)
O, we apply the following resetting rule
Reinitiaaliz (k) (b)
(k) with probability p with probability 1 p
(k)
(K)
(10)
(k)
This resetting mechanism recycles the weights masked as zeros by all the masks, preventing the networks from becoming overly sparse. A side benefit of this resetting mechanism is the enhancement of neural plasticity (Lyle et al.) 2023; Nikishin et al., (2024), which helps alleviate the primacy bias (Nikishin et al. 2022) in reinforcement learning. Unlike methods that reinitialize entire layers resulting in abrupt performance drops (Nikishin et al. (022). our resetting approach selectively targets weights indicated by the learnable masks, thus avoiding significant performance disruptions, as shown in Section
3.4 Critic ensembles with learnable masks
In actor-critic algorithm frameworks, we further apply Kaleidoscope to central critics as efficient way to implement ensemble-like critics. By facilitating dynamic partial parameter sharing, Kalei- doscope enables heterogeneity among critic ensembles. Furthermore, by regularizing the diversity
5

---

among critic functions, we can control ensemble variances. This approach is elaborated in subsequent paragraphs.
Adaptive partial parameter sharing for critic ensembles In the standard MATD3 algo- rithm (Ackermann al. 2 two critics with independent parameters maintained to mitigate overestimation risks. However, using separate parameters typically results a low update-to-de to-tta (UTD) ratio (Hiraoka al. 2022). To address this issue, propose enhance the UTD ratio by employing Kaleidoscope parameter sharing among ensembles of critics. Specifically, we maintain a single of parameters do and K masks [Mj] 5 to distinguish the critic functions, resulting in K ensembles [Q(: Φy ر with Φ, = do @ Mý.
To be specific, we update the critic networks by minimizing the temporal difference (TD) error loss
Lo(d;) E a (v'' ass',,''
(11)
with
y' = +
min j=l...,K
Q(s
, t,,..,, Tm (n ( E;
(12)
And the policies are updated by the mean estimation of the ensembles
1 K
VJ(0,) ES'DD Vami(o;
Q(s ........ aN (0 )1)
(13)
we also apply diversity regular- ization to critic masks to prevent critics functions from collapsing to identical ones. The diversity regularization maximize for the critic ensembles is expressed
Critic ensembles diversity regularization As in Section 3.2
Jrllv (s
Σ i= J=
Σ
ce o (M. M2)
(14)
Intuitively, as training progresses, this term encourages divergence among the critic masks, leading to increased model estimation uncertainty. This process fosters a gradual shift from overestimation to underestimation. As discussed in prior research (Hiraoka et al. [0022] Lan et al., [2020] Chen et al.) 202T; Wang al., (2021b), overestimation can encourage exploration, beneficial
early training
stages, whereas underestimation alleviates еπrоr accumulation (Fujimoto et al. 2018). which is
preferred in the late training stage. We formally provide the overall training objective for critic ensembles in Appendix A.1.1 A.1.I
Periodically reset To further promote diversity among critic ensembles and counteract the reduction in network capacity caused by masking, we implement a resetting mechanism similar to sequentially reinitialize the masks MC following
that described in Section 3.3 In particular, cyclic pattern, as illustrated in [3] In this way, each critic function's 's mask is trained on distinct data segments, leading different biases.
In summary, by adopting Kalcidoscope parameter sharing with learnable masks, we establish a cost-effective implementation for critic ensembles that enjoy high UTD ratio. Through enforcing distinctiveness among the masks, we subtly control the differences among critic functions, thereby improving the value estimations in MARL.
4 Experimental Results
In this section, we integrate Kaleidoscope with the value-based MARL algorithm QMIX and the actor-critic MARL algorithm MATD3, and evaluate them across eleven scenarios in three benchmark tasks.

---

Table 1: Methods compared in the experiments. Here, "adaptive" indicates whether the sharing
scheme evolves during training.
Methods
Paradigm
Sharing level
Adaptive
Descriptions
NoPS
No sharing
No
Agents have distinct parameters
FuPS
Full sharing
Networks Networks
No
Agents share all the parameters Agents share all the parameters with agent IDs in input Agents are clustered to share pa- rameters within each cluster Agents share all the parameters ex- cept for distinct action heads Agents share specific neurons based fixed, random pruning Agents share parameters based on distinct, learnable masks
FuPS
Full sharing
No
ScPS
Partial sharing
Networks
No
MultiH
Partial sharing
Layers
No
SNP
Partial sharing
Neurons
No
Kaleidoscope
Partial sharing
Weights
Yes
4.1 Experimental Setups
Environment descriptions We test our proposed Kaleidoscope on three benchmark tasks: MPE (Lowe et al. 2017), MaMuJoCo (Peng et al., 202T] and SMACv2 (Ellis et al. 2024). For the discrete tasks MPE and SMACv2, we integrate Kaleidoscope and baselines with QMIX (Rashid et al. 2020) and assess the performance. For the continuous task MaMuJoCo, MATD3 (Ackermann et al., (219).. We use five random seeds for MPE and MaMuJoCo and three random seeds for SMACv2, reporting averaged results and displaying the 95% confidence interval with shaded areas. The chosen benchmark tasks reflect a mix of discrete and continuous action spaces and both homogeneous and heterogeneous agent types, detailed further in Appendix A.2
employ
Baselines In the following, we compare our proposed Kaleidoscope with baselines (Christianos) et al. 2021 Kim and Sung 2023), as listed Table I For both Kaleidoscope and the baselines, in scenarios with fixed agent types (MPE and MaMuloCo), we assign one mask per agent. For SMACv2, where agent types vary, we assign one mask per agent type. We use official implementations of the baselines where available; otherwise, we closely follow the descriptions from their respective papers, integrating them into QMIX or MATD3. Hyperparameters and further details are provided in A.I.. A.1.3
Appendix
4.2 Results
Performance We present the comparative performance of Kaleidoscope and baselines in Figure and F5gure Overall, Kaleidoscope demonstrates superior performance, attributable to the flexibility of the learnable masks and the effectiveness of diversity regularization. Additionally, we observe that FuPS + ID generally outperforms NoPS, except for the Ant-v2-4x2 scenario (Figure [4c). This advantage largely due FuPS's higher sample efficiency; a single transition data sample updates the model parameters N times in FuPS + ID, once for each agent, compared to just once in NoPS. Consequently, FuPS + models learn faster from the same number of transitions. Similarly, Kaleidoscope benefits from this mechanism as it shares weights among agents, allowing a single transition update the model parameters multiple times. Furthermore, by integrating policy heterogeneity through learnable masks, Kaleidoscope enables diverse agent behaviors, as illustrated in the visualization results in Figure Ultimately, Kaleidoscope effectively balances parameter sharing and diversity, outperforming both full parameter sharing and non-parameter sharing approaches.
Cost analysis Despite its superior performance, Kaleidoscope does not increase computational com- plexity at test time compared to the baselines. We report the test time averaged FLOPs comparison of Kaleidoscope and baselines in Table We see that due to the masking technique, Kaleidoscope has lower FLOPs compared baselines, thereby enjoying a faster inference speed when being deployed.
7

---

Kaleidoscope
SNP
MultiH
SePS
+
FuPS
NoPS
4000
Test Return
Test Return
5000
200
4000
2000
1000
-20
8.00
8.00
0.50
Timestaps
166
Timesteps
167
Timesteps
(a) World
(b) Push
(c) Ant-v2-4x2
(d) Hopper-v2-3xl
6000
Test Return
neturn
Test Return
0000
4000
800
4000
4000
2000
2000
2000
100
8.00
0.75
1.00
8.00
8.00
8.00
0.25
1,00
Timesteps
107
Timesteps
167
Timesteps
107
Timesteps
167
(e) Walker2D-v222x3
(1) Walker2D-v2-6x1
(g) HallCheetah-v222x3
(h) Swimmer-v2-10x2
Figure Performance comparison with baselines on MPE and MaMuJoCo benchmarks.
Test Win Itate 96
60
Test Win Rate %
60
Test Win Rate 86
Kaleidoscope
SNP
MultiH
40
40
20
20
20
FuPS
NoPS
8.0
1.0
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Timesteps
Timesteps
Timesteps
107
(a) Terran 5
(b) Protoss
S vs
(c) 5evs___vvs$
Figure 5: Performance comparison with baselines on SMACv2 benchmarks.
Ablation studies We conduct ablation studies to assess the impact key components Kaleido- scope, with results presented in Figure Specifically, we compare Kaleidoscope with three ablations: 1) Kaleidoscope w/o reg, which lacks the regularization term in Equation masks to be distinct. 2) Kaleidoscope w/o reset, which does not reset parameters. 3) Kaleidoscope w/o ce, which does not use Kaleidoscope parameter sharing in critic ensembles and instead maintains two independent sets of parameters for critics. From the results, we observe that diversity regular- ization contributes the most to the performance of Kaleidoscope. Without it, masking degrades the performance due the reduced number of parameters in each policy network. Resetting primarily aids learning in the late stages training when needed, which aligns with the observation made by Nikishin et al. (2022). Notably, even with resetting, the performance does not experience abrupt drops thanks to the guidance provided by the masks on where to reset. When ablating the critic ensembles with Kaleidoscope parameter sharing, we observe inferior performance from the beginning of the training. This is because the critic ensembles with Kaleidoscope parameter sharing enable a higher UTD ratio of the critics, discussed in Section 3.4
(8) that encourages
we conduct experiments to study the impact of mask designs. The results are shown in Figure Figure 7 Specifically, compare original Kaleidoscope with two alternative mask design choices: 1) Kaleidoscope w/ neuron masks, where adaptive masking techniques are applied to neurons rather than weights. 2) Kaleidoscope w/ fixed masks, where the masks initialized at the beginning of training and kept fixed throughout the learning process. The results show that performance drops with either alternative design choice, demonstrating that Kaleidoscope's superior performance originates from the flexibility of the learnable masks on weights.
Furthermore,
More results on hyperparameter analysis included in Appendix B.2

---

Table 2: Averaged FLOPs (with calculation methods detailed in Appendix A.3) across different methods. Results are first normalized with respect to the FuPS + ID model for each scenario and then averaged across scenarios within each environment (detailed results in Appendix B.T]. The lowest costs are highlighted in bold.
Methods
NoPS
FuPS
FuPS +ID
SePS
MultiH
SNP
Kaleidoscope
MPE
1.0x
0.992x
1.0x
1.0x
1.0x
0.988x
0.901x
MaMuJoCo
1.0x
0.985x
1.0x
1.0x
1.xx
0.900x
0.680x
SMACv2
1.0x
0.992x
1.0x
1.0x
1.0x
0,988x
0.890x
6000
6000
Test Retur
Test Reture
Return
200
4000
200
4000
Tast
2000
2000
8.00
8.00
Timesteps
1e5
Timesteps
167
Timesteps
166
Timestops
1e7
(a) World
(b) Anx-v2-4x2
(a) World
(b) Ant-v2-4x2
Figure 6: Ablation studies.
Figure 7: Comparison on mask designs.
Visualization We visualize the trained policies of Kaleidoscope on World, as shown in Figure Sa The agents exhibit cooperative dewide-and-conen strategies (four red agents divide into two teams and surround the preys), contrasting with the homogeneous policies depicted in Figure [ We further examine the distinctions in the agents' masks and present the results in Figure [bb] First, we observe that by the end of the training, each agent has developed a unique mask, revealing that distinct masks facilitate diverse policies by selectively activating different segments of the neural network weights. Second, throughout the training process, we note that the differences among the agents' masks evolve dynamically. This observation confirms that Kaleidoscope effectively enables dynamic parameter sharing among the agents based on the learning progress, empowered by the adaptability of the learnable masks. More visualization results are provided in Appendix B.3
Related Work
Parameter sharing First introduced by Tan 1993). parameter sharing has been widely adopted in MARL algorithms (Foerster et al.] [0018] Rashid et al. 2020: Yu et al. 2022), due to its simplicity and high sample efficiency (Grammel et al.) 2020). However, schemes without parameter sharing typically offer greater flexibility for policy representation. To balance sample efficiency with policy representational capacity, some research efforts aim to find effective partial parameter sharing schemes.
Notably, SePS (Christianos et al. (22TT) first clusters agents based on their transitions at the start of training and restricts parameter sharing within these clusters. Subsequently, SNP (Kim and Sung 2023) enables partial parameter sharing by utilizing the lottery ticket hypothesis (Su et al. 2020 to initialize heterogeneous network structures. Concurrent to our work, AdaPS (Li et al. 2024) combines SNP and SePS by proposing a cluster-based partial parameter sharing scheme. While these methods have shown promise in certain domains, their performance potential is often limited by the static nature of the parameter sharing schemes set early in training. Our proposed Kaleidoscope distinguishes itself by dynamically learning specific parameter sharing configurations alongside the development of MARL policies, thereby offering enhanced training flexibility.
Agent heterogeneity in MARL behaviors among agents, previous methods have explored concepts such as diversity and roles. Specifically, diversity- based approaches aim to enhance pairwise distinguishability among agents based on identities (Jiang and Lu (021). trajectories (Li et al.) 2021), or credits assignment (Liu jet al., 2023 Hu et al., 2023) through contrastive learning techniques. Concurrently, role-based strategies, sometimes referred to as skills (Yang et al.) 2020) or subtasks (Yuan et al.) 2022). employ conditional policies to differentiate agents by assigning them to various conditions. These conditions may be based on agent identities Yangg al. 2022). local observations (Yang al. 2020). local
To incorporate agent heterogeneity in MARL and enable diverse

---

Timesteps
Timesteps 2M
0.06
Agent
0.04
0.00
1
Agent
Agent
Agent
(a) Trained policies.
(h) Pairwise mask differences among agents different training timesteps.
Figure 8: Visualization on World.
histories Wang et al.,
2020 2021a Yuan et al. 2022) or joint histories (Liu et al. 2021 Iqbal
al. 2022: Zeng )023). This line of researches mainly focus on module design and operate separately from parameter-level adjustments, making them orthogonal to our approach. Nevertheless, integrating these methods with our work could potentially enhance performance further.
Sparse networks deep reinforcement learning (RL) Although relatively few, there are some noteworthy recent attempts to find sparse networks for deep RL. In particular, PoPS (Livne and Cohen (020) prunes the dense networks post-training, achieving significantly reduced execution time complexity. Additionally, Yu et al. 2020) validate the lottery ticket hypothesis within the RI. domain, producing high-performance models even under extreme pruning rates. Subsequent efforts, including DST (Sokar et al. 2022). TE-RL* (Graesser et al.) 2022) and RLx2 (Tan al.) 2023) employ topology evolution (TE) techniques to further decrease the training costs. While these developments utilize sparse training techniques, which similar to the methods we employ, their primary focus is on reducing training and execution costs in single-agent settings. In contrast, our work leverages sparse network strategies as means to enhance parameter sharing techniques, aiming to improve MARL performance.
Conclusions and Future Work
In this work, introduced Kaleidoscope, novel adaptive partial parameter sharing mechanism for MARL. leverages distinct learnable masks to facilitate network heterogeneity. applicable to both agent policies and critic ensembles. Specifically, Kaleidoscope is built three technical components: STR-empowered learnable masks, network diversity regularization, and periodic resetting mecha- nism. When applied agent policy networks, Kaleidoscope balances sample efficiency and network representational capacities. In the context of critic ensembles, improves value estimations. By combining our proposed Kaleidoscope with QMIX and MATD3, we have empirically demonstrated its effectiveness across various MARL benchmarks. This study shows great promises in developing adaptive partial parameter sharing mechanisms enhance the performance of MARL. For future work, it is interesting to further extend Kaleidoscope to other domains such offline MARL or meta-RL.
Acknowledgements
This work was supported by the Hong Kong Research Grants Council under the NSFC/RGC Collab- orative Research Scheme grant CRS HKUST603/22. And thank the anonymous reviewers for their valuable feedback and suggestions.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023, GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023).
Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. 2019. Reducing overestimation bias in multi-agent domains using double centralized crities. arXiv preprint arXiv:1910.01465 (2019).
10

---

Milad Alizadeh, Javier Fernández-Marqués, Nicholas D Lane, and Yarin Gal. 2018. An empirical study of binary neural networks' optimisation. In Proceedings of the International Conference on Learning Representations.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprini arXiv:/308.344 /308.3432 (2013).
Xinyue Chen, Che Wang. Zijian Zhou, and Keith W. Ross. 2021. Randomized Ensembled Double Q-Learning; Learning Fast Without a Model. In Proceedings of the 9th International Conference on Learning Representations.
Filippos Christianos, Georgios Papoudakis, Arrasy Rahman, and Stefano V. Albrecht. 2021. Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing. Proceedings of the 38th International Conference Machine Learning, Vol. 139, PMLR, 1989-1998.
Lei Deng. Guoqi Li, Song Han, Luping Shi, and Yuan Xie. 2020. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proc. IEEE 108, 4 (2020), 485-532.
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. 2024. Smacv2: An improved benchmark for coopera- tive multi-agent reinforcement learning. In Advances Neural Information Processing Systems, Vol.36.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. 2020. Rigging the lottery: Making all tickets winners. In Proceedings of the International Conference on Machine Learning. PMLR, 2943-2955
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. 2016. Learning to communicate with deep multi-agent reinforcement learning. Advances Neural Information Processing Systems, Vol. 29.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. 2022. Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning. In Proceedings of the International Conference Machine Learning. PMLR, 6863-6877.
Scott Fujimoto, Herke van Hoof, and David Meger. 2018. Addressing Function Approximation Error in Actor-Criti Methods. In Proceedings of the 35th International Conference Machine Learning, Vol. 80. PMLR, 1582-1591.
Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. 2022. The state of sparse training in deep reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 7766-7792.
Nathaniel Grammel, Sanghyun Son, Benjamin Black, and Aakriti Agrawal. 2020. Revisiting pa- rameter sharing in multi-agent deep reinforcement learning. arXiv preprint 136255 (2020).
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto. Takashi Onishi, and Yoshimasa Tsuruoka. 2022. Dropout Q-Functions for Doubly Efficient Reinforcement Learning. The Tenth Interna- lional Conference on Learning Representations.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research 22, 241 (2021), 1-124.
Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih wei Liao. 2021. Rethinking the Implementation Tricks and Monotonicity Constraint Cooperative Multi-Agent Reinforcement Learning. (2021). arXiv:2102.03479 [cs.LG]

---

Zican Hu, Zongzhang Zhang, Huaxiong Li, Chunlin Chen, Hongyu Ding, and Zhi Wang. 2023. Attention-Guice Contrastive Role Representations for Multi-Agen Reinforcement Learning. arXiv preprint arXiv:2312.04819 (2023).
Shariq lqbal, Robby Costales, and Fei Sha. 2022. ALMA: Hierarchical learning for composite multi-agent tasks. In Advances in Neural Information Processing Systems, Vol. 35. 7155-7166
Jiechuan Jiang and Zongqing Lu. 2021. The emergence of individuality. In Proceedings of the International Conference Machine Learning. PMLR, 4992-5001.
Woojun Kim and Youngchul Sung. 2023. Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems. 1942-1950.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. 2020. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the International Conference Machine Learning. PMLR, 5544-5555.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. 2020. Maxmin Q-learning: Con-
trolling the Estimation Bias of Q-learning. In Proceedings of the 8th International Conference Learning Representations, Addis Ababa, Ethiopia, April 26-30, 2020.
Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. 2021. Celebrating diversity in shared multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 34. 3991-4002
Dapeng Li, Na Lou, Bin Zhang, Zhiwei Xu, and Guoliang Fan. 2024. Adaptive parameter sharing for multi-agent reinforcement learning. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE,
Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandkumar. 2021. Coach- player multi-agent reinforcement learning for dynamic team composition. In Proceedings of the International Conference on Machine Learning. PMLR, 6860-6870.
Shunyu Liu, Yihe Zhou, Jie Song, Tongya Zheng, Kaixuan Chen, Tongtian Zhu, Zunlei Feng, and Mingli Song. 2023. Contrastive identity-aware learning for multi-agent value decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 1195-11663..
Dor Livne and Kobi Cohen. 2020. Pops: Policy pruning and shrinking for deep reinforcement learning. IEEE Journal of Selected Topics in Signal Processing 14, (2020), 789-801.
Ryan Lowe, Yi Wu, Aviy Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent environments. In Advances in Neural Information
actor-critic for mixed cooperative-competne Processing Systems, Vol. 30.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney, 2023. Understanding plasticity in neural networks. In Proceedings of the International Conference on Machine Learning. PMLR, 23190-23211,
Gaurav Menghani. 2023. Efficient deep learning: survey on making deep learning models smaller, faster, and better. Comput. Surveys 55, 12 (2023), 1-37.
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. 2024. Deep reinforcement learning with plasticity injection. In Advances in Neural Information Processing Systems, Vol. 36.
Exgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. 2022. The primacy bias in deep reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 16828-16847.
Frans Oliehoek and Christopher Amato. 2016. concise introduction decentralized POMDPs. Springer.
12

---

Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. 2021. Bench- marking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks. In Pro- ceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. fett://arxiv... http://axxvvv..//a////. argg/bs//0000/ 0005/
Bei Peng. Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. 2021. Faemac: Factored multi-agent centralised policy gradients. In Advances in Neural Information Processing Systems, Vol. 34. 12208-12221.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. 2020. What's hidden in a randomly weighted neural network?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. 1893-11902.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research 21, (2020), 7234-7284.
Sven Seuken and Shlomo Zilberstein. 2007. Improved memory-bounded dynamic programming for decentralized POMDPs. In Proceedings of the Twenty-Thin -hird Conference Uncertuinty in Artificial Intelligence. 344-351.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. 2014. Deterministic policy gradient algorithms. In Proceedings of the International Conference on Machine Learning. Pmlr, 387-395.
Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. 2022. Dynamic Sparse Training for Deep Reinforcement Learning. In Proceedings of the Thirty- First International Joint Conference Artificial Intelligence. ijcai.org, 3437-3443.
Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason Lee. 2020. Sanity-checking pruning methods: Random tickets can win the jackpot. In Advances in Neural Information Processing Systems, Vol. 33. 11390-204011.
Gokul Swamy, Siddharth Reddy, Sergey Levine, and Anca D Dragan. 2020. Scaled autonomy: Enabling human operators to control robot fleets. In 2020 IEEE International Conference Robotics and Automation. IEEE, 5942-5948
Ming Tan. 1993. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Pro- ceedings of the 10th International Conference on Machine Learning. 330-337.
Yiqin Tan, Pihe Hu, Ling Pan, Jiatai Huang, and Longbo Huang. 2023, RLx2: Training a Sparse
Deep Reinforcement Learning Model from Scratch. In Proceedings of the Eleventh International Conference on Learning Representations.
Hang Wang, Sen Lin, and Junshan Zhang. 2021b. Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback. In Advances in Neural Information Processing Systems. 18778-247999
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. 2020. ROMA: multi-agent rein- forcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning. 9876-9886.
Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. 2021a. RODE: Learning Roles to Decompose Multi-Agent Tasks. In Proceedings of the 9th International Conference on Learning Representations.
Jiachen Yang, Igor Borovikov, and Hongyuan Zha. 2020. Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems. 1566-1574.
Mingyu Yang, Jian Zhao, Xunhan Hu, Wengang Zhou, Jiangcheng Zhu, and Houqiang Li. 2022. LDSA: Learning dynamic subtask assignment in cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 35. 1698-1710.
13

---

Wang Ying and Sang Dayong. 2005. Multi-agent framework for third party logistics in E-commerce. Expert Systems with Applications 29, 2 (2005), 431-436.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Wu. 2022. The surprising effectiveness of ppo in cooperative multi-agent games. In Advances in Neural Information Processing Systems, Vol. 33. 24611 1--224..
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S. Morcos. 2020. Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP. In Proceedings of the 8th International Conference on Learning Representations.
Lei Yuan, Chenghe Wang, Jianhao Wang, Fuxiang Zhang, Feng Chen, Cong Guan, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. 2022. Multi-Agent Concentrative Coordination with Decentralized Task Representation. In Proceedings of the Thirty-Firss International Joint Conference on Artificial Intelligence. 599-605.
Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. 2023. Survey of Progress on Coopera- tive Multi-agent Reinforcement Learning in Open Environment. arXiv preprint arXiv:2312.01058 (2023).
Xianghua Zeng, Hao Peng, and Angsheng Li. 2023. Effective and stable role-based multi-agent collaboration by structural information principles. In Proceedings of the AAAI Conference Artificial Intelligence, Vol. 37. 11772-11780.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. 2023. A survey of large language models. arXiv preprint arXiv:2303.1822 (2023).
Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, and Yaodong Yang. 2024. Heterogeneous- agent reinforcement learning. Journal of Machine Learning Research 25 (2024), 1-67.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, et al. 2021. Smarts: An open-source scalable multi-agent rl training school for autonomous driving. In Proceedings of the Conference Robot Learning. PMLR, 264-285.
14

---

A Experimental details
A.1 Implementation details
A.1.1 Kaleidoscope with MATD3
Critic ensembles When incorporating Kaleidoscope into the MATD3 algorithm, the overall training loss for the critic ensembles becomes:
( 8*)
Σ j=l,..,K K
(o,))
Σ j=1...,
Loldo, 87) a Je (8"),
(15)
with Lo( 89) being the original MATD3 loss given in Equation
Telrv (89) being the diversity
regularization given in Equation 14p. and a being a coefficient balancing the original MARL
objective and the proposed diversity regularization. Note that although J.liv (s²) contains parameters Φu. we stop the gradients for de in (s)).
In the implementation, we apply layer-wise weights to the diversity regularization term Jehh (s*).
which is defined as
Joth (5
E
W
Σ Σ
1 @ (M,, M2,,
(16)
where / denotes the layer index of the neuron networks, I. represents the total number of layers, Mú, is the mask for agent at layer I, and the layer-wise weights are set as un = 2'. The intuition behind this choice is that features closer the output tend be more compact (Kusupati et al. 2020); consequently, assigning larger regularization weights to these layers may have more significant impact on the output action decisions. Our initial experiments empirically demonstrate that setting 1. Based on these findings, we
wi = 2 improves performance compared to the case where maintain this design choice throughout all our experiments.
In practice, we adaptively adjust a while maintaining a constant ratio between the MATD3 loss and the diversity loss, which treated as a hyperparameter:
Σ Σj=t... K Colov, m9)
IJ
ar
(17)
where a is hyperparameter, and the gradients for E= & fc( (3
are stopped.
Actors For the actors, the training objective is to maximize the following term
gلll (Oo,
gilll (0.)
Σ
J(0, 8:) + + (d))
(18)
where [Oo, 8:) is the original actor objective defined in Equation 3). Jdi ) is the diversity regu-
larization given in Equation 8) with layer-wise weights as in Equation (6), 3dd is the regularization coefficient. The value of g is determined by
gd EE [(p. s:)
3,
(19)
[dv
where 3 is a constant hyperparameter, similar the approach used in Equation 17 for the critic ensembles.
A.1.2 Kaleidoscope with QMIX
When incorporating Kaleidoscope into the QMIX algorithm (Rashid et al., 2020), we apply Kalei- doscope parameter sharing only to the local Q networks. Consequently, the training loss is defined
Call (fu. 8) = L(Bu. 8) - gd. Jdiy
(20)
where
L(Ou, s) =
o0t1)// (v - Qioll' o' 0'; s))
(21)
with for =
r maxa Quor 10t11,000000 a, ) and 0- representing the parameters of a target
network as in DQN.
15

---

A.1.3
Network architecture and hyperparameters
Codebase codebase:
Our implementation of Kaleidoscope and baseline algorithms are based on the following
* HARL
Zhong et al. 2024) (MATD3 implementation): https:// Bwwp:/hhuu..cooPP--MMMM..
HARL]
EPyMARL (Papoudakis al.
2021) (QMIX implementation for MPE):
https://github. woe/ commm///ooeaaagttt//////aarr
PyMARL2 et 2021) (QMIX (0mi for SMACv2):
thub. com www.///eeeeee////eeeeeeeeee
SePS (Christianos et al. 2021 https//ggtabb..ommmm-sssetttssts
The code for Kaleidoscope is publicly available httpp////ggttttoooo//////aaaaaaa
Network architecture In line with prior works (Zhong et al., 2024 Papoudakis et al. 2021
et al. 2021), we employ deep neural networks consisting of multilayer perceptrons (MLPs) with rectified linear unit (ReLU) activation functions and gated recurrent units (GRUs) to parameterize the actor and critic networks. Moreover, when the masking technique is applied to critic ensembles, incorporate layer normalization between the MI. layers and ReL.U activations (Hiraoka et al. (202)). In Kaleidoscope, the masking technique is applied to the MLP layers, and the resetting mechanisms described Sections 3.3 and 3.4are are applied to the last three layers the respective neural networks, following Nikishin al. 2022).
To ensure a fair comparison, we implement our method and all the baselines using the same codebase with the same set of hyperparameters, with the exception of method- specific ones. The common hyperparameters are listed in Tables 3 The Kaleidoscope-specifie hyperparameters provided in Table
Hyperparameters
Table Common hyperparameters used for MATD3 in the MaMuJoCo domain.
Hyperparameter
Value
Number of layers
Hidden sizes
256
Discount factor y
0.99
Rollout threads
10
Critic Ir
1 1 10-3 3
Actor Ir
5 10-4
Exploration noise
0.1 1000
Batch size
Replay buffer size
1 x 106
Number environment steps
10 x 10a
n_step'
(5, 10, 20)
Here we adopt the per-scenario finetuned
value for this hyperparameter as provided by HARL.
A.2 Environmental details
Codebase
The environments used in this work are listed below with descriptions in Table
MaMuJoCo (Peng et al. 202 https://github.com/
mujoco
MPE (owe et al.
2017
Papoudakis et al. 2021):
https://github..co//eettttta
multiagent- catpeeett-ppaiicleeeeee envs
SMACv2 EElis et al.
2024
attps://githur http:/htubb.omm/o//il/////a/
16

---

Table 4: Common hyperparameters used for QMIX in the MPE domain.
Hyperparameter
Value
Number of layers
Hidden sizes
64
Discount factor y
0.99
Lr
5 x 10
Initial
1.0
Final
0.05
Batch size
32
Replay buffer size
5000
Number of environment steps Double
3 x 106
True
Table 5: Common hyperparameters used for QMIX in the SMACv2 domain.
Hyperparameter
Value
Number of layers
Hidden sizes
64
Discount factor
0.99
Lr Initial
x 10-3
1.0
Final e
0.05
Batch size
128
Replay buffer size
5000
Number of environment steps
10 106
Double Q
False
MPE We extend the scenario settings provided in the original codebase to increase the complexity and challenge of the tasks. In World, we set the number of predators (agents) to 4, the number of prey to 2, the number of obstacles to and the number of forests for the predators to approach the prey while avoiding collisions with obstacles. The prey is attracted to the food and can hide from the predators the forests. In Push, we set the number of agents to the number of adversaries to 2, and the number of landmarks to 2. The goal of the game is for the agents to push the adversaries away from the landmarks. both scenarios, we pretrain the adversary (prey) policies using the MADDPG algorithm Lowe et al. (2017) and use these pretrained policies to test the performance of different algorithms
The objective of the game is
A.3 FLOPs calculation
To calculate the number of floating-poin ppinn operations (FLOPs) for a single forward pass of a sparse model, we sum the total number of multiplications and additions layer by layer, following the approach in Evci et al. (2020). For fully-connected layer, the FLOPs are computed as follows:
FLOPs (1 Sparsity) x In Out.
(22)
For GRU cell, the FL.OPs are computed as:
FLOPs 2 x (3 Hidden + 3 x In x Hidden + 13 x Hidden).
(23)
A.4 Experimental Infrastructure
The experiments on the SMACv2 benchmark were conducted using NVIDIA GeForce RTX 3090
GPUs, while the experiments on other benchmarks were performed using NVIDIA GeForce RTX 3080 GPUs. Each experimental run required less than 2 days to complete.
17

---

Table Hyperparameters used for Kaleidoscope.
Hyperparameter
Environment
Value
MaMuJoCo
0.1
Actor diversity coefficient B
MPE
0.5
SMACv2
5.0
MaMuloCo
0.5
Actors reset probability p
MPE
0.1
SMACv2
0.2
MaMuloCo 106
Actor reset interval
MPE
200 x 103
SMACv2
10
Number of critic ensembles K Critic ensembles diversity coeflicient Critic reset interval
MaMuJoCo
5
MaMuJoCo
0.1
MaMuJoCo
800 x 10a
Table 7: Environments details.
Environment
Action space
Agent types
Scenarios
Number agents
Ant-v2-4x2
Hopper-v2-3xl
3
Walker2D-v2-2x3
2
MaMuJoCo
Continuous
Hetcrogeneous, fixed
Walker2D-v2-6xl
6
HalfCheetah-v22223
2
Swimmer-v2-10x2
10
World
MPE
Discrete
Homogeneous, fixed
Push
5
Terran_5_vs_2 Protoss_5 5_ v__
SMACv2
Discrete
Heterogeneous, dynamic
5
Zerg_5 5 vs_5
5
B
More results and discussion
B.1
Detailed Costs
provide per-scenario FLOPs across different methods in Table 8 as supplement for Table
Table 8: Averaged FLOPs for different methods. Results normalized w.r.t. the FuPS ID model.
The lowest costs are highlighted in bold.
Scenarios
NoPS
FuPS
FuPS +ID
ScPS
MultiH
SNP
Kaleidoscope
World
1.0x
0.993x
1.0x
1.0x
1.0x
0.988x
0.897x
Push
1.0x
0.991x
1.0x
1.0x
1.0x
0.988x
0.904x
Ant-v2-4x2
1,0x
0.990x
1.0x
1.0x
1.0x
0.900x
0.640x
Hopper-v2-3x
1.0x
0.989x
1.0x
1.0x
1.0x
0.900x
0.721x
Walker2D-v2-2x3
1.0x
0.992x
1.0x
1.0x
1.0x
0.900x
0.731x
Walker2D-v2-6x1
1.0x
0,979x
1.0x
1.0x
1.0x
0.900x
0.763x
HalfCheetah-v222233
1.0x
0,993x
1.0x
1.0x
1.0x
0.900x
0.614x
Swimmer-v2-10x2
1.0x
0.968x
1.0x
1.0x
1.0x
0.900x
0.61
Terran_5 5_
1.0x
0,992x
1.0x
1.0x
1.0x
0.988x
0.890x
Protoss v__
1.0x
0.992x
1.0x
1.0x
1.0x
0.988x
0.895x
Zerg 5 v
1.0x
0.992x
1.0x
1.0x
1.0x
0.988x
0.885x
18

---

8000
8000
Test Return
6000
Test Return
6000
4000
a= 10
4000
B=10
a= 1
B=1
2000
a 0.1
2000
B=0.1
0.01
=
8.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00 1e7
Timesteps 1e7
Timesteps
(a) Critic ensembles diversity coefficient
(b) Actors diversify coefficient 3.
Figure 9: Hyperparameter analysis.
B.2 Hyperparameter Analysis
We conduct further analysis on the hyperparameters a and 88 and present the results in Figure 9 The hyperparameter controls the variance the critic ensembles. As shown in Figure 9a we observe that an excessively small results in degraded performance because it reduces the critic ensembles to a single critic network, causing value estimation to suffer from severe overestimation. Conversely, an excessively large also deteriorates performance, possibly due to increased estimation bias. For 3, as illustrated in Figure 9b] an overly small /3 leads to degraded performance because reduces the Kaleidoscope parameter sharing to full parameter sharing, confining the policies to be identical. An overly large also negatively impacts performance, as it may cause the training objective to deviate too much from minimizing the original MARL loss.
In general, we recommend setting both hyperparameters between 0.1 and 1. However, the optimal hyperparameter values may vary across different scenarios. For fair comparisons, we maintain the same set of hyperparameters across all scenarios our experiments. Nevertheless, further tuning of these hyperparameters has the potential to enhance performance.
B.3 Further Visualization Results
To better understand how learnable masks in Kaleidoscope affect the performance through policies, we visualize the pairwise mask differences among agents and the agent trajectories at different training stages in Figure I0 As training progresses, the test return increases and diversity loss decreases, indicating better performance and greater diversity among agent policies. Correspondingly. mask differences among agents increase, and the agent trajectory distribution becomes more diverse.
B.4 Limitations
Here we discuss some limitations of Kaleidoscope.
First, suggested by results in Appendix B. the optimal hyperparameters vary from scenario to scenario. Therefore, using the same hyperparameters across all scenarios may not yield the best performance for Kaleidoscope. Developing automatic scheme that utilizes environmental information to determine these hyperparameters would be beneficial.
Second, as the environments used in this work contain no more than 10 agents, we assign distinct mask for each agent. However, when the problem scales hundreds of agents, this vanilla implemen- tation may fail. In such cases, possible approach is to cluster N agents into K (K N) groups and train K masks with Kaleidoscope. This would reduce computational costs and achieve better trade-off between sample efficiency and diversity. Within the same group, agents share all parameters, while agents from different groups share only partial parameters. Techniques for clustering agents based on experience, proposed by CCristianos et al. (2021), could be useful.
19

---

0.000
Test Return
200
Test Return
-0.005
-0.010
Timesteps
00
T= IM
T 2M
T- 3M
Pairwise Mask Differences
Agent
Agent 2
Agent 3
Agem- 4
Agent Trajectories t-SNE
Figure 10: Further visualization on World.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's contributions by proposing a novel parameter sharing technique in MARL (Section B) and claiming its superior performance compared existing methods, which supported by the experimental results (Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
20

---

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix B.4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't tt acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
21

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The method and implementation details are provided in Section 3 and Ap- pendix A.I] and the experiment settings and environment details are described in Section and Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be a way access this model for reproducing the results a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soures source models, may be that access the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code is available httt:p//ggttubb.ooo//x/xx///ssss
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
22

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
Code//obeeeeeeeieeeeeee ss www.nsssiisysyinoiioo for more details. http://uuu..eee/oooouuubbssooooo
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiment settings are provided in Section with details elaborated in Appendices/A.11 A.] and A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The main results are reported with 95% confidence error bars in Figures 4 to
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap. etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK report 1-sigma error bars, but should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were caleulated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
23

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Please see Appendix A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::///aarrrr....oppppp////haa
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make tion due to laws or regulations their jurisdiction).
preserve anonymity (e.g., if there is special considera-
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This paper presents work whose goal advance the field of Reinforcement Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended
(e.g., disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The experiments in this work conducted in simulated game environments, thereby presenting minimal risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The papers corresponding to the environments used are cited in Section and the codebases utilized are listed in Appendices A.1.3 and A.2
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, soperswithcoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
25

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "lb7vuGhxTC", "reviewer": "Reviewer_WQkZ", "summary": "This paper introduces Kaleidoscope, an adaptive partial parameter sharing method for multi-agent reinforcement learning (MARL). Kaleidoscope aims to balance the benefits of full parameter sharing (sample efficiency) with the flexibility of non-parameter sharing (policy diversity). It achieves this by using learnable binary masks to control which parameters are shared between agent networks and within critic ensembles. The authors demonstrate Kaleidoscope's effectiveness on various MARL benchmarks, showcasing superior performance compared to existing parameter sharing approaches.", "questions": "How does the computational overhead of Kaleidoscope scale with the number of agents?\n\nImpact of Mask Sparsity: Does the sparsity level of the masks induced by STR vary across different tasks and environments? Are there any insights into how the sparsity level affects the policy diversity and overall performance?\n\nIntegration with other MARL Algorithms: The paper demonstrates Kaleidoscope with MATD3 and QMIX. How straightforward would it be to adapt the method to other MARL algorithms such MAPPO? Are there any specific challenges or considerations?\n\nReal-world Applications: Can you envision any specific real-world applications where Kaleidoscope's ability to promote agent heterogeneity would be particularly beneficial?", "limitations": "The authors have properly addressed the limitations of their paper.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "*Originality*: The application of STR for dynamic partial parameter sharing in MARL is novel and well-motivated. The paper clearly distinguishes itself from prior work on partial parameter sharing that relies on fixed initializations.\n\n*Quality*:  The proposed method is technically sound. The combination of STR with diversity regularization and periodic resetting is well-reasoned and contributes to the effectiveness of Kaleidoscope. Experimental results across diverse MARL benchmarks provide strong support for the claims. The ablation is also helpful to understand the contribution each component of Kaleidoscope has on final performance.\n\n*Clarity*: The paper is generally well-written and organized. The core idea of using learnable masks is clearly conveyed through figures and explanations. The technical details are adequately provided, allowing for reproducibility.\n\n*Significance*: Addressing the trade-off between sample efficiency and policy diversity is a crucial challenge in MARL. Kaleidoscope's adaptive and dynamic nature makes it a potentially valuable contribution to the field. The improved performance on diverse benchmarks suggests that the proposed method can be broadly applicable.", "weaknesses": "The paper primarily focuses on environments with a relatively small number of agents. The scalability of Kaleidoscope to environments with  tens or hundreds of agents. In that case it is not clear if a large number of masks could have a negative performance compared with the parameter sharing baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. Regarding you questions and suggestions, we would like to provide clarifications and additional results below. If you have any follow-up questions or comments, please let us know and we will be happy to have further discussions.\n\n**Q1:** The paper primarily focuses on environments with a relatively small number of agents. The scalability of Kaleidoscope to environments with tens or hundreds of agents. In that case it is not clear if a large number of masks could have a negative performance compared with the parameter sharing baseline.\n\n**A1:**  \n\n- The benchmark environments we use (MPE, MaMuJoCo, SMACv2) are widely adopted and challenging in MARL. They cover various characteristics (as specified in Table 7 in Appendix A.2): discrete/continuous action spaces, heterogeneous/homogeneous agent types, and varying numbers of agents. This diversity makes them suitable for testing MARL algorithms.\n- We agree that scalability is a long-standing challenge in MARL. We acknowledge that scalability to hundreds of agents is a potential limitation of our approach, which we have discussed in Appendix B.3.\n    - **Protential solution**: To scale Kaleidoscope to hundreds of agents, a possible approach is to cluster $N$ agents into $K$ ($K \\lt N$) groups and train $K$ masks with Kaleidoscope. This would reduce computational costs and achieve a better trade-off between sample efficiency and diversity. Within the same group, agents share all parameters, while agents from different groups share only partial parameters. Techniques for clustering agents based on experience, as proposed in [1], could be useful. However, to truly scale current MARL algorithms to hundreds of agents, besides better parameter sharing schemes, we also need consider problems such as severe partiall observation, effective credit assignment and complex coordination among agents, which we will leave for future work.\n\n**Q2:** How does the computational overhead of Kaleidoscope scale with the number of agents?\n\n**A2:** In theory, the computational overhead of Kaleidoscope and all baselines scales linearly with the number of agents. However, thanks to PyTorch's parallel implementation framework, the actual training and execution time scales sublinearly, unless memory becomes a bottleneck.\n\n**Q3:** Impact of Mask Sparsity: Does the sparsity level of the masks induced by STR vary across different tasks and environments? Are there any insights into how the sparsity level affects the policy diversity and overall performance?\n\n**A3:** Generally, the final sparsity levels of the masks are similar across different tasks. Below are the results, followed by a discussion. Since overall sparsity comparison can be difficult to interpret across environments (affected by network structure, RNN layers, etc.), we report the average sparsity of the fully connected layers where Kaleidoscope is applied\n\n| Scenarios          | Final Sparsity    |\n|--------------------|-------------------|\n| World              | 0.329             |\n| Push               | 0.358             |\n| Ant                | 0.340             |\n| Hopperm            | 0.191             |\n| Walker2D           | 0.211             |\n| HalfCheetah        | 0.350             |\n| Swimmer-v2         | 0.337             |\n| Terran             | 0.248             |\n| Protoss            | 0.226             |\n| Zerg               | 0.321             |\n\n\nKaleidoscope tends to induce higher sparsity levels with a larger number of agents. However, overall sparsity across different environments does not vary significantly, generally converging between 0.2 and 0.3. This is because an overly sparse network may lack sufficient representational capacity for complex policies, especially if the original network is small, while networks that are not sparse enough may fail to induce diversity.\n\n**Q4:** Integration with other MARL Algorithms: The paper demonstrates Kaleidoscope with MATD3 and QMIX. How straightforward would it be to adapt the method to other MARL algorithms such MAPPO? Are there any specific challenges or considerations?\n\n**A4:**  It should be fairly straightforward to adapt Kaleidoscope to other MARL algorithms because it mainly operates at a network level rather than an algorithm level. We used MATD3 and QMIX to illustrate Kaleidoscope because they represent the actor-critic and value-based families, respectively. While incorporating Kaleidoscope into different families of algorithms may vary slightly, implementation within the same family should be similar. To incorporate Kaleidoscope into MAPPO (another actor-critic algorithm), one should follow the Kaleidoscope-MATD3 implementation.\n\n**Q5:** Real-world Applications: Can you envision any specific real-world applications where Kaleidoscope's ability to promote agent heterogeneity would be particularly beneficial?\n\n**A5:** Kaleidoscope would be most useful in scenarios where agents need to share some aspects of their policies but not end up with identical policies. For example, two dexterous hands cutting a steak: one hand uses a fork to hold the steak, while the other uses a knife to cut it. Here, the agents should share the grasping skill but not compete for the same utensil and cause clashes. Since it's challenging to know the exact level of heterogeneity needed for a specific multi-agent task beforehand, a flexible framework like Kaleidoscope can be beneficial.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XFC0vt1QZt", "reviewer": "Reviewer_Y5Eh", "summary": "The paper introduces a novel adaptive partial parameter sharing scheme in multi-agent reinforcement learning (MARL) to enhance policy heterogeneity while maintaining high sample efficiency. The key innovation, Kaleidoscope, employs a set of common parameters and multiple sets of distinct, learnable masks for different agents, encouraging policy diversity without sacrificing the benefits of parameter sharing. This approach dynamically balances the trade-off between full parameter sharing and non-parameter sharing, thereby bridging the gap between the two. Furthermore, Kaleidoscope is extended to critic ensembles within actor-critic algorithms to improve value estimations. Empirical evaluations across various environments, including multi-agent particle environment, multi-agent MuJoCo, and StarCraft multi-agent challenge v2, demonstrate that Kaleidoscope outperforms existing parameter sharing methods.", "questions": "1. What is the purpose of citing LLM works in lines 35 and 36?\n2. Are there any other works discussing parameter sharing apart from references [12,13,14] in this paper?\n3. How to distinguish `Neurons` and `Weights` in Table 1?\n4. In the experiments, are algorithms using similar sizes of networks? For a fair comparison, the network sizes for agent policies and critics should be similar.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper investigates the limits of parameter sharing in MARL and conducts conprehensive experiments.", "weaknesses": "### Methodology\n1. The proposed method is similar to SNP, and the difference is not fully discussed.\n2. The learned masks can not be explained. The proposed method can be regarded as enlarging the network into a Mix-of-Expert network with a dense router, which is expected to improve the performance. However, the proposed method does not leverage the relationships among agents.\n3. Pseudocode is lacking. The proposed method has several components, it is better to include a pseudocode in the main paper.\n4. The proposed method maintains N masks whose sizes are similar to those of the actor and critic networks, which may result in low training efficiency.\n\n### Experiments\n1. The authors should consider implementing a simple and intuitive partial parameter sharing network as a baseline, such as a shared trunk with multiple action heads.\n2. Apart from Table 2, the authors should compare the training costs among those algorithms.\n3. The ablation study is incomplete. The main new idea behind the proposed method is to share parameters using learnable masks. Because of this, the authors should show how this main new idea improves performance instead of getting rid of secondary methods in Figure 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. Here are our clarifications. If you have any follow-up questions or comments, please let us know and we will be happy to have further discussions.\n\n**Q1:** The proposed method is similar to SNP...\n\n**A1:** As listed in Table 1, our proposed method differs from SNP in the following key aspects:\n- Kaleidoscope learns masks that indicate which parameters to share during training, whereas SNP relies on fixed mask initialization. We then further design diversity regularization and resetting mechanism to facilitate the mask learning with the objective to boost the task performance.\n- Kaleidoscope applies masks to weights, while SNP applies masks to neurons (activation outputs).\n\nThese design choices give **Kaleidoscope greater flexibility** (elaborated below) in parameter sharing and result in better performance (supported by Fig. 4-5).\n- The learnable nature of masks provides adaptive PS during training.\n- The weight-level sharing allows for partial sharing that is more precise (supported by Fig. 9 in the response PDF).\n\n**Q2:** The learned masks can not be explained. The proposed method can be regarded as enlarging the network into a Mix-of-Expert network with a dense router, ... does not leverage the relationships among agents.\n\n**A2:** In Fig. 7 of the original manuscript, we visualize mask differences among agents throughout the training process, reflecting the pairwise similarity between agent policies. Initially, agents share all parameters and gradually learn diverse policies through different masks. During the rebuttal period, we further visualized agent trajectories as training proceeds, with results provided in Fig. 7 of the response PDF.\n\n**Q3:** Pseudocode is lacking...\n\n**A3:** We appreciate the suggestion and will revise the manuscript accordingly. Kaleidoscope generally requires an extra resetting step, as shown in Fig. 3 of the original manuscript. It follows the same end-to-end training procedure as the base algorithms (MATD3 or QMIX), with modified network structures and loss functions. \n\n**Q4:** The proposed method maintains N masks whose sizes are similar to those of the actor and critic networks, which may result in low training efficiency.\n\n**A4:** Please see A6.\n\n**Q5:** ... implementing ... a shared trunk with multiple action heads.\n\n**A5:** We follow the suggestion and implement this baseline as MultiH during the rebuttal period. The results are reported as purple curves in Fig. 3-4 in the response PDF. Overall, the performance of MultiH lies between FuPS and NoPS. Like other non-adaptive partial parameter sharing algorithms, MultiH finds a fixed middle point between full parameter sharing and no parameter sharing. While this particular middle point may be favored in certain scenarios, its performance across different scenarios is not stable due to the lack of flexibility. This observation motivates us to develop an adaptive partial parameter sharing algorithm such as Kaleidoscope.\n\n**Q6:** Apart from Table 2, the authors should compare the training costs among those algorithms.\n\n**A6:** \n- **Cost results:** We summarize the training and execution costs of all methods in Table 2,9,10 in global rebuttal text. **Overall, Kaleidoscope enjoys a higher performance with lower execution complexity but does introduce some extra training overhead.** The training overhead is comparable with some baselines such as SePS. \n- **Discussion:** Kaleidoscope sacrifices training computational cost for the following benefits:\n    - During training, Kaleidoscope achieves better policy diversity while maintaining high sample efficiency.\n    - During execution, Kaleidoscope enjoys better performance with lower overhead.\n\n    **We believe this tradeoff is worthwhile** because:\n    - In a learning task, training is conducted once. After deployment, execution performance and efficiency are what matter.\n    - In an RL setting, sample efficiency is more crucial than computational efficiency because online data in the real world is usually difficult to collect.\n    - Training computational cost can potentially be reduced by employing strategies such as mixed precision training.\n\n**Q7:** The ablation study is incomplete. The main new idea ... share parameters using learnable masks...\n\n**A7:** We appreciate the suggestion and have added an ablation with fixed masks rather than learnable masks in Fig. 6 of the response PDF. This new ablation underperforms Kaleidoscope with the same sparsity level. It showcases the need to use learnable masks to dynamically adjust the parameter sharing throughout the training (as supported by Fig. 7 in the original manuscript).\n\n**Q8:** What is the purpose of citing LLM works in lines 35 and 36?\n\n**A8:** To support the current trend of scaling up model sizes and motivate the need for an effective partial parameter-sharing technique. Although our method focuses on MARL, we believe parameter sharing is potentially a topic of interest in other multi-agent systems. We will polish the wording to make this clearer.\n\n**Q9:** Are there any other works discussing parameter sharing apart from references [12,13,14] in this paper?\n\n**A9:** The references in our manuscript are good representatives of recent progress on parameter sharing in MARL. However, as reviewer BN4p suggested, we will add another concurrent work to the related work section: AdaPS [2], which combines SNP and SePS by proposing a cluster-based partial parameter-sharing scheme.\n\n**Q10:** How to distinguish Neurons and Weights in Table 1?\n\n**A10:** Neurons refer to the output of the activation function, while weights refer to the network parameters. In a fully-connected layer, removing a neuron is equivalent to removing all the weights associated with that neuron.\n\n**Q11:** ... are algorithms using similar sizes of networks?...\n\n**A11:** Yes, we use the same network structure across different algorithms, with details listed in Appendix A.1.3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KWvTypmpz7", "reviewer": "Reviewer_BN4p", "summary": "This paper presents an approach to multi-agent reinforcement learning.  In this approach, the agent network has an overall set of parameters.  These parameters are transformed by agent-specific masks.  The masks are learnable.  Agent policies are encouraged to be diverse through a diversity-regularisation term.  Masks and diversity regularisations are also applied to the critic network parameters.  Experimental results compare the performance of Kaleidoscope with other methods.", "questions": "It would be nice if the authors are able to address the points in the Weaknesses section.", "limitations": "No.  The current limitations of the work are presented in an Appendix.  It would be good to see this fleshed out with more details and presented as part of the main text.  For example, it would be interesting if the authors provide a guideline for how their method would scale with the number of agents.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The introduction of masking as part of the learnable parameters for MARL agents is interesting.  It removes a previous limitation of parameter sharing, and allows this to be flexible.  The contributions are made clear by the authors. Figure 2 illustrates the concept of masking well, which is important to the core of the work.  The manuscript is generally easy to read, barring some minor comments pointed out in the Questions section.", "weaknesses": "- The current limitations of the work are presented in an Appendix.  It would be good to see this fleshed out with more details and presented as part of the main text.  For example, it would be interesting if the authors provide a guideline for how their method would scale with the number of agents. \n- While the presentation of the material is clear, it is repetitive in places.  It would be nice to make the text more concise and provide some more interpretation of the results. \n- In general, the Figure captions can be made more self-contained. \n- Minor comments:\n   * In Section 2, after line 78, it appears that the meaning of $O$ is not explained in the MDP $\\mathcal{M}$. \n   * Line 89 appears to have either grammatical errors or a confusing choice of words between singular and plural. \n   * Line 277 has a typo in MARL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. Regarding your questions and suggestions, we would like to provide clarifications below. If you have any follow-up questions or comments, please let us know, and we will be happy to discuss further.\n\n**Q1:** The current limitations of the work are presented in an Appendix. It would be good to ... as part of the main text. For example, it would be interesting if the authors provide a guideline for how their method would scale with the number of agents.\n\n**A1:** We appreciate the suggestion and will elaborate on how to protentially scale Kaleidoscope to more agents. Instead of assigning one unique mask to each agent, we can consider clustering $N$ agents into $K$ ($K \\lt N$) groups and train $K$ masks with Kaleidoscope. This would reduce computational costs and achieve a better trade-off between sample efficiency and diversity. Within the same group, agents share all parameters, while agents from different groups share only partial parameters. Techniques for clustering agents based on experience, as proposed in [1], could be useful. \n\n**Q2:** While the presentation of the material is clear, it is repetitive in places. It would be nice to make the text more concise and provide some more interpretation of the results.\n\n**A2:** We appreciate the suggestion and provide further visualization of agent trajectories in Fig. 7 in the response PDF. We visualize the pairwise mask differences among agents and the agent trajectories at different training stages. As training progresses, the test return increases and diversity loss decreases, indicating better performance and greater diversity among agent policies. Correspondingly, mask differences among agents increase, and the agent trajectory distribution becomes more diverse. This reflects:\n\n- The masks become more diverse as the training proceeds.\n- The more diverse masks results in more diverse policies (reflected by the trajectories).\n- The diverse policies lead to better performance.\n\n**Q3:** More self-contained Figure captions and typos.\n\n**A3:** Thank you for catching those, and we will revise them accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5fZfFdRb9a", "reviewer": "Reviewer_CAY2", "summary": "The authors propose a method for adaptive parameter sharing in Multi-Agent Reinforcement Learning (MARL), by using learnable weight masks for each agent. They combine this with a regularization method to encourage diversity in the masks and a resetting mechanism to reuse masked parameters with a certain probability. They showed that their method can outperform parameter sharing baselines on a set of MARL benchmarks.", "questions": "1. Is there a motivation for using STR over other methods that learn different activations per agent, like SNP?\n2. Can you provide more details on the evaluation procedure, such as the evaluation interval and how many evaluation steps are used?\n3. Was there a reason for not including FuPS+ID as a baseline in the experiments?", "limitations": "The authors address the limitations in the appendix.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow. Their methodology and context are explained well. Furthermore, the authors provide a clear motivation for their work, to allow heterogeneous behaviour in MARL, while still leveraging the sample efficiency of parameter sharing, which is an important problem in MARL. \n- Applying soft threshold reparameterization (STR) [1] in MARL appears to be novel. \n- The authors provide a range of experiments on common MARL benchmarks. They also provide a detailed ablation study to show the importance of the different components of their method and measure the computation cost (FLOPs) of their method during test time.  They also provide a hyperparameter analysis in the appendix, which is useful.\n\n1] Kusupati, A., Ramanujan, V., Somani, R., Wortsman, M., Jain, P., Kakade, S., & Farhadi, A. (2020, November). Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning (pp. 5544-5555). PMLR.", "weaknesses": "- The authors do not include FuPS+ID (full parameter sharing, conditioning on one-hot encoded agent ids) as a baseline in their experiments. This was used as a baseline in prior work [1,2] and is the most common parameter sharing implementation in MARL, therefore it would be a useful comparison to have. \n- The authors appear to have missed a relevant related work [3] that also uses pruning masks for adaptive parameter sharing in MARL. This should be included in the related work section and discussed in the context of the proposed method.\n- The motivation for using a method like STR, which learns different use different weights for each agent, versus a method that uses different activations per agent (like SNP [2]) is not clear to me. I understand that STR learns the pruning schedule itself and this is useful, but it is not clear that learning weights is the best way. A discussion on this might be relevant if future works build on these methods. \n- From the experimental results, it is not clear that the proposed method is significantly better than the baselines. A lot of the results of the proposed method are within 95% confidence intervals of other methods (and vice versa). This makes it hard to say that the proposed method is significantly better. I recommend using the Interquartile Mean (IQM) as suggested by [4], with at least 5 seeds (as you have done for all environments except SMAC V2) to help clarify if the proposed method is better in a statically significant way. Furthermore, I am unable to find the details of the evaluation procedure in the paper, such as the evaluation interval and how many evaluation steps are used. This should be included in the paper.\n- An analysis of the computation cost (speed and memory) during training time might be relevant. Parameter sharing generally reduces the number of parameters required, but with learnable masks, this might not be the case. This should be discussed in the paper.\n\n1] Christianos, F., Papoudakis, G., Rahman, M. A., & Albrecht, S. V. (2021, July). Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning (pp. 1989-1998). PMLR.\n\n2] Kim, W., & Sung, Y. (2023). Parameter sharing with network pruning for scalable multi-agent deep reinforcement learning. arXiv preprint arXiv:2303.00912.\n\n3] Li, D., Lou, N., Zhang, B., Xu, Z., & Fan, G. (2024, April). Adaptive parameter sharing for multi-agent reinforcement learning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6035-6039). IEEE.\n\n4] Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., & Bellemare, M. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, 29304-29320.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We appreciate your questions and suggestions and have provided clarifications below. Please let us know if you have any follow-up questions or comments; we would be happy to discuss further.\n\n**Q1:** The authors do not include FuPS+ID ... a baseline in prior work [1,2] and is the most common ...\n\n**A1:** We apologize for the misunderstanding. **The FuPS in the original manuscript in fact refers to FuPS+ID, as the reviewer mentioned.** We agree that adding agent IDs is the default PS implementation in MARL, and therefore we did not explicitly mention it. We will rename it as FuPS+ID for clarity.\n\n**Q2:** .. missed a relevant related work [2] that also uses pruning masks for adaptive parameter sharing in MARL...\n\n**A2:** We appreciate the suggestion and will incorperate discussion regarding this concurrent work into the related work section: AdaPS [2] combines SNP and SePS by proposing a cluster-based partial parameter sharing scheme. \\\nIt is also worthnoting that in our manuscript, \"adaptive\" refers to whether the sharing scheme evolves during training (see Table 1 in our manuscript). Although more flexible than SePS, AdaPS is not adaptive in this sense because its masks are initialized at the beginning of training and kept fixed throughout the training. Without the learnable masks, AdaPS cannot adaptively adjust the parameter sharing scheme during the training to boost the performance. Its performance highly relies on good initialization. A fixed partial parameter sharing scheme in general can not exhance the joint network representational capacity as our proposed Kaleidoscope does (see analysis in line 136-143 in our manuscript). \n\n\n**Q3:** The motivation for using ... different weights for each agent, versus ... different activations ... is not clear to me. I understand that STR learns the pruning schedule itself and this is useful, but it is not clear that learning weights is the best way...\n\n**A3:** We appreciate this insightful question. In fact, we also considered this when designing our method. During the rebuttal period, we conducted an experiment to justify this design choice, with results provided in Fig. 9 in the response PDF. As the results indicate, masking weights is more effective than masking activations. The rationale is that masking weights offers better flexibility. In a fully-connected layer, masking an activation is equivalent to masking all the weights connected to that activation. In Kaleidoscope, masks are learnable. In mask learning, better flexibility corresponds to more parameters to optimize, increasing the chance of finding optimal masks. \\\nWe agree this is an important design choice to justify and will incorporate these new results and discussions into our manuscript.\n\nTo sum up, comparing with baselines, **our Kaleidoscope enjoys better flexibility at two levels**:\n- The learnable nature of masks provides adaptive PS during training (as you already mentioned).\n- The weight-level sharing allows for partial sharing that is more precise (as verified by this additional experiment).\n\n\n**Q4:** ... not clear that the proposed method is significantly better than the baselines... I recommend using IQM ... to help clarify if the proposed method is better in a statically significant way ... details of the evaluation procedure ...\n\n**A4:** Following your suggestion, we now report IQM with 80% CI in the response PDF. For more seeds in SMAC v2, we are currently running those, but the results are not fully ready due to the limited time of the rebuttal period. Regarding the evaluation procedure, we follow the standard configurations provided by the corresponding codebase, listed below for your reference:\n\n| Environment   | Evaluation Interval | Evaluation Steps/Episodes |\n|-----------|---------------------|------------------|\n| MPE       | 50000 steps         | 100 episodes|\n| MaMuJoCo  | 10000  steps        | 40 episodes|\n| SMACv2    | 10000  steps        | 32 episodes|\n\n\n**Q5:** An analysis of the computation cost (speed and memory) during training time might be relevant...\n\n**A5:** \n\n- **Cost results:** We follow your suggestion and summarize the training and execution costs of all methods in Table 2,9,10 in global rebuttal text. **Overall, Kaleidoscope enjoys a higher performance with lower execution complexity but does introduce some extra training overhead.** The training overhead is comparable with some baselines such as SePS. \n- **Discussion:** Kaleidoscope sacrifices training computational cost for the following benefits:\n    - During training, Kaleidoscope achieves better policy diversity while maintaining high sample efficiency.\n    - During execution, Kaleidoscope enjoys better performance with lower overhead.\n\n    **We believe this tradeoff is worthwhile** because:\n    - In a learning task, training is conducted once. After deployment, execution performance and efficiency are what matter.\n    - In an RL setting, sample efficiency is more crucial than computational efficiency because online data in the real world is usually difficult to collect.\n    - Training computational cost can potentially be reduced by employing strategies such as mixed precision training.\n\nWe will incorporate these results and discussions into our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful comments and valuable feedback. In our work, we propose a novel adaptive partial parameter sharing scheme that fosters policy heterogeneity while maintaining high sample efficiency in MARL tasks. This approach leads to superior policies in terms of performance and execution cost.

We are pleased that reviewers find our method interesting (BN4p), novel and well-motivated (WQkZ, CAY2), with conprehensive experiments (WQkZ, Y5Eh, CAY2) and good writing quality (WQkZ, BN4p, CAY2). In response to reviewers' concerns and suggestions, we provide the following additional results here and in the response PDF:
- **Further visualization results** (Fig. 7 in the response PDF): We visualize the pairwise mask differences among agents and the agent trajectories at different training stages. As training progresses, the test return increases and diversity loss decreases, indicating better performance and greater diversity among agent policies. Correspondingly, mask differences among agents increase, and the agent trajectory distribution becomes more diverse.
- **Baseline MultiH** - Agents share all parameters except for individual action heads (Purple curves in Fig. 3-4 in the response PDF)
- **Ablation study on fixed masks vs. learnable masks** (Grey curves in Fig. 6 in the response PDF)
- **Comparison results on masking on weights/activations** (Fig. 9 in the response PDF)
- **Updated execution cost and training costs** (Table 2, 9, 10 below)


Table 2: Averaged Testing FLOPs 

| Methods       | NoPS | FuPS+ID | MultiH | SePS | SNP   | Kaleidoscope  |
|---------------|------|---------|--------|------|-------|---------------|
| **MPE**       | 1.0x | 1.0x | 1.0x | 1.0x | 0.988x | **0.901x**    |
| **MaMuJoCo**  | 1.0x | 1.0x | 1.0x | 1.0x | 0.900x | **0.680x**    |
| **SMACv2**    | 1.0x | 1.0x | 1.0x | 1.0x | 0.988x | **0.890x**    |

Table 9: Averaged Training Wall Time

| Methods   | NoPS  | FuPS+ID | MultiH | SePS  | SNP    | Kaleidoscope |
|-----------|-------|---------|--------|-------|--------|--------------|
| MPE       | 2.315x| **1.0x**| 1.453x | 2.497x| 1.437x | 2.086x       |
| MaMuJoCo  | 1.0x  | 1.0x    | 1.103x | 0.979x| **0.965x** | 1.524x   |
| SMACv2    | 1.813x| 1.0x    | 2.212x | 3.511x| **0.978x** | 1.379x   |

Table 10: Averaged Training Memory

| Methods   | NoPS  | FuPS+ID | MultiH | SePS  | SNP    | Kaleidoscope |
|-----------|-------|---------|--------|-------|--------|--------------|
| MPE       | 1.005x| **1.0x**| **1.0x**| 1.058x| 1.002x | 1.236x       |
| MaMuJoCo  | 1.004x| **1.0x**| **1.0x**| 1.017x| **1.0x**| 1.059x     |
| SMACv2    | 1.360x| **1.0x**| 1.109x | 1.938x| 1.018x | 3.182x      |



References

[1] Christianos, Filippos, et al. "Scaling multi-agent reinforcement learning with selective parameter sharing." International Conference on Machine Learning. PMLR, 2021. \
[2] Li, Dapeng, et al. "Adaptive parameter sharing for multi-agent reinforcement learning." ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.

Meta-review: This paper presents Kaleidoscope, a novel approach for adaptive parameter sharing in Multi-Agent Reinforcement Learning (MARL) using learnable weight masks. The method incorporates regularization to promote policy diversity and a resetting mechanism for parameter reuse. The experimental results demonstrate that Kaleidoscope outperforms existing baselines on various MARL benchmarks.

The paper is well-written and provides strong empirical support, with thorough experiments and analysis. Although it initially lacked comparisons with certain baselines and did not fully address some related work, the authors have effectively addressed these issues in their rebuttal. They clarified the role of FuPS+ID and provided additional experiments to justify their approach over alternative methods. Lastly, the initial motivation for using STR over methods that learn different activations per agent was not entirely clear. The authors conducted additional experiments to justify their choice, showing that weight-level masking offers superior flexibility and performance.

Overall, the authors have addressed the reviewers' main concerns effectively. The paper's contributions are substantial, offering a valuable advancement in parameter sharing for MARL. Therefore, it is recommended to accept the paper, provided that the revisions discussed in the rebuttal are incorporated into the final version.
Decision: Accept (poster)