Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Training
Authors: Pihe Hu
Shaolong Li
Zhuoran Li
Ling Pan
Longbo Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-Agent Reinforcement Learning
Dynamic Sparse Training
Value Learning
Primary area: reinforcement_learning

Abstract: Deep Multi-agent Reinforcement Learning (MARL) relies on neural networks with numerous parameters in multi-agent scenarios, often incurring substantial computational overhead. Consequently, there is an urgent need to expedite training and enable model compression in MARL. This paper proposes the utilization of dynamic sparse training (DST), a technique proven effective in deep supervised learning tasks, to alleviate the computational burdens in MARL training. However, a direct adoption of DST fails to yield satisfactory MARL agents, leading to breakdowns in value learning within deep sparse value-based MARL models. Motivated by this challenge, we introduce an innovative Multi-Agent Sparse Training (MAST) framework aimed at simultaneously enhancing the reliability of learning targets and the rationality of sample distribution to improve value learning in sparse models. Specifically, MAST incorporates the Soft Mellowmax Operator with a hybrid TD-($\lambda$) schema to establish dependable learning targets. Additionally, it employs a dual replay buffer mechanism to enhance the distribution of training samples. Building upon these aspects, MAST utilizes gradient-based topology evolution to exclusively train multiple MARL agents using sparse networks. Our comprehensive experimental investigation across various value-based MARL algorithms on multiple benchmarks demonstrates, for the first time, significant reductions in redundancy of up to $20\times$ in Floating Point Operations (FLOPs) for both training and inference, with less than 3% performance degradation.

Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Training
Pihe Hu'
Shaolong Li"
Zhuoran Li
Tsinghua University Beijing. China hupihe@gmail.co com
Central South University
Tsinghua University
Changsha, China
Beijing, China
maaolonglii@@@maann com
lizr20Gmails tsinghua. llkangua.eddu.oom edu.
Ling Pan
Longbo Huang Tsinghua University
Hong Kong University of
Science and Technology Hong Kong, China lingpandust. hk
Beijing, China
longbohuang@tsinghuaa.c
Abstract
Deep Multi-agent Reinforcement Learning (MARL) relies on neural networks with numerous parameters multi- multi-agent scenarios, often incurring substantial computational overhead. Consequently, there is urgent need to expedite training and enable model compression MARL. This paper proposes the utilization dynamic sparse training (DST), a technique proven effective in deep supervised learning tasks, to alleviate the computational burdens MARL training. However, direct adoption of DST fails to yield satisfactory MARL agents, leading to break- downs in value learning within deep sparse value-based MARL models. Motivated by this challenge, introduce innovative Multi-Agent Sparse Training (MAST) framework aimed at simultancously enhancing the reliability of learning targets and the rationality of sample distribution to improve value learning in sparse mod- els. Specifically, MAST incorporates the Soft Mellowmax Operator with a hybrid TD-(A) schema to establish dependable learning targets. Additionally, it employs a dual replay buffer mechanism to enhance the distribution training samples. Building upon these aspects, MAST utilizes gradient basdeent-baedd topology evolution to exclusively train multiple MARL agents using sparse networks. Our compre- hensive experimental investigation across various value-based MARL algorithms on multiple benchmarks demonstrates, for the first time, significant reductions in redundancy of up to 20x Floating Point Operations (FLOPs) for both training and inference, with less than 3% performance degradation.
1 Introduction
Multi-agent reinforcement learning (MARL) (Shoham and Beyton-Brown, 2008 coupled with deep neural networks, has only revolutionized artificial intelligence but also showcased remarkable success across a wide range of critical applications. From mastering multi-agent video games such
Quake III Arena [Jaderberg al. 2019), StarCraft II [Mathieu al., 2021]. Dota [Berner let al. 2019), and Hide and Seek [Baker et al. 2020) to guiding autonomous robots through intricate real-world environments (Shalev-Shwartz et al. 2016 Da Silva al. 2017 Chen al., 2020b), deep
MARL has emerged as an indispensable and versatile tool for addressing complex, multifaceted challenges. Its unique capability capture intricate interactions and dependencies among multiple
"Equal contribution. 1 Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

agents has spurred novel solutions, solidifying its position as a transformative paradigm across various domains (Zhang al. 202T Albrecht al. 2023].
However, the exceptional success of deep MARI. comes at a considerable computational cost. Training these agents involves the intricate task of adapting neural networks to accommodate an expanded parameter space, especially in scenarios with substantial number of agents. For instance, the training regimen for AlphaStar [Mathieu et al. 2021]. tailored for StarCraft II, extended over grueling 14-day period, employing 16 TPUs per agent. Similarly, the OpenAI Five [Berner et al. 2019] model for Dota 2 underwent an extensive training cycle spanning 180 days and harnessing thousands of GPUs. This exponential increase in computational demands as the number of agents grows (and the corresponding joint action and state spaces) poses a significant challenge during MARL deployment.
To tackle these computational challenges, researchers have delved into dynamic sparse training (DST), a method that trains neural network models with dynamically sparse topology. For example, RigL [Evei et al. 2020] can train 90%-sparse network from scratch in deep supervised learning without performance degradation. However, in deep reinforcement learning (DRL), the learning target evolves in a bootstrapping manner [Tesauro et al. [995], and the distribution of training data is path-dependent (Desai et al. 2019), posing additional challenges to sparse training. Improper sparsification can result in irreversible damage to the learning path [Ig] et al. 2020]. Initial attempts at DST in sparse single-agent DRL training have faced difficulties in achieving consistent model compression across diverse environments, as documented in (Sokar et al. 2022 Graesser al.) 2022] This is mainly because sparse models may introduce significant bias, leading to unreliable learning targets and exacerbating training instability as agents learn through bootstrapping. Moreover, the partially observable nature of each agent makes training non-stationarity inherently more severe in multi-agent settings. Collectively, these factors pose significant challenges for value learning in each agent under sparse models.
We present a motivating experiment in Figure T where we evalu- ated various sparse training methods on the 3s5z task from SMAC (Samvelyan et al. 2019) using a neural network with only 10%
Performanca(15
of its original parameters. Classical DST methods, including SET
Mocanu et al., 2018] and Rigl. (Evei et al. 2020], , demonstrate
poor performance in MARI, scenarios, along with using static sparse networks (SS). Additionally, RLx2 as proposed in [Tan et al.] [2022 proves ineffective for multi-agent settings, despite enabling DST for single-agent settings with 90% sparsity. In contrast, our MAST framework in this work achieves a win rate of over 90%. In addition, the only prior attempt to train sparse MARL agents, as described in (Yang et al.] 2022], prunes agent networks during training with weight grouping [Wang et al., 2019]. However, this approach fails to maintain sparsity throughout training, and only achieves a final model sparsity of only 80%. Moreover, their experiment is confined to two-user environment, PredatorPrey-v2., in MuloCo [Todorov et al. 2012]. These observations highlight the fact that, despite its potential, the application of sparse networks in the context of MARL remains largely unexplored (A comprehensive literature review is deferred in Appendix A. Consequently, a critical and intriguing question arises:
SET RigL RLx2 MAST
Figure 1: Comparison of dif- ferent sparse training methods.
Can we train MARL agents effectively using ultra-sparse networks throughout?
We affirmatively address the question by introducing a novel sparse training framework, Multi-Agent Sparse Training (MAST). Since improper sparsification results in network fitting errors in the learning targets and incurs large policy inconsistency errors in the training samples, MAST ingeniously integrates the Soft Mellowmax Operator with hybrid TD-(A) schema to establish reliable learning targets. Additionally, it incorporates a novel dual replay buffer mechanism to enhance the distribution of training samples. Leveraging these components, MAST employs gradient-based topology evolution to exclusively train multiple MARL agents using sparse networks. Consequently, MAST facilitates the training of highly efficient MARL agents with minimal performance compromise, employing ultra-sparse networks throughout the training process.
Our extensive experimental investigation across various value-based MARL algorithms on multiple SMAC benchmarks reveals MAST's ability to achieve model compression ranging from 5x to 20x, while incurring minimal performance trade-offs (under 3%). Moreover, MAST demonstrates an

---

impressive capability to reduce the Floating Point Operations (FLOPs) required for both training and inference by up to 20x, showeasing a significant margin over other baselines (detailed in Section
2
Preliminaries
Deep MARL We model the MARL problem as a decentralized partially observable Markov decision process (Dec-POMDP) [Oliehoek al. 2016). represented by tuple (NV S. U, P. 2, O,n). Deep Multi-Agent Q-learning extends the deep Q learning method [Mnih et al. 2013] to multi- agent scenarios [Sunchag al. 2018 Rashid et al. 2020b [Son et al. | The agent- wise Q function is defined over its history Tú as Qi for agent 2. Subsequently, the joint action-valus function QT u) operates over the joint action-observationn history and joint action u. The objective, given transitions (T. T. T') sampled from the experience replay buffer B, is to minimize the mean squared error loss [(0) on the remporal-differeme
(TD) error ö y Qua (T, u). Here, the TD target = + 7y maxu' Qor u'), where Qiot is the target network for the joint action Q-function, periodically copied from Qio-- Parameters of Quot are updated using 0' 0 aVo(()), with o representing the learning rate.
We focus algorithms that adhere to the Centralized Training with Decentralized Execution (CTDE) paradigm Kraemer and Banerjec| 2016]), within which, agents undergo centralized training, where the complete action-observation history and global state available. However, during execution, they are constrained to individual local aetionnobservation histories. To efficiently implement CTDE, the Salvidual--Global---a -loball-aaaamummm (IGM) property (Son et al. [2019) in Eq. (T). serves as a key mechanism:
arg max Qun((: u) = (arg max Q1 (s, u1)
arg max QN (s, (N,,))
(1)
Many deep MARL algorithms Yu al. 2022 Pan al. 2022 adhere the IGM criterion, such as the QMIX series algorithms
Rashid et al. 2020bb a
Pan et al. 2021]. These algorithms employ
mixing network fa with non-negative weights, enabling the joint Q-function be expressed as (s, u) f. (Qi (s, u1), QN (s, uN)).
Dynamic Sparse Training Dynamic sparse training (DST), initially proposed in deep supervised learning, can train a 90% sparse network without performance degradation from scratch, such as in ResNet-50 (He et al. 2016) and MobileNet [Howard et al. 2017]. In DST, the dense network is randomly sparsified at initialization, as shown in Figure 2 and its topology is dynamically changed during training by link dropping and growing. Specifically. the topology evolution mechanism in MAST follows the RigL method (Evei al.] [020]), which improves the optimization of sparse neural networks by leveraging weight magnitude and gradient information to jointly optimize model parameters and connectivity.
Dense Network
Sparse Initialization
Link Drop
Link Grow
Figure 2: Illustration of dynamic sparse training.
RigL periodically and dynamically drops a subset of existing connections with the smallest absolute weight values and concurrently grows an equiva-
Algorithm Topology Evoltion (Evei et al. 2020
1: O,, Ni, sy: parameters, number of parameters, sparsity of layer I.
lent number of empty connections with the largest gradients. The pseudo-code of RigL is given in Algorithm []] where the symbol @ denotes the element-wist wise multiplication operator, M2 sym- bolizes the binary mask that delineates the sparse topology for the network 0 and & is the update fraction in training step f. This process maintains the network sparsity throughout the training with a strong evolutionary ability that saves training FLOPS and gives a sparse model after training. However, in DRL, the learning target evolves in a bootstrapping manner Tesauro et al.) 1995]. such that the distribution of training data is path-dependent (Desai et al., 2019), posing additional challenges to sparse training. Moreover, the partially observable nature of each agent exacerbates training non-stationarity, particularly in multi-agent settings. These factors collectively present significant hurdles for value learning in each agent under sparse models. As illustrated in Figure
2: for each layer do
3: k C: M
4: Larop Arg (opK( C Ma,
5: Ignow = o Vanu LI,
6: Update Ma, according to Taan and grow
7: G, +- ④ Mo.
8: end for
3

---

attempts to train ultra-sparse MARL models using simplistic topology evolution or the sparse training framework for single-agent RL have failed to achieve satisfactory performance. Therefore, MAST introduces innovative solutions to enhance value learning in ultra-sparse models by simultaneously improving the reliability of learning targets and the rationality of sample distribution.
3 Enhancing Value Learning in Sparse Models
This section outlines the pivotal components of the MAST framework for training sparse MARL agents. MAST introduces innovative solutions to enhance the accuracy of value learning ultra- sparse models by concurrently refining training data targets and distributions. Consequently, the topology evolution MAST effectively identifies appropriate ultra-sparse network topologies. This approach aligns with single-agent DRL, where sparse training necessitates co-design with the value learning method as described in (Tan et al., 2022). However, the partially observable nature of each agent exacerbates training non-stationarity in multi-agent settings. As illustrated in Figure 3 MAST implements two key innovations to achieve accurate value learning in ultra-sparse models: 1) hybrid TD(A) targets combined with the Soft Mellowmax operator mitigate estimation errors arising from network sparsity, and ii) dual replay buffers to reduce policy inconsistency errors due to sparsification.
Q1 (t (2(t)u)) mD
Q uF)
On-Policy Buffer
Off-Policy Buffer
Storing
Interaction
Ouor (T,
TD(2) with Soft Mellowmax
Environment
Sampling
(of, uf-1)
(1-x)
7. (O))
Dual buffers
Agent Agent
Mixing Network
Figure 3: An example of the MAST framework based on QMIX.
3.1 Improving the Reliability of Training Targets
Initially, we observe that the expected error is amplified under sparse models, motivating the intro- duction of multi-step TD targets. However, different environments may require varying values of step lengths to achieve optimal performance. Consequently, we focus on hybrid TD(A) targets, which can achieve performance comparable to the best step length across different settings. Additionally, find that the overestimation problem remains significant in sparse models. To address this, propose the use of the Soft Mellowmax operator in constructing learning targets. This operator is effective in reducing overestimation bias without incurring additional computational costs.
Hybrid TD()) Targets. In deep multi-agent Q-learning, temporal difference (TD) learning is fundamental method for finding an optimal policy, where the joint action-value network is iteratively updated by minimizing squared loss driven by the TD target. Let Mo be a binary mask representing the network's 's sparse topology, and denote the sparse network as Ô B @ Mo. where C signifies element-vee wise multiplication. Since sparse networks operate within a reduced hypothesis space with fewer parameters, the sparse network may induce a large bias, making the learning targets unreliable, as evidenced in [Sokar al. 2022).
Moreover, we establish Theorem 3.1 to characterize the upper bound of the expected multi-step TD error under sparse models, where the multi-step return at (s,, u,) is Tn (s,, u,) = Σm=0 " + 7 max, Qur(St4m. 0) under sparse models. As Eq shows, the expected multi-step TD error comes from two parts, intrinsical policy inconsistency error and network fitting error. Thus, Eq ( implies that the expected TD error will be enlarged if the network is sparsified improperly with larger network fitting error. Indeed, the upper bound of the expected TD error will be enlarged if the network fitting error is increased. Subsequently. it is infeasible for the model leam good policy. Eq. (2) also shows that introducing a multi-step return target discounts the network fitting error by factor of 7 in the upper bound of the expected TD error. Thus, employing multi-step return T.(") with a sufficiently large n, or even Monte Carlo methods (Sutton and Barto 2018), can effectively diminish the TD error caused by network sparsification for ry
Theorem 3.1. Denote the largel policy at timestep I, and p as the behavior policy generating the transitions (St, Ue ...... Sttns Ut+n). Denote the network fitting error as E(s, u) 1Qur(s, u.

---

(s 1. Then. the expected error between the multi-step TD target Tn conditioned on transitions from the behavior policy p and the true joint action-value function Vin
E (T, u.)] (ioc( ue)l S B,, BE (ee(stttn=((sttn)) + e( T (st+n)))
Network fitting error
(2)
+ Qirr(s, un) - Qia (5c: ue) + "epeEEpll Q (St-n (T Qi (stnn p ((sttn))l..
Policy incousistenyy evror
Discounneed policy inconsistency error
Proof. Please refer to Appendix A.3
However, the Monte Carlo method is prone to high variance, suggesting that optimal TD target in sparse models should be a multi-step return with judiciously chosen step length, balancing network fitting error due to sparsification and training variance. Figure 4 illustrates model perfor- mance across different step lengths and model sizes, revealing that an optimal step length exists for various model sizes. Moreover, the optimal step length increases model size decreases, which aligns with Theorem 3.T due to the increased network fitting error in models with higher sparsity. The above facts suggest the need to increase the
step length in the learning targets to maintain the performance of sparse models. However, the optimal step length varies across different set- tings. To address this, we introduce the TD(A)
32.5%
10.0%
7.5%
5.0%
19% T309
5% TDIN
target [Sutton and Barto 2018) to achieve
good trade-off: T (T
X)
2n= >
for À E [0, 1]. This target averages all possible multi-step returns (Tn J into single return using exponentially decaying weights, provid- ing computationally efficient approach with episode-fome data. In Figure 4. we also plot two representative performances of TD(A) under 10% and 5% model sizes, which are both close the optimal step length under different model sizes.
5.
Multi step length
Figure 4: Performances of different step lengths.
Previous studies [Fedus et al. 2020 have highlighted that an immediate shift to multi-step targets exacerbate policy inconsistency enror as shown in Eq. (2) Since the TD(A) target T7x averages all potential multi-step returns j an immediate transition to this target may encounter similar issues. To address this challenge, we adopt hybrid strategy inspired by the delayed mechanism proposed in [Tan et al. 2022]. Initially, when the training step is less than threshold To, we employ one-step TD targets (7) to minimize policy inconsistency errors. As training progresses and the policy stabilizes, we transit to TD(A) targets to mitigate sparse network fitting errors. This mechanism ensures consistent and reliable learning targets throughout the sparse training process.
Soft Mellowmax Operator. The max operator in the Bellman operator poses well-known theoret- ical challenge, namely overestimation, which hinders the convergence of various linear and non-linear approximation schemes TTitsiklis and Van Roy 1996). Deep MARL algorithms, including QMIX [Rashid et al. 2020b). also grapple with the overestimation problem. Several works have addressed this issue in dense MARL algorithms, such double critics [Ackermann et al. 2019], weighted critic updates (Sarkar and Kalita 202T]. the Softmax operator [Pan et al. 2021] and the Sub-Avg operator Wu et al. 2022]. J. However, these methods introduce additional computational costs, sometimes even doubling the computational budget, which is infeasible for our sparse training framework.
We turn attention to the Soft Mellowmax operator, which has been proven effective in reducing overestimation for dense MARL algorithms in [Gan et al.) 22021) For MARL algorithms satisfying the IGM property in Eq. ①. we replace the max operator in Q; with the Soft Mellowmax operator in Eq. (3) to mitigate overestimation bias in the joint-action Q function within sparse models:
sm. (Qc(T,)))
1
log Σ Ex
Eweu exp (aQu w()) exp (wQ, (T, u))
exp (aQ; (T, u))
(3)
LNEH
where w > 0, and a E R. Let T be the value estimation operator that estimates the value of the next state s'. Theorem 3.2 shows that the Soft Mellowmax operator can reduce the severe overestimation bias in sparse models.
Theorem 3.2. Let B(T) (T max Viae (s', be the bias of value estimates of T. For arbitrary joint-action Q-function Quan if there exists some V (s') such that
5

---

Vin (s') Vioe (5',2') for different joint actions, Ew (Qur (s', u²) ad)- Vin (s')) = 0, and
TU Ew (Quer (s', u') - Vin (6))) C (C > 0), then B (TRigI-MMIX-SM < B (TRigI-QMIX))
Proof. Please refer to Appendix A.4
Our empirical investigations reveal that overestimation remains a signif- icant performance barrier in sparse models, resulting in substantial per- formance degradation. Figure [2 il- lustrates the win rates and estimated values of QMIX with and without our Soft Mellowmax operator on 3s5z in the SMAC. Figure 5(a) shows that the performance of RigL-QMInE MMLL-MMIX-SM out- performs RigL-QMIX, while Figure demonstrates that the Soft Mellowmax operator effectively mitigates overestimation bias. These findings highlight that QMIX still faces overestimation issues in sparse models and underscore the efficacy of the Soft Mellowmax operator in addressing this problem.
Ret ---0
Tol Tol-ghn
wwwuaa WENS
The hot. Rat.Croo
(a) Win rates
(b) Estimated values
Figure 5: Effects of Soft Mellowmax operator.
Additionally, the Soft Mellowmax operator introduces negligible extra computational costs, it averages the Q function over each agent's ss individual action spaces rather than the joint action spaces used in the Softmax operator [Pan al. 2021], which grow exponentially with the number of agents.
3.2 Improving the Rationality of Sample Distribution
Although have improved the reliability of the learning target's confidence discussed above, the learning process can still suffer from instability due to improper sparsification. This suggests the need enhance the distribution of training samples to stabilize the training process. Improper sparsification can lead to irreversible damage to the learning path [g] et al. 2020) and exacerbate training instability as agents learn through bootstrapping.
Generally, sparse models are more challenging to train compared to dense models due to the reduced hypothesis space (Evci et al. 2019]. Therefore, it is important to prioritize more recent training samples to estimate the true value function within the same training budget. Failing to do so can result in excessively large policy inconsistency emror,, thereby damaging the learning process irreversibly.
MAST introduces a dual buffer mechanism utilizing two First-in-First-O0 (FIFO) replay buffers: Bi (with large capacity) and Bg (with small capacity), with some data overlap between them. While B, adopts an off- off-policy style, B, follows an on-policy approach. During each training step, MAST samples bj episodes from B, and b2 episodes from B2, conducting a gradient update using combined batch size of (b1 + b2). For instance, in our experiments, we set [B1] |B2] = 50 : 1 and
by : b2 = 3 1. Generally, training with online data enhances learning stability, the behavior policy closely matches the target policy. Conversely, training with offline data improves sample efficiency but can lead to instability.
Figure 6 illustrates the training dynamics for RigL-QMIX in the SMAC's 3s5z task. The green curve with large variances highlights the training instability of QMIX in sparse models. However, with the integration of dual buffers, QMIX's 's training stability and efficiency are significantly improved under sparse conditions, leading to con- sistent policy enhancements and higher rewards. Notably, the dual buffer mechanism does not enhance dense training, suggesting that this approach is particularly effective in sparse scenarios where net- work parameters are crucial for ensuring stable policy improvements.
tay- cIr Iay. Dualba
HiBsn daps
Figure 6: Different buffers.
Although prior works (Schaul et al. 2015 Hou al. 2017 [Banerjee] et al. 2022) have explored prioritized or dynamic-capacity buffers, their applicability in this context may be limited due to the data being in episode form in aalue-based deep MARL algorithms, making it difficult to determine the priority of each training episode. Similarly, the dynamic buffer approach in Tan et al. 2022 is also inapplicable, as the policy distance measure cannot be established for cpisode-Form data. This further emphasizes the unique effectiveness of the dual buffer approach in enhancing training stability for sparse MARL models.

---

We highlight that the performance improvement observed with MAST not due providing more training data samples to agents. Instead, it results from a more balanced training data distribution enabled by the utilization of dual buffers. The primary objective of incorporating dual buffers is to shift the overall distribution of training data. Figure 7 illustrates the distribution of samples induced by different policies, where behavior policy and target policy are defined in Theorem 3.I] Specifically, samples in the original single buffer subject to the distribution of the behavior policy (blue), which introduces policy inconsistency error d1. However, by utilizing dual buffers, the distribution of training samples shifts towards the more recent samples from the
Density
d2
d2
Behavior
Shilled Target
Sumple
Policy
Policy Policy
Space
Figure 7: Distribution Shift: d1 and d2 are distribution distances.
target policy, as there policy buffer. This reduces the policy inconsistency in our dual buffers, thereby bolstering the stability and effectiveness of the learning process under sparse models. Additionally, the extra on-policy buffer does not significantly reduce sample efficiency, as its capacity is small.
4 Experiments
In this section, we conduct comprehensive performance evaluation of MAST across various tasks in the StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al. 2019) benchmark. Additional experiments on the multi-agent MuJoCo (MAMuJoCo) [Pngg et al. 2021] benchmark are provided in
Appendix B.9 MAST serves as a versatile sparse training framework specifically tailored for value
decompostion based MARL algorithms. In Section 4.1 we integrate MAST with state-of -ott------rrrr
value-based deep MARL algorithms, including QMIX [Rashid et al., 2020b], WQMIX [Rashid al. 2020a], and RES [Pan et al. [202]] We also apply MAST to hybrid value-based and policy- based algorithm, FACMAC [Peng et al. 2021]. Subsequently, we assess the performance of sparse models generated by MAST in Section 4.2 Furthermore, comprehensive ablation study MAST
components is detailed in Appendix [..7 Each reported result represents the average performance over eight independent runs, each utilizing distinct random seeds.
Table 1: Comparisons of MAST with different sparse training baselines: "Sp." stands for "sparsity",
"Total Size" means total model parameters, and the data is all normalized w.r.t. the dense model.
Total Size
FLOPs
FLOPs
Tiny
SS
SET
RigL
RLx2
MAST
Alg.
Env.
Sp.
(Train)
(Test)
(%)
(%)
(%)
(%
(%)
(%)
3m
95%
0.066x
0.051x
0.050x
98.3
91.6
96.0
95.3
12.1
100.9
2s3z
95%
0.062x
0,051x
0.050x
83.7
73.0
77.6
69.4
45.8
98.0
Q-
3s5z
90%
0.109x
0.101x
0.100x
68.2
34.0
52.3
45.2
50.1
99.0
MIX
64*
90%
0.106x
0.100x
0.100x
58.2
40.2
67.1
48.7
9.9
97.6
Avg-
92%
0.086x
0.076x
0.075x
77.1
59.7
73.2
64.6
29.8
98.9
3m
90%
0.108x
0.100x
0.100x
98.3
96.9
97.8
97.8
98.0
98.6
2s3z
90%
0.106x
0.100x
0.100x
89.6
75.4
85.9
86.8
87.3
100.2
WQ-
3s5z
90%
0.105x
0.000x
0.100x
70.7
62.5
56.0
50.4
60.7
96.1
MIX
64*
90%
0.104x
0.100x
0.100x
51.0
29.6
44.1
41.0
52.8
98.4
Avg.
90%
0.106x
0.100x
0.100x
77.4
66.1
70.9
69.0
74.7
98.1
3m
95%
0.066x
0,055x
0.050x
97.8
95.6
97.3
91.1
97.9
99.8
2s3z
90%
0.111x
0.104x
0.100x
96.5
92.8
92.8
94.7
94.0
98.4
RES
3s57
85%
0.158x
0.154x
0.150x
95.1
89.0
90.3
92.8
86.2
99,4
64*
85%
0.155x
0.151x
0.150x
83.3
39.1
44.1
35.3
72.7
104.9
Avg.
89%
0.122x 0.116x
0.1
93.2
79.
81.1
78.5
87.7
100.6
4.1
Comparative Evaluation
Table presents comprehensive summary of our comparative evaluation in the SMAC benchmark. where MAST is benchmarked against the following baseline methods: (i) Tiny: Utilizing tiny dense networks with a parameter count matching that of the sparse model during training. (ii) SS: Employing static sparse networks with random initialization. (iii) SET [Mocanu et al. 2018]: Pruning connections based on their magnitude and randomly expanding connections. (iv) Rigl. [[vci et al.) 2020]: This approach leverages dynamic sparse training, akin to MAST, by removing and adding
7

---

connections based on magnitude and gradient criteria, respectively. (v) RLx2 (Tan et al. 2022]: A
specialized dynamic sparse training framework tailored for single-agent reinforcement learning.
We set the same sparsity levels for both the joint Q function Qio. and each individual agent's Q function Qi- For every algorithm and task, the sparsity level indicated in correspnossss the highest admissible sparsity threshold of MAST. Within this range. MAST's performance consistently remains within a 3% margin compared to the dense counterpart, effectively representing the minimal sparse model size capable of achieving performance parity with the original dense model. All other baselines are evaluated under the same sparsity level as MAST. We assess the performance of each algorithm by computing the average win rate per episode over the final 20 policy evaluations conducted during training, with policy evaluations taking place at 10000-step intervals. Identical hyperparameters are employed across all scenarios.
Performance Table I [unequivocally illustrates MAST's substantial performance superiority over
all baseline methods in all four environments across the three algorithms. Notably, static sparse (SS) consistently exhibits the lowest performance on average, highlighting the difficulty of finding optimal sparse network topologies in the context of sparse MARL models. Dynamic sparse training methods, namely SET and RigI., slightly outperform SS, although their performance remains unsatisfactory. Sparse networks also, on average, underperform tiny dense networks. However, MAST significantly outpaces all other baselines, indicating the successful realization of accurate value estimation through our MAST method, which effectively guides gradient-based topology evolution. Notably, the single-agent method RLx2 consistently delivers subpar results in all experiments, potentially due to the sensitivity of the step length in the multi-step targets, and the failure of dynamic buffer for episode-fore form training samples.
To further substantiate the efficacy of MAST, we conduct per- formance comparisons across various sparsity levels in 3s52, as depicted in Figure 8 This reveals an intriguing observa- tion: the performance of sparse models experiences a sharp decline beyond critical sparsity threshold. Compared to con- ventional DST techniques, MAST significantly extends this critical sparsity threshold, enabling higher levels of sparsity while maintaining performance. Moreover, MAST achieves higher critical sparsity threshold than the other two algorithms with existing baselines, e.g., SET and RigL, achieving sparsity level of over 80% on average. However, is essential note that the Softmax operator in RES averages the Q function over joint action spaces, which grow exponentially with the number of agents, resulting in significantly higher computational FLOPPs and making it computationally incomparable to MAST. The detailed FLOPS calculation is deferred to Appendix B.4.2
10
0.8
0.
Trn
Righ
Sparsty96)
Figure 8: Different sparsity.
FLOP Reduction and Model Compression In contrast to knowledge distillation or behavior cloning methodologies, exemplified by works such as [Livne and Cohen 2020 Vischer et al. 2022). MAST maintains a sparse network consistently throughout the entire training regimen. Consequently, MAST endows itself with unique advantage, manifesting in remarkable acceleration of training FLOPs. We observed an acceleration of up to 20x in training and inference FLOPs for MAST-QMIX in the 2s3z task, with an average acceleration of 10x, 9x, and 8x for QMIX, WQMIX, and RES- QMIX, respectively. Moreover, MAST showcases significant model compression ratios, achieving reductions in model size ranging from 5x to 20x for QMIX, WQMIX, and RES-QMIX, while incurring only minor performance degradation, all below 3%.
ERCMAOO SAIEE
IRCMAE Deren INCMAE- SIT INCMAE- Aod IACHACNASS
3.0
ISCMAC Denge
3.3
FNCMAC-SET GET
HEMMEE NECAA REEL MACHAC-MAST
FREMAC Aay
TACHAC MAST
0.8
8.0
LB LX LTE Tie
022
L30
LE
LX
135
220
C25
Narmn
120 222
138
275
130
(a) MMM
(b) 2c_vs_
vss_62zg
(c) bane vs_ bane
Figure 9: Mean episode win rates on different SMAC tasks with MASTFACMAC.
8

---

Results on FACMAC In addition to pure value- based deep MARL algorithms, we also evaluate MAST with a hybrid value-based and policy-based algorithm, FACMAC [Peng et al. [02T], in SMAC. The results are presented in Figure [8] From the figure, we observe that MAST consistently achieves a performance comparable with that of the dense models, and outperforms other methods in three environments, demonstrating its applicability across different algorithms.
4.2 Sparse Models Obtained by MAST
We conduct a comparative analysis of diverse sparse network architectures. With identical sparsity levels, distinct sparse architectures lead to different hypothesis spaces. As emphasized in [Frankle and Carbin 2019]. specific architectures, such as the "winning ticket," outperform randomly generated counterparts. We compare three architectures: the "random ticket" (randomly sampled topology held constant during training), the "winning ticket" (topology from MAST or RigL run and kept unchanged during training). and the "cheating ticket" (trained by MAST).
Figure TO illustrates that both the "cheating ticket" and "winning ticket" by MAST achieve the highest performance, closely approach- ing the original dense model's performance. Importantly, using a fixed random topology during training fails fully exploit the bene- fits of high sparsity, resulting in significant performance degradation. Furthermore, RigL's "winning ticket" fares poorly, akin to the "ran- dom ticket." These results underscore the advantages of our MAST approach, which automatically discovers effective sparse architec- tures through gradient- based topology evolution, without the need for pretraining methods, e.g., knowledge distillation [Schmitt et al.) 2018). Crucially, our MAST method incorporates key elements: the hybrid TD(A) mechanism, Soft Mellowmax operator, and dual buffers. Compared to RigL, these components significantly improve value estimation and training stability in sparse models facilitating efficient topology evolution.
Whiiig Ticatbyy MAST whine Tlouthy Aggl sanden Triee
Figure 10:
Comparison of dif- ferent sparse masks.
Figure showcases the evolving sparse mask of a hidden layer during MAST-QMIX training in 3s5z, capturing snapshots at 0, 5, 10, and 20 million steps. In Figure [ the light pixel in row and column j indicates the existence of the connection for input dimension j and output dimension i, while the dark pixel represents the empty connection. Notably, a pronounced shift in the mask is evident at the start of training, followed by gradual convergence of connections within the layer onto a subset of input neurons. This convergence is discernible from the clustering of light pixels forming continuous rows in the lower segment of the final mask visualization, where several output dimensions exhibit minimal or no connections. This observation underscores the distinct roles played by various neurons in the representation process, showing the prevalent redundancy in dense models.
Figure 1I: Visualization of weight masks in the first hidden layer of agent by MAST-QMIX.
When sparsifying agent networks, MAST only re- quires setting the total sparsity. The sparsity of dif- ferent agents is determined automatically by MAST, by concatenating all the agent networks together in our implementation based on PyMARL (Samvelyan al. [2199] (detailed in Appendix B.3). and treating these networks as a single network during topology evolution with only a total sparsity requirement. We visualize the trained masks of different agents in 3s5z in Figure 12()) including the masks of two stalkers and two zealots, respectively.
Stalkers
shea
(a) Agent mask visual- (b) Different sparsity ization.
patterns.
Figure 12; Agent roles.

---

Interestingly, we find that the network topology in the same type of agents looks very similar. However, stalkers have more connections than zealots, which aligns with the fact that stalkers play more critical roles due to their higher attack power and wider attack ranges. This observation highlights an advantage of the MAST framework, i.e., can automatically discover the proper sparsity levels for different agents to meet the total sparsity budget. To further validate this point, we compare the adaptive allocation scheme with fixed manually-set patterns in Figure [2(b) The manual patterns include (stalker- 10%, zealot- 10%). (stalker-8% zealot- 12%), and (stalker- 14%, zealot-6%). The
results also show that the adaptive sparsity allocation MAST outperforms other manual sparsity patterns, demonstrating the superiority of MAST.
Conclusion
This paper introduces MAST, novel sparse training framework for valued-based deep MARL. We identify and address value estimation errors and policy inconsistency caused by sparsification, two significant challenges training sparse agents. MAST offers innovative solutions: hybrid TD(A) target mechanism combined with the Soft Mellowmax operator for precise value estimation under extreme sparsity, and dual buffer mechanism reduce policy inconsistency and enhance training stability. Extensive experiments validate MAST's effectiveness in sparse training, achieving model compression ratios of 5x x with minimal performance degradation and up remarkable 20x reduction in FLOPSs for both training and inference.
Acknowledgement
This work was supported by the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2020AAA0108000 and 2020AAA00108033,
10

---

References
Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. Reducing overestimation bias in multi-agent domains using double centralized critics. arXiv preprint 1rXiv:1910.01465 2019.
SV Albrecht, F Christianos, and L Schäfer. Multi-agent reinforcement learning: Foundations and modern approaches. Massachusetts Institute of Techhology: Cambridge, MA, USA, 2023.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference Learning Representations, 2020.
Chayan Banerjee, Zhiyong Chen, and Nasimul Noman. Improved soft actor-critic: Mixing prioritized off-policy samples with on on-policy experiences. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1.XXi:9912.066688 2019.
Christopher Brix, Parnia Bahar, and Hermann Ney. Successfully applying the stabilized lottery ticket hypothesis to the transformer architecture. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, pages 3909-3915, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33:15834-15846, 2020a.
Yu-Jia Chen, Deng-Ka -ai Chang, and Cheng Zhang. Autonomous tracking using a swarm of uavs: A constrained multi-agent reinforcement learning approach. IEEE Transactions on Vehicular Technology, (9(11):1302-117111 2020b.
Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrecht. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning, pages 1989-1998. PMLR, 2021.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling- arXiv preprint 1rXiv:1412.3555, 2014.
Arnau Colom. Empirical analysis of exploration strategies in qmix. 2021.
Felipe Leno Da Silva, Ruben Glatt, and Anna Helena Reali Costa. Simultaneously learning and ad- vising in multiagent reinforcement learning. In Proceedings of the 16th conference autonomous agents and multiagent systems, pages 1100-1108, 2017.
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. EMNLP-IJCNLPP 2019, page 153, 2019.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.0484// 2019.
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Newral Information Processing Systems, 30, 2017.
Utku Evei, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. ICML. 2019 Workshop on Identifying and Understanding Deep Learning Phenomena, 2019.
11

---

Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943-2952. PMLR, 2020.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney, Revisiting fundamentals of experience replay. In International Conference on Machine Learning, pages 3061-3077 PMLR, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International conference on learning representations, 2019.
Yaozhong Gan, Zhe Zhang. and Xiaoyang Tan. Stabilizing q learning via soft mellowmax operator. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7501-7509, 2021.
Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning. International Conference on Machine Learning, pages 7766-7792. PMLR, 2022.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In Autonomous Agents and Multiagent Systems: AAMAS 2017 Workshops, Best Papers, São Paulo, Brazil, May 8-12, 2017, Revised Selected Papers 16, pages 66 66-83. Springer, 2017.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruming, trained quantization and huffman coding. In International conference on learning representations, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Yuenan Hou, Lifeng Liu, Qing Wei, Xudong Xu, and Chunlin Chen. A novel ddpg method with prioritized experience replay. In 2017 IEEE international conference on systems, man, and cybernetics (SMC), pages 316-321. IEEE, 2017.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 0ee./004440040 2017
Maximilian Igl, Gregory Farquhar, Jelena Luketina. Wendelin Bochmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International Conference Learning Representations, 2020.
Max Jaderberg, Wojciech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human- level performance in 3d multiplayer games with population-based reinforcement learning. Science, 3344(6443):59--666 2019.
Woojun Kim and Youngchul Sung. Parameter sharing with network pruning for scalable multi-agent deep reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 1950, 2023.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based connection sensitivity. In International conference on learning representations, 2019.
Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement relt-ageetrinnfoommmmen learning. Advances in Neural Information Processing Systems, 34:3991 14002, 2021.
12

---

Dor Livne and Kobi Cohen. Pops:
Policy pruning and shrinking for deep reinforcement learning. IEEE Journal of Selected Topics in Signal Processing, 14(4):789-801. 801, 2020.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. In International conference on learning representations, 2018.
Michael Mathieu, Sherjil Ozair, Srivatsan Srinivasan, Caglar Gulcehre, Shangtong Zhang, Ray Jiang, Tom Le Paine, Konrad Zolna, Richard Powell, Julian Schrittwieser, et al. Starcraft ii unplugged: Large scale offline reinforcement learning. In Deep RL Workshop NeurIPS 2021, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:13/2.5602, 2013.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 91):23383, 2018.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pages 2498-2507. PMLR, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1264-11272, 2019.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pages 4646 4655. PMLR, 2019.
Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume Springer, 2016.
Ling Pan, Tabish Rashid, Bei Peng, Longbo Huang, and Shimon Whiteson. Regularized softmax deep multi-agent q-learning. Advances Neural Information Processing Systems, 34:1365-1377, 2021.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In International conference on machine learning, pages 17221-17237 PMLR, 2022.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic diffcrentiation in pytorch. 2017.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemae: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34:1 12288-22-12 2021.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding
monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 33:10199-102111 10210, 2020a.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234-7284 2020b.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, hhs/1992.004443 2019.
Tamal Sarkar and Shobhanjana Kalita. A weighted critic update approach to multi agent twin delayed deep deterministic algorithm. In 2021 IEEE 18th India Council International Conference (INDICON), pages 1-6. IEEE, 2021.
13

---

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952. 2015.
Simon Schmitt, Jonathan Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning. arXiv preprint arXiv: 03835,, 2018.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter E Latham, and Yee Teh. Powerpropa- gation: A sparsity inducing weight reparameterisation. Advances in neural information processing systems, 34:28889-28903, 2021.
Shai Shalev-Shwarde Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv: 2016.
Yoav Shoham and Kevin Leyton-Browe Brown. Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008.
Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. Dynamic sparse training for deep reinforcement learning. 2022.
Kyunghwan Son, Daewoo Kim, Wan Kang. David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pages 5887-5896. PMLR, 2019.
Suraj Srinivas, Akshayvarun Subramanya, and R Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pages 145, 2017.
Chuangchuang Sun, Macheng Shen, and Jonathan P How. Scaling up multiagent reinforcement learning for robotic systems: Learn an adaptive sparse communication graph. In 2020 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 11755-11762 11762. IEEE, 2020.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanetot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, et al. Value-decoompositoon networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2085-2087, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yiqin Tan, Pihe Hu, Ling Pan, Jiatai Huang, and Longbo Huang. RIx2: Training sparse deep rein- forcement learning model from scratch. In International Conference on Learning Representations, 2022.
Gerald Tesauro al. Temporal difference learning and td-gammon. Communications ofthe the ACM, 38 (3):58-68, 1995.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEFF/RSJ international conference intelligent robots and systems, pages 5026-5033. IEEE, 2012.
JN Tsitsiklis and B Van Roy. An analysis of temporal-differte differencee learning with function approxima- tiontechnical. Rep. LIDS-P-2322). Lab. Inf. Decis. Syst. Massachusetts Inst. Technol. Tech. Rep, 1996.
Marc Aurel Vischer, Robert Tjarko Lange, and Henning Sprekeler. On lottery tickets and minimal task representations in deep reinforcement learning. In International conference learning representations, 2022.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by
preserving gradient flow. In International Conference on Learning Representations, 2020.
Xijun Wang, Meina Kan, Shiguang Shan, and Xilin Chen. Fully learnable group convolution for acceleration of deep neural networks. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 9049-9058, 2019.
14

---

Haolin Wu, Jianwei Zhang, Zhuang Wang, Lin, and Hui Li. Sub-avg: Overestimation reduction for cooperative multi-agent reinforcement learning. Neurocomputing. 474:94-106, 2022.
Je Yang, JaeUk Kim, and Joo- Young Kim. Learninggroup: real-time sparse training on fpga via learnable weight grouping for multi-agent reinforcement learning. In 2022 International Conference on Field-Pres Prolrr-rogaea Technology (ICFPT), pages 1-9. IEEE, 2022.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang. Alexandre Bayen, and Yi Wu. The surprising effectivenesss of ppo cooperative multi-agent games. Advances in Neural Information Processing Systems, 15:24611-2460 2022.
Haonan Yu, Sergey Edunov. Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. In International conference learning representations, 2020.
Hongjie Zhang. Zhuocheng He, and Jing Li. Accelerating the deep reinforcement learning with neural network compression. 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2019.
Kaiqing Zhang, Zhuoran Yang. and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pages 321-384, 2021.
15

---

Supplementary Materials
Additional Details for MAST Framework
16
A.I
Comprchensive Related Work
16
A.2
Decentralized Partially Observable Markov Decision Process
17
3
Proof of Theorem 3.
17
A44
Proof of Theorem 3.2
18
A.5
MAST with Different Algorithms
18
A.6
6
Limitations of MAST
21
Experimental Details
B.
Hardware Setup
21
2
Environment
B.3
Hyperparameter Settings
22
B. 4
Calculation of Model Sizes and FLOPs
22
Training Curves of Comparative Evaluation in Section 4.1
................................
26
B.6
Standard Deviations of Results in Table
27
B.7
Ablation Study
28
Sensitivity Analysis for Hyperparameters
29
B,9
Experiments on QMIX in Multi-Agent MuloCo
30
B. 10
Experiments FACMAC in SMAC
30
Visualization Sparse Masks
30
A
Additional Details for MAST Framework
A.1
Comprehensive Related Work
Sparse networks, initially proposed in deep supervised learning can train 90%-sparse network without performance degradation from scratch. However, for deep reinforcement learning, the learning target is not fixed but evolves in a bootstrap way [Tesauro et al. 1995]. and the distribution of the training data can also be non-stationary (Desai et al., 2019), which makes the sparse training more difficult. In the following, list some representative works for training sparse models from supervised learning to reinforcement learning.
Sparse Models in Supervised Learning Various techniques have been explored for creating sparse networks, ranging from pruning pre-trained dense networks (Han al. 2015 2016 Srinivas
et al. (017]] to employing methods like derivatives [Dong et al. 2017 Molchanov et al. [009]] regularization Louizos al. 2018), dropout [Molchanov al. [017]] and weight reparameterization
(Schwarz et al. [2021]. Another avenue of research revolves around the Lottery Ticket Hypothesis (LTH) Frankle and Carbin 2019), which posits the feasibility of training sparse networks from scratch, provided a sparse "winning ticket" initialization is identified. This hypothesis has garnered
support in various deep learning models [Chen et al., [2020a] Brix al.) [2020]. Additionally, there is body of work dedicated to training sparse neural networks from the outset, involving techniques that evolve the structures of sparse networks during training. Examples include Deep Rewiring
(DeepR) [Bellec et al. 2018), Sparse Evolutionary Training (SET) [Mocanu et al. 2018)] Dynamic Sparse Reparameterization (DSR) [Mostafa and Wang 2019]. J, Sparse Networks from Scratch (SNFS) [Dettmers and Zettlemoyer 2019 and Rigged Lottery (RigL) [Evci et al. 2020 Furthermore,
16

---

methods like Single-Shot Network Pruning (SNIP) [eee et al. 2019] and Gradient Signal Preservation (GraSP) [Wang et 2020] are geared towards identifying static sparse networks prior training.
Sparse Models in Single-Agent RL Existing research [Schmitt al. 2018
Zhang et al., 2019
has employed knowledge distillation with static data to ensure training stability and generate small dense agents. Policy Pruning and Shrinking (PoPs) [Livne and Cohen 2020 generates sparse agents through iterative policy pruning, while the LTH in DRL is first indentified in (Yu et al. 2020J. Another line of investigation aims train sparse DRL models from scratch, eliminating the necessity of pre- rre-training dense teacher. Specifically, (Sokar et al., 2022] introduces the Sparse Evolutionary Training (SET) approach, achieving& remarkable 50% sparsity level through topology evolution in DRL. Additionally, [Graesser et al., [2022] observes that pruning often yields superior results, with plain dynamic sparse training methods, including SET and RigL, significantly outperforming static sparse training approaches. More recently, Rl.x2 [Tan et al., 2022] has demonstrated the capacity to train DRL agents with highly sparse neural networks from scratch. Nevertheless, the application of RLx2 in MARL yields poor results, as demonstrated in Section
Existing works have made attempts to train sparse MARL agents, such (Yang al. 2022]. which prunes networks for multiple agents during training, employing weight grouping Wang et al. 2019]. Another avenue of sparse MARL research seeks to enhance the scalability of MARL algorithms through sparse architectural modifications. For instance, (Sun iet al. [2200] proposes the use of a sparse communication graph with graph neural networks to reduce problem scale. [Kim and Sung 2023] adopts structured pruning for deep neural network to extend the scalability. Yet another strand of sparse MARL focuses on parameter sharing between agents to reduce the number of trainable parameters, with representative works including [Gupta et al.] [017]] Li et al. 2021 Christianos et al. [021]. However, existing methods fail to maintain high sparsity throughout the training process, such that the FLOPs reduction during training is incomparable to the MAST framework outlined in our paper.
Sparse Models in MARL
A.2 Decentralized Partially Observable Markov Decision Process
We model the MARL problem as decentralized partially observable Markov decision process (Dec-POMDP) [Oliehoek et al. 2016). represented by a tuple S.U. P.r, 2,r,2,0. O,7), 7), where N = {1..... N) denotes the finite set of agents, S is the global state space, 2 is the action space for an agent, P is the transition probability, r is the reward function, Z is the observation space for an agent, O is the observation function, and and 7 [0, 1) is the discount factor. At each timestep each agent i E N receives an observation z E Z from the observation function O(s,)) i) S x V + Z due to partial observability, and chooses an action u, Ut which forms a joint action u E U = 2t" The joint action u taken by all agents leads to a transition to the next state s' according to transition probability P(s' s, u) S U x S + [0. 1] and joint reward r(s, u) : S l 1 R. As the time goes by, each agent E W has an action-observane
history E T = (3 x 2))", " where T is the history space. Based on T;, each agent outputs an action u; according to its constructed policy n, (u; TT) T x 21 + [0, 1). The goal of agents is to find an optimal joint policy π = (TI,.... ,N), which maximize the joint cumulative rewards J π = Eut-n(-laa (tlee)())))))))))))) P..sap((Psssss. |E y'r(st, ud). where so is the initial state. The joint action-value function associated with policy π is defined as (st u²) = (1 P(t ((144)
EE= 7'r(Stti: ux-i)].
A.3 Proof of Theorem 3.1
Proof. By the definitions of multi-step targets, we have
Bp (T uc)
=Bp]
y Kresk + 7 max Quor (St+n, ui
k=0
a-1
E
*retk + "ua (Sttn, (st+n));))
k=
17

---

a-1
=Ep
7 Tetk + QuQua (St+ns P(ttn: ((8)-n):
k=
+ 7
Rp Quml (St+n: (8t+n)):0) Que
ppl
7 retk + -"Qiir ($t+m, ((sc))))
k=0
+ 7 " (Qua (#e+ro p(setn); () Qie (8i+ns (((++))
+ 7 "E, 10t (St+m ($c4n): 0) Quo (8t+ns p ((tttn);
-Qa (st, un) "
"pppcpp(sttn (a:i+Hnnn((::0000 (()++n))] BP Quai (8ttm: TT (T)ttn); Qwo (8t+n, p(st+n);
Besides, we also have
Ep Qior (8t+n: a (8(4m);;)) 0)
Ouo (Se+no ((844n):
=Bp Que (#t+ 7 (8ttn); ;) 0) - Qioe (deen; π (8t+n)))
+ EplQie (St+ns 7 (8(4n)) - Qio ($t+ns p(((+n)]]
+ Epl (8e4n, ((8t+n)) Qu (8t+ns ((3)4m); 6)]
Ep (e(sttns (8)44))]] Epl (st+as π ($1+n)) Qhat (setm: ((sc+n))) (Eeei)))lelleeseeeeeeee (ele((sst P(Sem))]] (5)
Thus, combining Eq. (4) and Eq. (5) gives
ET (Se: ux] Qix (st, ut) = 7 Ep 2c (8 p(()+n)) + c(sttn, ot (8t+n)))
Network fitting errer
Qha (se; uz) Qia (8t, u.) " 1 (8ttm: 7t (8t+a)) Qhe (st-n, p(
Policy inconsistency
Discounted policy inconsistency emoor
A.4 Proof of Theorem 3.2
Relace the Softmax operator in the proof of Theorem 3 in Pan et al. 2021 with Eq.
33) gives the
result directly.
A.5 MAST with Different Algorithms
In this section, we present the pseudocode implementations of MAST for QMIX Rashid et al. 2020b] and WQMIX Rashid et al., 2020a| in Algorithm 2 and Algorithm 3 respectively. It is noteworthy that RES [Pan et al. 2021] exclusively modifies the training target without any alterations to the learning protocol network structure. Consequently, the implementation of MAST with RES mirrors that of QMIX.
Crucially, MAST stands as versatile sparse training framework, applicable to range of value decomposition- based MARL algorithms, extending well beyond QMIX, WQMIX and RES. Fur- thermore, MAST's three innovative components-hyhiee buffer-can can be employed independently, depending on the specific algorithm's 's requirements. This flexible framework empowers the training of sparse networks from the ground up, accommodating wide array of MARL algorithms.
TD(A), Soft Mellowmax operator, and dual
In the following, we delineate the essential steps of implementing MAST with QMIX (Algorithm) 2 The steps for WQMIX are nearly identical, with the exception unrestricted agent networks and the unrestricted mixing network's 's inclusion. Also, note that we follow the symbol definitions from
[Colom] 2021 Algorithm 2 and
'Note that WQMIX encompasses two distinct instantiations, namely Optimistically meaiittccall--Weigheeed (OW) QMIX and Centrally-Weighted (CW) QMIX. In this paper, we specifically focus OWQMIX.
18

---

Gradient-based Topology Evolution: The process of topology evolution is executed within Lines in Algorithm 2 Specifically, the topology evolution update occurs intervals of A. timesteps. For comprehensive understanding of additional hyperparameters pertaining to topology evolution, please refer to the definitions provided Algorithm
TD Targets:
Hybrid TD(A) with Soft Mellowmax operator is computed in the Line 25 in Algo-
rithm 2 which modify the TD target y as follows:
(()
if <
To-
=
(1 (1- E X" A""1T(")
Otherwise.
(6)
Algorithm 2 MAST-QMIX
1: Initialize sparse agent networks, mixing network and hyperneworr with random parameters and random masks Mg with determined sparsity S
2: 0 <- @ Mg // Start with random sparse network
3: Initialize target networks 0 + 0
4: Set the learning rate to a
5: Initialize the replay buffer B1 + 0 with large capacity C1 and B2 + 0 with small capacity C2
6: Initialize training step +- 0
7: while step < Taa do
8: 0-0
9: t initial state
10: while 8t * terminal and episode limit do
11:
for each agent do
7i + + U ((o, =-11}
13:
e --csilon-schcculllstty
aargmax.. Q (rf' ug)
with probability -
14:
uf randint(1,[UU]
// e-greedy exploration
with probability
15:
end for
16:
Get reward and next state St-1
Bi B, [St, up Tt, st+1) // Data in the buffer is of episodes form.
18:
B2 + B2 U (st, ue (t.11)
<<t 1.step < step +1
20:
end while
if |B > batch-size then
22:
b - random batch of episodes from 33 and B2 // Sample from dual buffers.
23:
for each timestep in each episode batch b do
24:
Qiot +- Mixing-network (1Q) u/), QN,T", " Hypereeworr((((::)
25:
Compute TD target y according to Eq. 6 TD(A) targets with Seft Mellowmax
operator:
26:
end for
27:
AQ
y Qrat
28:
AO V 08 (AAro))
29:
Ó -
aA0
30:
end if
31:
if step mod A. = 0 then
Topology Evolution(networks; by Algorithm I
33:
end if
if step mod
where the target network update interval then i 8 // Update target network.
0- - 0- ④ Mg
37:
end if
38: end while
19

---

Algorithm 3 MAST-(OWJQMIX
1: Initialize sparse agent networks, mixing network and hypernetwork with random parameters and random masks M2 with determined sparsity S.
2: Initialize unrestricted agent networks and unrestricted mixing network with random parameters
@ and random masks Ms with determined sparsity S.
3: 0 <- A ④ Ma à t Ø ④ M // Start with a random sparse network
4: Initialize target networks i - 0.0
5: Set the learning to rate a
6: Initialize the replay buffer B1 - 0 with large capacity C1 and B2 t 0 with small capacity C2
7: Initialize training step f- 0
8: while step < max do
9:
5,0,
10:
initial state while 8t * terminal and episode limit do
11:
12:
for each agent do
13:
7f + - U (
14:
< eessllon ppsilon-schccuule(step))
uf argmax,,, Q(T u²:0)
with probability 1 with probability
15:
// e-greedy exploration
randint(L,|Ui]
end for
17:
Get reward Ty and next state St-1
Bi B, [St, Up Tt, St+1) // Data in the buffer is of episodes form.
19:
B2 + B, U (st, ur Tt, 8t+1)}
t +-t+1, step step +
21:
end while
22:
if [B, > batch-size then
23:
b + random batch of episodes from B1 and B2 // Sample from dual buffers.
24:
for each timestep in each episode batch b do
25:
Qrot Mixing-network (1Q)(T): 0): .... Qu(Ti" (Z:06) Hypernetwork(s::)) ( 6))
26:
Unrestricted Unrestricted-Mixing--
(Qi (T. ut: (), Qn(F, ut; 0);
27:
Compute TD target with target Unrestricted-Mixing network according to Eq. (6)
TD()) targets with Soft Mellowmax operator.
w(s,, ue) + {1: Qiot < a, otherwise.
28:
29:
end for
30:
AQ y Qiot
31:
AO + V 07 Σ w(s, u) {AQ
32:
δ d-b 8 aAd
33:
AQ y y-Q Q²-
34:
Aò v =(
35:
3 + -- add
36:
end if
37:
if step mod Am. 0 then
38:
Topology [_o__luuulii((enworrsss)) and Topology Baalooyy_EEvluuiionnnttttts by Algorithm
39:
end if
40:
if step mod 0, where is the target network update interval then
41:
0
0, à à
i
0 2 Mg. δ- -- 3-GM, 3
end if
44: end while
20

---

Here, A
[0, is a hyperparameter, and Te
Eist tr
yat (smm (QI(TI,),....... (QN(Q(((N)))) where fo denotes the mixing network and Q: is the target network of Q; The loss function of MAST, (0),,, is defined as:
()) = "(euun.")-BiBB (us - Qtor(s, u))²
(7)
Dual Buffers: With the creation of buffers B and 22, the gradient update with data sampled from dual buffers is performed in Lines 21 30 in Algorithm
A.6 Limitations of MAST
This paper introduces MAST, novel framework for sparse training in deep MARL, leveraging gradient-based topology evolution to explore network configurations efficiently. However, under- standing its limitations crucial for guiding future research efforts.
Hyperparameters: MAST relies on multiple hyperparameters for its key components: topology evolution, TD(A) targets with Soft Mellowmax Operator, and dual buffers. Future work could explore methods to automatically determine these hyperparameters or streamline the sparse training process with fewer tunable settings.
Implementation: While MAST achieves efficient MARL agent training with minimal performance trade-offs using ultra-sparse networks surpassing 90% sparsity, its current of unstructured sparsity poses challenges for running acceleration. The theoretical reduction FLOPs might not directly translate reduced running time. Future research should aim to implement MAST in structured sparsity pattern to bridge this gap between theoretical efficiency and practical implementation.
B Experimental Details
In this section, offer comprehensive experimental insights, encompassing hardware configurations, environment specifications, hyperparameter settings, model size computations, FLOPs calculations, and supplementary experimental findings.
B.1 Hardware Setup
Our experiments are implemented with PyTorch 2.0.0 Paszke et al. 2017 and run on 4x NVIDIA GTX Titan X (Pascal) GPUs. Each needs about 12 ~ 24 hours for QMIX WQMIX, and about 24 ~ 72 hours for RES for two million steps. depends on the environment types.
B.2 Environment
We assess the performance of MAST framework using the SMAC benchmark (Samvelyan et al.) 2019]. dedicated platform for collaborative multi-agent reinforcement learning research based on Blizzard's StarCraft II real-time strategy game, specifically version 4.10. It is important to note that performance may vary across different versions. Our experimental evaluation encompasses four distinct maps, each of which is described in detail below.
3m: An easy map, where the agents are 3 Marines, and the enemiesa are 3 Marines.
2s3z: An easy map, where the agents are 2 Stalkers and 3 Zealots, and the enemies are 2 Stalkers and Zealots.
3s5z: An easy map, where the agents are 3 Stalkers and 5 Zealots, and the enemies are 3 Stalkers and Zealots.
2c_vs_64zg: hard map, where the agents are 2 Colossi, and the enemies 64 Zerglings.
We also evaluate MAST on the Multi-Agent MuJoCo (MAMuJoCo) benchmark from [Peng et al.) 2021], which is an environment designed for evaluating continuous MARL algorithms, focusing on cooperative robotic control tasks. It extends the single-agent MuloCo framework included with OpenAl Gym [Brockman al. 2016). Inspired by modular robotics, MAMuJoCo includes scenarios with large number of agents, aiming stimulate progress in continuous MARL by providing diverse
21

---

and challenging tasks for decentralized coordination. We test MAST-COMIX in the Humanoid, Humanoid Standup, and ManyAgent Swimmer scenarios in MAMuJoCo. The environments are tested using their default configurations, with other settings following FACMAC (Peng et al. 2021]. Specifically, we set the maximum observation distance to = 0. the ManyAgent Swimmer scenario, we configure 10 agents, each controlling a consecutive segment of length 2. The agent network architecture of MAST-COMIX uses an MLP with two hidden layers 400 dimensions each, following the settings in FACMAC. All other hyperparameters are also based on FACMAC.
B.3 Hyperparameter Settings
Table provides a comprehensive overview of the hyperparameters employed in our experiments for MAST-QMIX, MAST-WQMIH and MAST-RES. It includes detailed specifications for network parameters, RI. parameters, and topology evolution parameters, allowing for a thorough understanding of our configurations. Besides, MAST is implemented based the PyMARL [Samvelyan et al. 2019] framework with the same network structures and hyperparameters as given in Table 3 We also provide hyperparameter recommendation for three key components, i.e. gradient-based topology evolution, Soft Mellowmax enabled hybrid TD(A) targets and dual buffers, in Table for deployment MAST framework in other problems.
Table 2: Recommendation for Key Hyperparameters in MAST.
Category
Hyperparameter
Value
Topology Evolution
Initial mask update fraction Co
0.5
Mask update interval Am
200 episodes
Burn-in time To
3/8 of total training steps
À value in TD(A)
0.6 or 0.8
TD Targets
a in soft mellow-max operator w soft mellow-max operator
10
Offline buffer size C1
5 x 103 episodes
Dual Buffer
Online buffer size C2
128 episodes
Sample partition
2:6
of online and offline buffer
Besides, to extend the existing QMIX to continuous action spaces, we utilized COMIX from FACMAC (Peng al., [021] for our experiments, which employs the Cross-Entropy Method (CEM) for approximate greedy action selection. The hyperparameter configuration of CEM also follows FACMAC settings.
B.4
Calculation of Model Sizes and FLOPs
B.4.1
Model Size
First, we delineate the calculation of model sizes, which refers to the total number of parameters within the model.
For a sparse network with L fully-connected layers, the model size, as expressed in prior works (Evei al. 2020 Tan et al., 2022), can be computed using the equation:
1. Minear
(1-S))400
(8)
1=1
where S; represents the sparsity, I is the input dimensionality, and O is the output dimen- sionality of the 1-th layer.
For sparse network with L GRU layers, considering the presence of gates in single layer, the model size can be determined using the equation:
MGRU 11- x 3 b x (fr + 4))
(9)
1=1
where hy represents the hidden state dimensionality.
22

---

Table 3: Hyperparameters of MAST-QMIX, MAST-WQMIX and MAST-RES.
Category
Hyperparameter
Value
Optimizer
RMSProp
Learning rate a
x 10
Discount factor Y Number of hidden units
0.99
64
per layer of agent network
Hidden dimensions in the
64
GRU layer of agent network Embedded dimensions of mixing network
32
Shared Hyperparameters
Hypernet layers of mixing network Embedded dimensions of hypernetwork Activation Function
64
ReLU
Batch size B
32 episodes
Warmup steps
50000
Initial e
1.0
Final
0.05
Double DQN update
True
Target network update interval I
200 episodes
Initial mask update fraction Ço
0.5
Mask update interval Am
timesteps of 200 episodes
Offline buffer size Ci
x 10 episodes
Online buffer size C2
128 episodes
Burn-in time To
7.5 x 10a
@: in soft mellow-max operator
w in soft mellow-man operator
10
Number of episodes in a sampled batch
20
of offline buffer S
Number of episodes in a sampled batch of online buffer S₂
12
Hyperparameters for MÁST-QMIX
Linearly annealing steps for
50k
value in TD(A)
0.8
Linearly annealing steps for e
100k
X value TD(X)
0.6
Coefficient of Qtat loss
Hyperparameters for MAST-WQMIX
Coefficient of Q* loss Embedded dimensions of unrestricted mixing network Embedded number of actions of unrestricted agent network a in weighting function
256
0.1
Linearly annealing steps for e
50k
Hyperparameters for MAST-RES
À value TD(X)
0.8
À value in Sormax operator
0.05
Inverse temperature |
5.0
23

---

Specifically, the "Total Size" column in Table within the manuscript encompasses the model size, including both agent and mixing networks during training. For QMIX, WQMIX, and RES, target networks are employed as target agent networks and target mixing networks. We denote the model sizes of the agent network, mixing network, unrestricted agent network, and unrestricted mixing network as M Agent. Mixx M UUnnestrictedAAaen and MUnsstrrccod-Miix respectively. Detailed calculations of these model sizes are provided in the second column of Table
Table 4: FLOPs and model size for MAST-QMIX, MAST-WQMIX and MAST-RES.
Training FLOPs
Inference FLOPs
Algorithm
Model size
MAST-QMIX
A.M + 2MMsx
(B(FO + FLOPSMI
FLOP
M Magent + MMix+ + UMMustrrceud--ggal 2M Lessssiccceed-Mi
FFLSSS AAIIFLOSSAgee Agent + FLOPSMiX) +
MAST-WQMIX
4B FFOP LLOPPsursriitte--ppee
FLOPSAgem
4B 4B-FLOPs Unretriidiis
4B 4.3FSOPS. FLOPPAgeem AAPen +
MAST-RES
2. + 2.MMix
FLOPS
(5 + nm) FLOPSMiX
B.4.2 FLOPs Calculation
Initially, for a sparse network with L fully-connected layers, the required FLOPSs for a forward pass computed as follows (also adopted in [Evci et al. 2020) and [Tan et al. 2022
FLOPs Σ (1-5)) -)) 1)0,,
(10)
where S/ is the sparsity, I is the input dimensionality, and O₂ is the output dimensionality of the 1-th layer. Similarly, for a sparse network with L GRU [Chung et al. 2014) layers, considering the presence of 3 gates in a single layer, the required FLOPs for forward pass are:
FLOPs = Σa (1- x3 hu x [2 + I1) 1].
(1I)
1=1
where hy is the hidden state dimensionality.
We denote B as the batch size employed in the training process, and FL.OPS and FLOPSMi as the FLOPs required for forward pass in the agent and mixing networks, respectively. The inference FLOPs correspond exactly to FLOPs detailed in the last column of Table 4 When it comes to training FLOPs, the calculation encompasses multiple forward and backward passes across various networks, which will be thoroughly elucidated later. Specifically, we compute the FLOPs necessary for each training iteration. Additionally, we omit the FLOPs associated with the following processes, as they exert minimal influence on the ultimate result:
Interaction with the environment: This operation, where agents decide actions for interac- tion with the environment, incurs FLOPs equivalent to FLOPsAgem- Notably, this value is considerably smaller than the FLOPs required for network updates, as evident in Table given that B >>> 1.
Updating target networks: Each parameter in the networks is updated as 0' +- 0. Conse- quently, the number of FLOPs in this step mirrors the model size, and is thus negligible.
Topology evolution: This element is executed every 200 gradient updates. To be precise, the average FLOPSs involved in topology evolution are computed as B x 2FLOPs for the (1-Sa))A
agent, and B x
2FLOPS for the mixer. Given that Am = 200, the FLOPs incurred by Stmn) (
topology evolution are negligible.
Therefore, our primary focus shifts the FLOPPs related updating the agent and mixer. We will first delve into the details for QMIX, with similar considerations for WQMIX and RES.
B.4.3 Training FLOPs Calculation in QMIX
Recall the way to update networks in QMIX is given by
0
a
1
Σ (y: Qua a;;;))))))
(12)
B
24

---

where B is the batch size. Subsequently, we can compute the FLOPs of training as:
FLOP Strain
FLOPSTD
larget FLOPs Scompule Joss. Stackwardd
(13)
where FLOPSTDD larget.
Jons- computing the TD targets in forward pass, loss function forward pass, and gradients in backward pass (backward propaaattoo,,,, respectively. By Eq. 6 and 7 we have:
FLOPS Scompute Joss and FLOP Shackward pass refer to the numbers of FLOPs in
FLOPSTD_ERRE target B x (FLOPSAgen) Agent + FLOPSMiX):
(14)
FLOPs FLOPScompute Scompute Joss B x (FLOPS Agent + FLOPSMix).
For the FLOPs of gradients backward propagation, FLOP Shackward pass compute two times the computational expense of the forward pass, which is adopted in existing literature [Evci et al. 2020 i.e.,
FL.OP Spackward, pass
B x 2 (FLOPSAges FLOPSMix),
(15)
Combining Eq. Eq. 14 and Eq. 15 the FLOPs of training in QMIX is:
FLOPS Strain
B x 4 x (FLOPSAgen SAgent + FLOPSMix).
(16)
B.4.4
Training FLOPs Calculation in WQMIX
The way to update the networks in WQMIX is different from that in QMIX. Specifically, denote the parameters of the original network and unrestricted network as 0 and o, respectively, which are updated according
0 8 0
ao B
Σ
w(s; aTx - (sor 0;;))
(17)
① - aV,
1 B
E (s,,
where B is the batch size, w is the weighting function, Q- is the unrestricted joint action value function. As shown in Algorithm the way compute TD target in WQMIX is different from that in QMIX. Thus, we have
FLOPSTD PLOPeeDDtarget target
B (FLOPS
FLOP SUneesriccee--Mi
(18)
In this paper, we take an experiment on one of two instantiations of QMIX. i.e., OW-QMIX RRahid al. 2020a] Thus, the number of FLOPS in computing loss is
FLOPs ccompute FLOP5coompntc loss = B x (FLOPS Agent + FLOPPMix + FLOPP UnrestiiteddAAgnn + FLOP FLOPS Unnesrriceed-M//
(19)
where unrestricted-agent and unrestricted-nin have similar network architectures as Qror and Qioe to,
respevtively. The FLOPs of gradients backward propagation can be given as
FLOP FLOPspackward Sbackward pass
B x x (FLOP Aggett FLOPSMi +FLOFFFOSS Unrestrrrudd + FLOPs LLOnstnnntiiitt--MM )
(20)
Thus, the FLOPs of training in WQMIX can be computed by
FLOPStrain Bx (3FLOPSAgent -3FLO 4LLL
(UnFLLOOPUnssriiessM-k))))) (21)
B.4.5 Training FLOPs Calculation in RES
Calculations of FLOPs for RES are similar to those in QMIX. The way to update the network parameter RES is:
- 0 a B (Qoassulss
(22)
where B is the batch size. Meanwhile, note that the way to compute TD target in RES Pan et al. 2021 includes computing the approximate Softmax operator, have:
FLOPS = B x (FLOPSAgem + n x m x (2FLOPsMx)).
(23)
where is the number of agents, is the maximum number of actions agent take in scenario. Other terms for updating networks are the same as QMIX. Thus, the FLOPs of training in RES can be computed by
FLOPStran = B x (4FLOPs Agent + (3+ 2 x n. X m) FLOPSMiX)
(24)
25

---

B.5 Training Curves of Comparative Evaluation in Section 4.1
Figure 13 Figure 14| and Figure 13 show the training curves of different algorithms in four SMAC environments. The performance is calculated as the average win rate per episode over the last 20 evaluations of the training. MAST outperforms bascline algorithms all four environments in all three algorithms. We smooth the training curve by 1-D filter by scipy. scipy.signal. savgol_filte in Python with window window_lengthe 1ength=21 and polyorder=2.
These figures unequivocally illustrate MAST's substantial performance superiority over baseline methods in all four environments across the three algorithms. Notably, static sparse (SS) consistently exhibit the lowest performance on average, highlighting the difficulty of finding optimal sparse network topologies in the context of sparse MARL models. Dynamic sparse training methods, namely SET and RigL, slightly outperform (SS), although their performance remains unsatisfactory. Sparse networks also, average, underperform tiny dense networks. However, MAST significantly outpaces all other baselines, indicating the successful realization of accurate value estimation through our MAST method, which effectively guides gradient-batt based topology evolution. Notably, the single-agent method RLx2 consistently delivers subpar results all experiments, potentially due to the sensitivity of the step length in the multi-step targets, and the failure of dynamic buffer for episode-form training samples.
0.
MAST
MAST
3.50
(a) MAST-QMIX 95% on 3m
(b) MAST-QMIX 95% 283z
527
MAST
24
MAST
2.4
3.2
(c) MAST-QMIX 90% 3a5z
(d) MAST-QMIX 90% 642g
Figure 13: Training processes of MAST- QMIX on four SAMC benchmarks.
26

---

Cenve
10.
0.4
6.2
6.00
(a) WQQIX 90% on
(b) MAST-WQMIX 90% on 2s32
2.0
Rgg.
FLC
0.33
025
850
375
3.26
2.00
1.75
200
8:00
325
3356
0.75
1200
1.5
200
(c) MAST- 90% on 3a5z
(d) MAST-WQMIX 90% on 64zg
Figure 14: Training processes of MAST-WQMIX on four SAMC benchmarks.
FL22
MAST
MAST
64
0.4
6.2
60
(a) MAST-RES 95% 3m
(b) MAST-RES 90% 2a3z
3.0
Danae
WAST
T00 001 120
MAST
02
32
(c) MAST-RES 85% 3s52
(d) MAST-RES 85% on 642g
Figure 15: Training processes of MAST-RES on four SAMC benchmarks.
B.6 Standard Deviations Results in Table
Table 5 showcases algorithm performance across four SMAC environments along with their cor- responding standard deviations. It's important to note that the data in Table is not normalized concerning the dense model. Notably, MAST's utilization of topology evolution doesn't yield in- creased variance results, demonstrating consistent performance across multiple random seeds.
27

---

Table 5: Results in Table ! with standard deviations
Alg.
Env.
Tiny(%)
SS(%)
SET(%)
RigL(%)
RLx2(%)
Ours(%)
3m
96.3++4.3
89.8+7.9
94.1+-5.9
93.4++9.5
11.9+
98.9++2.00
2s3z
10.8
70.4=13.1
74.91:6..
67.0=14.0
44.2:11.00
94.611.66
Q-
3s5z
61.2:+11.8
32.0+220.3
19.3+16.6
42.6+19.2
47.2+16.2
93.3+5.1
MIX
64*
24.01:29.9
37.3123.4
62.3421.9
45.2113.4
9.2.15.00
90.611.8
Avg.
13.8+14.7
57.4+16.2
70.1
62.0+16.5
28.1+17.2
94.3++499
3m
97.0+4.0
95.6+4.00
96.5+3.6
96.513.6
96.7+443
97.3++4.0
2s3z
86.0+77.9
12.4+12.4
82.5+10.9
83.3+10.3
83.8+9.9
96.2+t4.2
WQ-
3s5z
64.5+117.9
57.0++1..5
51.1+15.0
46.0+20.55
55.4111..
87.6
MIX
64*
43.8+27.4
25.4222.0
37,8+26.2
35.2
45.3+24.7
84.4
Avg.
68.5+13.5
62.2+13.0
64.0++1..5
65.8+11.4
70.3+12.5
91.4+559
3m
96.9+-4.1
94.7+4.8
9..4
90.3+7.4
97.0++3.8
102.2++3.2
2s3z.
95.8+3.8
92.215.9
92.2:5.5
94.0=5.7
93.455.5
97.7+2.6
RES
3s5z
92.2++.88
86.348.8
87.6+5599
90.0+7.3
83.6+9.2
96.4++344
64*
73.5+25.8
34.5429.6
38.9433.3
31.1436.2
64.1133.8
92.514.9
Avg.
89.6+
76.9+12.3
78.8+
76.3+14.1
84.5+H1.11
97.2+
Avg.
78.7+12.8
65.6113.9
72.0+11.77
67.8114.5
61.0+114.3
94.2
B.7 Ablation Study
We conduct a comprehensive ablation study on three critical elements of MAST: hybrid TD(A) targets, the Soft Mellowmax operator, and dual buffers, specifically evaluating their effeets QMIX and WQMIX. Notably, since MAST-QMIX shares similaritics with MAST-RES, our experiments focus on QMIX and WQMIX within the 3s5z task. This meticulous analysis seeks to elucidate the influence of each component on MAST and their robustness in the face of hyperparameter variations. The reported results expressed percentages and are normalized to dense models.
Hybrid TD(A) We commence our analysis by evaluating different burn-in time To in hybrid TD(A) in Table 6 Additionally, we explore the impact of different values within hybrid TD(A) in Table 7 These results reveal hybrid TD(A) targets achieve optimal performance with a burn-in time of To 0. 75M and performance improvements in WQMIX, while their impact on QMIX is relatively modest.
0.6. It is noteworthy that hybrid TD(A) targets lead significant
Table 6: Ablation study burn- in time To in hybrid TD(A).
Alg.
0.45M
0.75M
1M
1.5M
2M
QMIX RES
93.6
94.3
97.9
92.0
92.5
91.5
WQMIX
83.5
88.4
98.0
98.7
76.9
70.3
Avg.
88.5
91.3
97.9
95.4
84.7
80.9
Table Ablation study on À in hybrid TD(A).
Alg.
0
0.2
0.4
0.6
0.8
QMIX / RES
91.5
94.7
96.8
96.8
97.9
89.4
WQMIX
83.5
83.5
74.7
98.0
96.1
87.9
Avg.
87.5
89.1
85.7
97.4
97.0
88.6
Soft Mellowmax Operator The Soft Mellowmax operator in Eq.
introduces two hyperparam-
eters, and w. A comprehensive examination of various parameter configurations is presented
28

---

in Table 8 showing that the performance of MAST exhibits robustness to changes in the two hyperparameters associated with the Soft Mellowmax operator.
Additionally, is worth noting that the Softmax operator is also employed in (Pan al. [021] mitigate overestimation in multi-agent Q learning. To examine the effectiveness of various operators, including max, Softmax, Mellowmax, and Soft Mellowmax, we conduct comparative analysis in Fig- I6 Our findings indicate that the Soft Mellowmax operator surpasses all other baselines in alleviating overestimation, Al- though the Softmax operator demonstrates similar performance to the Soft Mellowmax operator, it should be noted that the Softmax operator entails higher computational costs.
Riy.-geix Mox
Fig. Mol-3
Rig. SKXX Sollman
Rig. SMXX Soll SollWellommoo
0.6
Milonntups
Figure 16: Comparison different operators.
Table 8: Ablation study the Soft Mellowmax operator.
1
10
10
Alg.
w = 10
w =
w
10
w = 5
w
10
QMIX / RES
97.9
100.0
98.9
96.8
97.9
WQMIX
98.0
92.3
87.9
92.3
85.7
Avg.
97.9
96.1
93.4
94.5
91.8
Dual Buffers each training step, we concurrently sample two batches from the two buffers, Bi and B2. We maintain fixed total batch size of 32 while varying the sample partitions b b2 within MAST. The results, detailed Table reveal that employing two buffers with a partition ratio of 6: yields the best performance. Additionally, performance when using data solely from a single buffer, whether it the online or offline buffer. This underscores the vital role of dual buffers in sparse MARL..
observed significant degradation in MAST's
Table 9; Ablation study dual buffers.
Alg.
8 0
6: 2
5 3
3: 5
2 6
0: 8
QMIX / RES
93.6
99.4
97.9
97.8
89.6
85.1
WQMIX
64.8
101.2
98.0
86.8
95.1
70.3
Avg.
79.2
100.3
97.9
92.3
92.4
77.7
B.8 Sensitivity Analysis for Hyperparameters
Table shoss the performance with different mask update intervals (denoted as Am) in different environments, which reveals several key observations:
Findings indicate that small Am negatively impacts performance, frequent mask adjustments may prematurely drop critical connections before their weights are adequately updated by the optimizer.
Overall, A moderate Am 200 episodes performs well in different algorithms.
Table 10: Sensitivity analysis on mask update interval.
A, 20
An 100 episodes
A. 200 episodes
Am 1000 episodes
Am. 2000 episodes
Alg.
episodes
QMIX/RES
99.4% 83.2%
97.7% 91.9%
99.0% 96.1%
100.6% 68.1% 1%
100.6% 71.5%
WQMIX
Avg.
91.3%
94.8%
97.5%
84.3%
86.0%
29

---

B.9 Experiments on QMIX in Multi-Agent MuJoCo
We compare MAST-QMIX with other baselines on the MAMuJoCo benchmark, and the results are presented in Figure 17 From the figure, we observe that MAST consistently achieves dense performance and outperforms other methods in three environments, except for ManyAgent Swimmer, where the static sparse network performs similarly to MAST. This similarity may be attributed the simplicity of the ManyAgent Swimmer environment, which allows random static sparse network to achieve good performance. This comparison demonstrates the applicability of MAST across different environments.
COMEE DeEGE DORIESS
COMOSDETTR
COMISG DEED
COMBLES
SOMOSSET
COVELMAST
COMPSMAST
-12
290
(a) 2-Agent Humanoid
(c) ManyAgent Swimmer Figure 17: Mean episode return different MAMuJoCo tasks with MAST-QMIX.
(b) 2-Agent HumanoidStanduu
B.10 Experiments on FACMAC in SMAC
In addition to pure value-based deep MARL algorithms, we also evaluate MAST with hybrid value- based and policy-based algorithm, FACMAC (Peng al. 2021], in SMAC. The results presented in Figure 18 From the figure, we observe that MAST consistently achieves dense performance and outperforms other methods in three environments, demonstrating its applicability across different algorithms.
3.0
FRCMACISS
BRCMACSS
FRCMAC-SET SET
ISCMAC-0 SET
NCMAA SET
ENCMAC Aay
Rist
0.2
WIl sups
10 101 S4pa
Rii 24450
bane_ _s_ bane Figure 18: Mean episode win rates on different SMAC tasks with MAST-FACMAC.
MMM
(b) _c_vs_64p_ vs_64zg
B.11 Visualization of Sparse Masks
We present series of visualizations capturing the evolution of masks within network layers during the MAST-QMIX training in the 3s5z scenario. These figures, specifically Figure 20 (a detailed view of Figure TT Figure 21 and Figure 22 offer intriguing insights. Additionally, we provide connection counts in Figure 23 and 25 for input and output dimensions in each sparse mask, highlighting pruned dimensions.
facilitate a clearer perspective on connection distributions, we sort dimensions based on the descending order of nonzero connections, focusing the distribution rather than specific dimension ordering. The connection counts associated with Figure TT the main paper given in Figure 19
30

---

Serzed Ingut dmmrstonn
Serted Insut dirrersion
Serted mimriioo
Sorted rnput dimersiinn
(a) T 0M (Input)
(b) T 0.5M (Input)
(c) T 1M (Input)
(d)
2M (Input)
35
10
10
(c) T OM (Output)
(f) T 0.5M (Output)
(g) T 1M (Output)
(h) T 2M (Output)
Figure 19: Number of nonzero connections for input and output dimensions in descending order of the sparse layer visualized Figure
During the initial phases of training, noticeable shift in the mask configuration becomes evident, signifying dynamic restructuring process. As the training progresses, connections within the hidden layers gradually coalesce into subset of neurons. This intriguing phenomenon underscores the distinct roles assumed by individual neurons in the representation process, thereby accentuating the significant redundancy prevalent in dense models.
Figure 20 provides insights into the input layer, revealing that certain output dimensions can be omitted while preserving the necessity of cach input dimension.
Figure 21
showcases analogous observations, reinforcing the idea that only subset of output neurons is indispensable, even within the hidden layer of the GRU.
Figure 22 presents distinct findings, shedding light the potential redundancy of certain input dimensions in learning the hyperparameters within the hypernetwork.
31

---

T GM
0.5M
Figure 20: The learned mask of the input layer weight of Q1. Light pixels in row i and column indicate the existence of the connection for input dimension j and output dimension i, while the dark pixel represents the empty connection.
T = 0M
T 0.5M
T= 1M
T= 2M
Figure 21: The learned mask of the GRU layer weight of Q1. Light pixels in row i and column indicate the existence of the connection for input dimension j and output dimension i, while the dark pixel represents the empty connection.
32

---

OM
0.5M
1M
The learned mask of the first layer weight of Hypemeework. Light pixels in row and column/ j indicate the existence of the connection for input dimension j and output dimension while the dark pixel represents the empty connection.
Figure 22:
33

---

18
120
Serzed ingut dmetsion
Seried insut dimeesiion
Sorted Irput meroion
Sonted rppt dmersiin
(a) T 0M (Input)
(b) T 0.5M (Input)
(c) T 1M (Input)
(d)
2M (Input)
48
out
supput
(e) T OM (Output)
(f) 0.5M (Output)
(g)
1M (Output)
(h) T 2M (Output)
Figure 23: Number of nonzero connections for input and output dimensions in descending order of the sparse layer visualized in Figure 20
Sarred dimesston
26
52
20 40 (b) T 0.5M (Input)
20
50
30 Sormed 4mmrsion
23 26
20
Ameriion
46
50
(a) T = OM (Input)
(c) T 1M (Input)
(d) T 2M (Input)
10
10
11s
Sarted euper mrentinn
Softed
Sorted eutpur mencin
autput dimescon
(e) T 0M (Output)
(f) 0.5M (Output)
(g)
1M (Output)
(h) T 2M (Output)
Figure 24: Number of nonzero connections for input and output dimensions in descending order of the sparse layer visualized in Figure 21
8
Sortel moat cinamaiion
40 Sented diress
Sunted tost dimeraat
Saied inpu deveraao
(a) T 0M
(b) T 0.5M
(c) T 1M
(d) 2M
Noowe conter mnt 12 R
NOVDe'S COD nen une 5 E
LS 24 (c) T
Saited 220 DS
40
Sinted
223 250
0M
( T 0.5M
(g)
T
1M
(h) T 2M
Figure 25: Number of nonzero connections for input and output dimensions in descending order of the sparse layer visualized in Figure 22
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly state the claims that training multi- agent reinforcement learning involves heavy computational load, and our proposed MAST framework can save up to 20 times FLOPs without performance degradation less than 3% for value-based deep MARL algorithms.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Our paper explores some shortcomings of the MAST framework, such as the multiple hyperparameters. Please refer to Appendix A.6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
35

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All our theoretical results are accompanied by clear, complete and correct proofs. All the assumptions present are clearly stated or referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We ensure that the configurations and details of each experimental result are clearly explained, and all experimental results are reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
36

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code will be open-sourced upon publication of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
public/guides/ http:///oooetumiiiiiiiiieeie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips. fttaps.cccpublicee wwtppbblccgguueeeeeoooooocpppped
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We ensure that the parameter configurations and details of each experimental result are clearly explained, such as the network architecture and training hyperparameters. Please refer to B.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For each experiment, we presented the results of eight random seeds and calculated their mean and standard deviation. Pleasr refer to B.6
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
37

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided the necessary computational resources, hardware setup, and duration required reproduce the experiments. Please refer B.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics Ethics/https:
Answer: [Yes]
Justification:
We ensure compliance with all aspects of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed
Guidelines:
The answer NA means that there is no societal impact of the work performed.
38

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly credited assets used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
39

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provided a README file alongside our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
40

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "PFh0t9FUSL", "reviewer": "Reviewer_fbhH", "summary": "This paper introduces dynamic sparse training (DST) to the Deep Multi-Agent Reinforcement Learning (MARL) settings for the first time in the literature. Furthermore, it shows that applying directly  DST algorithms to MARL does not lead to optimal results. Consequently, it proposes a new framework named Multi-Agent Sparse Training (MAST) which enhances the DST RigL algorithm with a hybrid TD-(λ) schema and a dual replay buffer mechanism in order to successfully cope with the challenging MARL settings. An extensive empirical validation is performed showing that MAST can reduce up to 20x the computational requirements at virtually no loss in performance.", "questions": "Q1) While the theoretical reduction in terms of computational resources is impressive, can you comment on the real wall-clock running time? I know that this is not possible when simulating sparsity with binary masks, but have you considered using some truly sparse implementation of the neural networks? While I am not sure how easy would be to do this for the GRU layer, there exists some sparse MLP implementations for supervised learning (e.g., Curci et al., Truly Sparse Neural Networks at Scale, arXiv:2102.01732, 2021) which may be easy to adapt to the MAST framework. This may allow you to design an experiment where you can scale up (in terms of the number of neurons) seriously the neural network for very large state or action spaces which as you mentioned is a typical challenge in MARL.\n\nQ2) (minor) I suggest adopting a uniform citation style in order to improve related work chronological readability. Currently, some of the references are cited using the year of the first preprint release on arXiv, while others are cited using the official publication year.", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This is an original paper which introduces for the first time DST to MARL.\n* The paper solves the inherent problems and the suboptimal behavior of directly applying DST to MARL by proposing a new framework MAST which is specially designed for MARL.\n* The paper is clear and well written. The source code is provided for easy reproducibility.\n* The extensive empirical validation shows the superiority of the proposed framework in comparison with the most common sense baselines as there is no other DST method specially designed for MARL.\n* It is likely that the paper to have a fair impact on the sparse training and multi-agent reinforcement learning communities.", "weaknesses": "* Up to my best understanding, there seems to be no striking weak points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and effort in reviewing our paper! Please find our responses to your comments below. We will be happy to answer any further questions you may have.\n\n---\n\n### Questions\n\n> **Q1**:  While the theoretical reduction in terms of computational resources is impressive, can you comment on the real wall-clock running time? I know that this is not possible when simulating sparsity with binary masks, but have you considered using some truly sparse implementation of the neural networks? While I am not sure how easy would be to do this for the GRU layer, there exists some sparse MLP implementations for supervised learning (e.g., Curci et al., Truly Sparse Neural Networks at Scale, arXiv:2102.01732, 2021) which may be easy to adapt to the MAST framework. This may allow you to design an experiment where you can scale up (in terms of the number of neurons) seriously the neural network for very large state or action spaces which as you mentioned is a typical challenge in MARL.\n\nOur primary objective is to exploit the computational redundancy in training MARL agents. Thus, we design the MAST framework to achieve this goal and aid various MARL algorithms specifically in sparse training scenarios. Existing works also focus on algorithmic FLOP reduction, such as (Sokar et al., 2022; Graesser et al., 2022; Tan et al., 2023). Our work, alongside these contributions, contributes to paving the way for more efficient and scalable multi-agent systems.\n\nAs sparse methods evolve in tandem with hardware co-design, the anticipated translation of FLOP reduction into wall-clock speedups becomes increasingly viable. Recent developments, such as specialized software kernels and dedicated hardware solutions, e.g., DeepSparse (NeuralMagic, 2021) and Cerebras CS-2 (Lie et al., 2022), as well as (Curci et al., 2021), signify promising strides toward realizing the benefits of unstructured sparsity during both training and inference stages. We believe this will be a very interesting future direction\n\n```\n(NeuralMagic et al., 2021) NeuralMagic. Deepsparse, 2021. URL https://github.com/neuralmagic/deepsparse.\n(Lie et al., 2022) Lie, S. Harnessing the Power of Sparsity for Large GPT AI Models. https://www.cerebras.net/blog/harnessing-the-power-of-sparsity-forlarge-gpt-ai-models, 2022.\n```\n\n> **Q2**: (minor) I suggest adopting a uniform citation style in order to improve related work chronological readability. Currently, some of the references are cited using the year of the first preprint release on arXiv, while others are cited using the official publication year.\n\nThank you for your suggestion. We will adopt a uniform citation style in our revision to improve the chronological readability of related works.\n\n---\nWe are grateful for your constructive suggestions, which have significantly guided our improvements. We hope our response addresses your concerns. If so, we would like to know if you could kindly consider raising your score rating. We will also be happy to answer any further questions you may have. Thank you very much!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bK4clLwyWc", "reviewer": "Reviewer_zAbe", "summary": "The paper presents a significant advancement in the field of MARL by introducing the MAST framework which aims at improving the Reliability of Training Targets and Improving the Rationality of Sample Distribution. Overall this paper is well-written and easy to follow, on a very interesting research direction with promising results.", "questions": "What kind of sparsification techniques are used in MAST? \nAlso are mixing networks sparsified?\nThe agent network in qmix is relatively very small, how is it possible to reach a 95% sparsification while maintaining a relatively high or even better results compared to the dense networks?", "limitations": "The authors did not address the limitations but I don't see a direct or potential negative societal impact from this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper did thorough research on finding the reasons with theoretical contribution and possible solutions for the poor performance of previous specification methods and introduced 2 novel designs to solve them.\n\nThe paper provides solid theoretical underpinnings for the proposed methods.\n\n\nThe experimental results look very promising.", "weaknesses": "An ablation study would be good to tell to how much extent the 2 designs are contributing to the performance improvement.\n\nThe overhead brought by the new design was not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in reviewing our paper! Please find our responses to your comments below. We will be happy to answer any further questions you may have.\n\n---\n### Weaknesses\n\n> **W1**: An ablation study would be good to tell to how much extent the 2 designs are contributing to the performance improvement.\n\nWe have provided an ablation study for our proposed techniques, including the Hybrid TD$(\\lambda)$ mechanism, Soft Mellowmax Operator, and Dual Buffers, in Appendix B.7. Our findings indicate that all three components contribute significantly to the overall performance improvement.\n\n> **W2**: The overhead brought by the new design was not discussed.\n\n- MAST incorporates novel TD targets with a dual buffer mechanism to train ultra-sparse MARL models. The designed TD targets can be efficiently computed by our algorithm, and the dual replay buffer mechanism does not introduce additional computational overhead compared to a same-size single buffer. Detailed FLOP calculations are provided in Appendix B.4.2.\n- Additionally, MAST uses gradient-based topology evolution to train sparse MARL agents exclusively. The sparse network topology evolves every 10,000 gradient update steps, making the overhead from this evolution negligible compared to the gradient updates.\n\nFollowing the reviewer's advice, we will include this discussion in our revision.\n\n### Questions\n\n> **Q1**: What kind of sparsification techniques are used in MAST? Also are mixing networks sparsified? The agent network in qmix is relatively very small, how is it possible to reach a 95% sparsification while maintaining a relatively high or even better results compared to the dense networks?\n\n- MAST employs the RigL method (Evci et al., 2020), which enhances the optimization of sparse neural networks by leveraging weight magnitude and gradient information to jointly optimize model parameters and connectivity. The mixing network is also sparsified to a specified degree, as illustrated in Fig. 3.\n- Sparse networks often have fewer parameters, which can make them easier to train under certain conditions. This can lead to improved performance compared to dense networks, as demonstrated in our experiments. Similar observations have also been reported in the literature (Evci et al., 2020; Tan et al., 2023).\n\n### Limitations\n> **L1**: The authors did not address the limitations.\n\nOur paper does address the limitations of the MAST framework, including the challenge of managing multiple hyperparameters. This discussion can be found in Appendix A.6.\n\n---\nWe are grateful for your constructive suggestions, which have significantly guided our improvements. We hope our response addresses your concerns. If so, we would like to know if you could kindly consider raising your score rating. We will also be happy to answer any further questions you may have. Thank you very much!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hILiCBSGjn", "reviewer": "Reviewer_8jXp", "summary": "The paper introduces the Multi-Agent Sparse Training (MAST) framework to address computational overhead in Multi-agent Reinforcement Learning (MARL) by enhancing value learning through the Soft Mellowmax Operator with a hybrid TD-(λ) schema and a dual replay buffer mechanism. MAST achieves significant reductions in computational redundancy with minimal performance degradation.", "questions": "- Is Figure 7 for illustration purposes (i.e. not using real data)? if so, I would suggest using real data to illustrate the idea and better show the significance of the issue.\n- How does this technique generalize to test environments other than StartCraft II?\n- How does the proposed approach compare to the single agent sparse training other than the RLx2 method?", "limitations": "More analysis of the method's limitations would be helpful.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The problem of applying sparsity training to MARL is a topic.\n- The paper is well-written and easy to understand.\n- Comprehensive evaluation and ablation results provide a good understanding of the method and design decision.", "weaknesses": "- This work focuses only on one benchmark (StarCraft II), applying it to other benchmarks can give us a better idea of the generalizability of the approach.\n- How does the proposed technique compare to single-agent RL sparse training work such as \"Sokar et al., 2022\". Are there existing methods that are already able to achieve high performance with the techniques proposed in this work? The authors also mentioned that \"Wang et al., 2019\" also prunes agent networks throughout training, so it would be good to compare also to this work to see the computation reduction and performance comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and effort in reviewing our paper! Please find our responses to your comments below. We will be happy to answer any further questions you may have.\n\n### Weaknesses\n\n> **W1**: This work focuses only on one benchmark (StarCraft II), applying it to other benchmarks can give us a better idea of the generalizability of the approach.\n\nWe validate the generalizability of MAST in two main ways:\n\n- **Other Benchmarks:** We conducted a comprehensive performance evaluation of MAST across various tasks in the SMAC benchmark. Additional experiments on the multi-agent MuJoCo (MAMuJoCo) benchmark (Peng et al., 2021) are also included in Appendix B.9.\n- **Other Algorithms:** MAST is designed as a versatile sparse training framework for value decomposition-based MARL algorithms. We integrate MAST with state-of-the-art value-based deep MARL algorithms, including QMIX, WQMI, and RES. Additionally, we apply MAST to a hybrid value-based and policy-based algorithm, FACMAC (Peng et al., 2021). Results are presented in Section 4 and Appendix B.\n\nThis comprehensive evaluation demonstrates the effectiveness and generalizability of MAST across different benchmarks and algorithms.\n\n> **W2**: How does the proposed technique compare to single-agent RL sparse training work such as \"Sokar et al., 2022\". Are there existing methods that are already able to achieve high performance with the techniques proposed in this work? The authors also mentioned that \"Wang et al., 2019\" also prunes agent networks throughout training, so it would be good to compare also to this work to see the computation reduction and performance comparison.\n\n- The single-agent RL sparse training work in (Sokar et al., 2022) uses SET (Mocanu et al., 2018) for topology evolution, but does not improve value learning under sparse models, resulting in low-sparsity RL models. In our experiments, the SET baseline can be viewed as the MARL version of that in (Sokar et al., 2022). As shown in Table 1, applying SET alone is insufficient to achieve high sparsity levels in MARL scenarios. We will annotate SET in Table 1 as (Sokar et al., 2022) in our revision.\n\n- The algorithm in (Wang et al., 2019) fails to maintain sparsity throughout training and only achieves a final model sparsity of 80%, which is lower than our results. Additionally, their experiments are limited to a two-agent environment, PredatorPrey-v2 in MuJoCo (Todorov et al., 2012). Therefore, we did not include a direct comparison with this work in our paper.\n\n### Questions\n\n> **Q1**: Is Figure 7 for illustration purposes (i.e. not using real data)? if so, I would suggest using real data to illustrate the idea and better show the significance of the issue.\n\nThank you for your suggestion. Figure 7 was originally intended for illustration purposes. We will update Figure 7 with real data in our revision to better demonstrate the significance of the issue. Additionally, the effectiveness of the dual buffer in improving training data distribution is validated in the ablation study in Appendix B.7.\n\n> **Q2**: How does this technique generalize to test environments other than StartCraft II?\n\nWe have conducted a comprehensive performance evaluation of MAST across various tasks in the SMAC benchmark. Additionally, we performed experiments on the multi-agent MuJoCo (MAMuJoCo) benchmark (Peng et al., 2021). Please refer to Appendix B.9 for details.\n\n> **Q3**: How does the proposed approach compare to the single agent sparse training other than the RLx2 method?\n\nWe have compared our algorithm with several single-agent sparse training methods, including SET, RigL, and RLx2. Specifically, SET and RigL were originally developed for deep supervised learning and later adapted for single-agent sparse training in DRL as shown in (Sokar et al., 2022) and (Graesser et al., 2022), respectively. Our results demonstrate that MAST significantly outperforms these baselines in the MARL setting. For detailed comparisons, please refer to Table 1 in Section 4.1.\n\n\n### Limitations\n> **L1**: More analysis of the method's limitations would be helpful.\n\nOur paper does address the limitations of the MAST framework, including the challenge of managing multiple hyperparameters. This discussion is provided in Appendix A.6.\n\n---\nWe are grateful for your constructive suggestions, which have significantly guided our improvements. We hope our response addresses your concerns. If so, we would like to know if you could kindly consider raising your score rating. We will also be happy to answer any further questions you may have. Thank you very much!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: A somewhat complex extension to multi-agent RL training, but thoroughly executed. I concur with the reviewers' consensus that this paper can be accepted: while the two more lukewarm reviewers did not explicitly respond to the rebuttal, my read of their concerns and the authors' response is that all major points are addressed when taking the (extensive) appendix into account.
Decision: Accept (poster)