Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning
Authors: Aleksandar Todorov
Juan Cardenas-Cartagena
Rafael F. Cunha
Marco Zullich
Matthia Sabatelli
Venue: Accepted by TMLR ()

Abstract: Plasticity loss, a diminishing capacity to adapt as training progresses, is a critical challenge in deep reinforcement learning. We examine this issue in multi-task reinforcement learning (MTRL), where higher representational flexibility is crucial for managing diverse and potentially conflicting task demands. We systematically explore how sparsification methods, particularly Gradual Magnitude Pruning (GMP) and Sparse Evolutionary Training (SET), enhance plasticity and consequently improve performance in MTRL agents. We evaluate these approaches across distinct MTRL architectures (shared backbone, Mixture of Experts, Mixture of Orthogonal Experts) on standardized MTRL benchmarks, comparing against dense baselines, and a comprehensive range of alternative plasticity-inducing or regularization methods. Our results demonstrate that both GMP and SET effectively mitigate key indicators of plasticity degradation, such as neuron dormancy and representational collapse. These plasticity improvements often correlate with enhanced multi-task performance, with sparse agents frequently outperforming dense counterparts and achieving competitive results against explicit plasticity interventions. Our findings offer insights into the interplay between plasticity, network sparsity, and MTRL designs, highlighting dynamic sparsification as a robust but context-sensitive tool for developing more adaptable MTRL systems.

Published in Transactions on Machine Learning Research (07/2025)
Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning
Aleksandar Todoroy
o.todorov. @@student. rug. ml
Juan Cardenas-Cartagenaa
j.d. caarderusscorlaggnaaaauu
Rafael F. Cunha!
r.f. r.f.cunha@rug.vm ml
Marco Zullich
m.zullich@rug..r ml
Matthia Sabatellit
m.sabalellirrug.t
University of Groningen, Groningen, The Netherlands
Reviewed on OpenReview: https: // openreview. net/ forum? id= 9L4223EfE9
Abstract
Plasticity loss, diminishing capacity to adapt training progresses, is a critical challenge in deep reinforcement learning. We examine this issue in multi-task reinforcement learning (MTRL), where higher representational flexibility is crucial for managing diverse and po- tentially conflicting task demands. We systematically explore how sparsification methods, particularly Gradual Magnitude Pruning (GMP) and Sparse Evolutionary Training (SET), enhance plasticity and consequently improve performance in MTRL agents. We evaluate these approaches across distinct MTRL architectures (shared backbone, Mixture of Experts, Mixture of Orthogonal Experts) on standardized MTRL benchmarks, comparing against dense baselines, and a comprehensive range of alternative plasticity intty-iiiinn or regulariza- tion methods. Our results demonstrate that both GMP and SET effectively mitigate key indicators plasticity degradation, such as neuron dormancy and representational collapse, These plasticity improvements often correlate with enhanced multi-task performance, with sparse agents frequently outperforming dense counterparts and achieving competitive results against explicit plasticity interventions. Our findings offer insights into the interplay between plasticity, network sparsity, and MTRL designs, highlighting dynamic sparsification as robust but context-sensitive tool for developing more adaptable MTRL systems.
1 Introduction
Although deep reinforcement learning (DRL) agents have demonstrated impressive results in various applica- tions (Levine et al., 2016; Silver et al., 2017; Bellemare et al., 2020; Mathieu et al., 2023), these achievements come with notable tradc-offs. Attaining statc-of-tho-art performance often relics on large-scale computational resources and heavily overparameterixed models (Botvinick et al., 2019; Glanois et al., 2022; Thompson et al., 2022), which may lead to agents that either generalize poorly (Kirk et al,, 2023) struggle to adapt to new tasks or data over time. The former issue is topic of interest within the transfer learning literature (Farebrother et al., 2020; Sabatelli & Geurts, 2021; Sasso et al., 2023; Zhu et al., 2023), while the latter is commonly referred to as plasticity loss (Nikishin et al., 2022; Lyle et al., 2023; Dohare et al., 2024). Plasticity loss manifests through several interconnected optimization pathologies: gradient interference leading to premature convergence (Lyle et al., 2024a), representational collapse, limiting the diversity of learned features
* Corresponding author

---

Published in Transactions on Machine Learning Research (07/2025)
(Moalla et al., 2024), and neuronal saturation or dormancy that reduces effective network capacity (Bjorck et al., 2021; Sokar et al., 2023). While these challenges have been primarily investigated within single-task RL, (Nikishin et al., 2023; Abbas et al., 2023; Klein et al., 2024; Nauman et al., 2024a; Dohare et al., 2024), in this paper, we study them under the lens of multi-task reinforcement learning (MTRL), where maintaining representational flexibility across diverse tasks with potentially conflicting demands is even more crucial (Teh et al., 2017; Sodhani et al., 2021; D'Eramo et al., 2024). It naturally follows that this increased need for dynamic adaptation can make MTRL agents especially vulnerable plasticity loss, as networks must simultaneously accommodate varied objectives without experiencing negative task interference (Liu et al., 2023). The necessity of determining which knowledge to share across tasks, and how to share it without harmful interference, further complicates the learning process (Devin et al., 2016; Sasso et al., 2023). Moreover, this challenge can be even further exacerbated by inefficient use of network capacity (Kumar el al., 2021), with significant portions of large networks becoming underutilized during training, ultimately hindering the acquisition of a universal policy capable of addressing multiple tasks concurrently. Recent work in neural network pruning offers a promising direction beyond mere compression, showing that sparse agents can match or even exceed dense counterparts in single-task RL (Livne & Cohen, 2020; Graesser et al., 2022; Obando-Ceron et al., 2024). Notably, methods like Gradual Magnitude Pruning (GMP) have shown positive effects on both single-task performance and plasticity (Obando-Ceron et al., 2024). Similarly, dynamic sparse training methods such as Sparse Evolutionary Training (SET) (Mocanu et al., 2018) have also proven effective in single-task RL, offering an alternative way to maintain and adapt sparsity throughout training (Graesser et al., 2022). These successes suggest that such sparsification approaches could address the optimization pathologies that undermine effective multi-task learning. Nonetheless, a systematic investigation of their impact MTRL, where representational flexibility demands are significantly higher (Devin et al., 2016), remains largely unexplored.
This paper investigates whether sparsification methods, specifically GMP and SET (Mocanu et al., 2018), can enhance plasticity in MTRL agents, thereby improving performance across multiple tasks simultancously. Our choice to explore these methods is motivated by their demonstrated success in single-task settings and the need to understand their efficacy in the MTRL domain. We evaluate this across various multi-task architectures, including shared backbones with task-specific heads (MTPPO), Mixture of Experts (MoE) (Ceron et al., 2024), and Mixture of Orthogonal Experts (MOORE) (Hendawy et al., 2024), using common MTRL benchmarks that range from partially observable enviromments with sparse reward to high-dimensional and continuous state and action spaces. Our central aim is to understand if the benefits of pruming can be primarily attributed to the mitigation of key plasticity loss indicators. Our experiments compare sparse agents against dense baselines, less adaptive sparsification techniques, and a suite of alternative poasticity-inuucin or regularization methods, including Layer Normalization (Ba et al., 2016; Tyle et al., 2024a), ReDo (Sokar et al., 2023), Reset (Ash & Adams, 2020; Nikishin et al., 2022), and Weight Decay.
Our main contributions are, therefore, threefold:
We establish that sparsification methods, particularly Gradual Magnitude Pruning (GMP) and Sparse Evolutionary Training (SET), serve effective mechanisms for mitigating key indicators of plasticity degradation in MTRL, such as neuron dormancy and representational collapse. While the extent of these benefits varies with network architecture, sparse agents, especially in MTPPO and MoE configurations, consistently exhibit improved plasticity profiles compared to their dense counterparts.
We empirically demonstrate that these plasticity improvements induced by sparsification often correlate with enhanced multi-task performance. Sparse agents frequently outperform dense baselines and demonstrate competitive performance against alternative, specialized methods explicitly designed to induce plasticity, as well as common regularization techniques.
We show that the impact of sparsification on both plasticity and performance is architocture-depedennt, offering insights into the interplay wetween network design, sparsity, and learning dynamics. This highlights sparsification as a valuable but context-sensitive tool in the MTRL toolkit. Furthermore, we highlight that beyond performance, sparsification offers inherent advantages such as potential for computational efficiency and distinct form of implicit regularization not fully replicated by common regularization methods.
2

---

Published in Transactions on Machine Learning Research (07/ (07/2025)
2 Background
This section provides the necessary context for our approach. We begin by outlining the mathematical preliminaries underlying our framework, including key concepts and notations from reinforcement learning. Subsequently, we review related work, focusing on recent advances in sparsity in deep reinforcement learning, plasticily loss, and multi-task learning.
2.1 Preliminaries
We consider the Partially Observable Markov Decision Process (POMDP), defined by tuple
(S,4, P, R. S2, 0, consisting of a state space S, an action space A, transition dynamics P : S x A + A(S),
a reward function R S.Sx x A + IR, observation space Ω2 observation probability function O x A + A()),
and discount factor y E [0, 1). At each timestep i, the agent is situated in the true state 8, E and performs an action at E A. This causes the agent to transition to a new state St+1 S, receiving an observation 0z+1 E 2 and reward rt+1 R(s; at). The objective is to learn a policy (a(a)));) with parameters 0 that maximizes the expected sum of discounted future rewards J(0). In MTRL, the agent must learn policy for distributtin of tasks T. We adopt the Block Contextual POMDP framework (Sodhani el al., 2021; Hendawy et al., 2024), defined as (C,S,A, where C represents the contextual space such that c E C identifics a specific task T ~ T. The mapping M'(c) provides the task-specific POMDP components (R², Pe, PP,S", , 7.). The policy is now conditioned on the current observation o E S2 and lask context. c C.C.C The objective is to maximize the expected return across all tasks ETTT [J=(O).
2.2 Related Work
Sparsity in Reinforcement Learning While deep reinforcement learning has traditionally relied on overparameterized networks, recent research challenges the necessity of such scale, suggesting that sparse networks can match or even exceed the performance of dense models (Livne & Cohen, 2020; Graesser et al., 2022). This trend highlights that DRL agents often underutilize their capacity (Kumar et al., 2021) or overfit. to early experiences (Nikishin et al., 2022), making them particularly amenable to the regularizing effects of sparsity. Pruning neural connections reduces model complexity and noise, offering form of structural regularization that can improve robustness and generalization (Jin et al., 2022). Our work investigates how sparsity-based methods can inffuence learning dynamics and plasticity in multi-task RL.
Plasticity Loss in Reinforcement Learning It is well known that Reinforcement learning systems face a unique form of non-stationarity stemming from evolving policies, shifting data distributions, and the bootstrapping nature of value updates. This can culminate in the form of plasticity loss, a reduced ability of the network to learn from new experiences, even within familiar data distributions (Lyle et al., 2022; Dohare et al., 2024). Plasticity loss often manifests as premature performance plateaus, training instability, and heightened sensitivity to hyperparameter settings (Igl et al., 2021; Berariu et al., 2023; Klein al., 2024). Current understanding attributes plasticity loss primarily to unstable learning targets that create challenging optimization landscapes, with associated symptoms like collinear gradients (Lyle et al., 2024a), representational collapse (Moalla et al., 2024), and volatile gradient norms under adaptive optimizers (Lyle et al., 2024b). Internally, networks may suffer from shifting activation distributions, neuron saturation, or increasing dormancy over time (Sokar et al., 2023; Bjorck et al., 2021). Several interventions have been proposed to mitigate these effects. These include resetting techniques, such as periodic last-layer reinitialization (Ash & Adams, 2020; Nikishin et al., 2022; 2023), parameter update modulation strategies like Hare and Tortoise networks (Lee et al., 2024), and various architectural or optimization-basse approaches, including weight decay (Sokar et al., 2023), deep Fourier features (Lewandowski et al., 2024a), and classification-bass value lcarning (Farebrother et al., 2024). Normalization layers have also demonstrated benefits in this regard (Bhatt et al., 2023). Notably, Obando-Ceron et al. (2024) showed that gradual pruming can outperfone many methods specifically designed to promote plasticity. This supports the broader notion that general- purpose regularization might offer a more robust and simpler solution to plasticity loss than domain-spocific mechanisms, reinforcing the broader lesson that simplicity often outperforms specialized interventions (Klein et al., 2024; Nauman et al., 2024a).
3

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Multi-Task Reinforcement Learning MTRL seeks to train single agent across multiple tasks, balancing knowledge sharing for transfer against the risk of negative interference. Common MTRL techniques include shared encoders with task-specific heads (Teh et al., 2017), modular network designs (Yang et al., 2020), reward normalization (Hessel et al., 2018), compositional policy learning (Sun et al., 2022), and gradient projection or masking stralegies (Yu et al.; 2020; Hendawy et al., 2024). Mfxtuure-of-Expertt models have also gained traction, often enhanced with attention or orthogonality constraints for better task separation (Ceron et al., 2024; Cheng et al., 2023). Similarly, maintaining weight matrix orthogonality through regularization has been explored to enhance plasticity in continual learning settings, which face similar challenges to MTRL (Chung et al., 2024). A key observation in MTRL. is that, unlike trends in supervised learning, simply scaling model capacity does not inherently guarantee performance improvements (Hansen et al., 2023; Ceron el al., 2024; Nauman el al., 2024b). Supervised approaches like SimBa, for instance, suggest. that gains from scaling require careful inductive biases (Lee et al., 2025a). In contrast, network sparsity has demonstrated improvements in both generalization and plasticity in RL without necessarily relying on increased model scale (Graesser et al., 2022; Obando-Ceron et al., 2024). Despite the promise of sparsity, its implications in MTRL have remained largely unexplored To the best of our knowledge, this work is the first to systematically examine how different sparsity-inducing techniques affect performance and plasticity in the multi-task regime. We aim to fill this gap by investigating how pruning and sparse connectivity can mitigate plasticity loss and promote stable, generalizable learning in complex task environments.
3 Experimental Setup
Our experiments systematically compare the effects of different sparsification approaches against dense baselines and two families of plasticity-enhaacingg interventions. The first family comprises explicit plasticity- restoring techniques that directly intervene the agent's parameters to counteract plasticity loss. These include ReDo (Sokar et al., 2023), which periodically resets dormant neurons based activity thresholds, and Reset (Ash & Adams, 2020; Nikishin et al., 2022), which reinitializes specific network layers at fixed intervals to combat primacy bias. The second family involves more implicit regularizatiio--assed mechanisms, which do not directly manipulate network dynamics but are known to stabilize training and encourage generalization. Specifically, we evaluate standard Weight Decay (WD) applied to dense agents, and Layer Normalization (LayerNorm) (Ba et al., 2016), which has recently been linked to mitigating plasticity loss by roducing covariate shift and promoting balanced neuron activations (Lyle et al., 2024a).
Unless otherwise specified, we benchmark these methods across three representative multi-task reinforcement learning algorithrs: MTPPO, a shared-policy baseline with task-specific heads; a model (Ceron et al., 2024); and MOORE (Hendawy et al., 2024), which incorporates orthogonal submodules for each task. This comprehensive evaluation allows us to disentangle the relative contributions of sparsity, explicit resets, and architectural regularization to plasticity preservation and multi-task performance. We report the normalized interquantile mean (IQM) with shaded regions indicating 95% stratified bootstrap confidence intervals, calculated using the rliable library (Agarwal et al., 2021).
(MoE)
Environment and Benchmarks We mostly consider the three multi-task MiniGrid (Chevalier-ooisver) et al., 2023) benchmarks proposed by Hendawy et al. (2024) MT3, MT5, and MT7, with the exception being made for the results presented in Section 4.3, which use the MetaWorld MT10 benchmark (Yu et al., 2021). All environment details are outlined in Appendix B. We note that to ensure fair comparison across tasks with inherently different reward scales, for MiniGrid, raw episodic returns are normalized with respect to the maximum achievable reward in each environment (see Appendix B.3).
Implementation and Training For MiniGrid, we use the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) via the mushroom r1 library (D'Eramo et al., 2021) and the code provided by Hendawy et al. (2024) for multi-task architectures. Performance is measured by the episodic return across all tasks within the respective benchmark. Tasks are sampled randomly with replacement at the beginning of each episode during training. For MctaWorld, we use the Multi- Task Melti-Headed Soft Actor-Critic (MTMH SAC) (Haarnoja et al., 2018; Yu et al., 2021) and track the mean success rate across all tasks. We outline full training details and hyperparameters in Appendix A.
4

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Sparse Methods To better characterize the role of sparsification in the MTRL setting, began with series of preliminary experiments comparing various sparsification ion strategies. Our goal was to identify methods that balance learning stability, generalization, and simplicity, while remaining compatible with the dynamic nature of multi-task settings. We evaluated several sparsification techniques, including: the Gradual Magnitude Pruning (GMP) schedule proposed by Zhu & Gupta (2017), Sparse Evolutionary Training (SET) (Mocanu al., 2018), and Lottery Ticket Hypothesis (LTH) style rewinding (Frankle Carbin, 2019). These initial experiments were conducted on the MT5 benchmark, selected as practical compromise: it is more challenging than MT3, allowing to meaningfully stress-test pruning strategies, yet significantly more computationally efficient than MT7, enabling extensive ablations at a reasonable cost. The results of this comparison are presented in Figure 1.
Among the three tested approaches, both GMP and SET resulted more stable learning dynamics and improved generalization performance. As shown in Figure 1, sparse models trained with GMP and SET not only surpass their dense counterparts but also outperform single-task base- lines. Conversely, LTH- models fail to yield signif- icant improvements over single-task training, highlighting their limited capacity to adapt in multi-task settings. We note that these results are consistent with findings in single-task reinforcement learning (Graesser et al., 2022;
Dense
GMP
SET
LT1]
ST PPO
Normalized IQM
0.4
Obando-Ceron et al., 2024), and further underscore the ad- vantages of sparsity mechanisms that adapt progressively throughout training. Given these insights, the remainder of our experimental study focuses on the two approaches that consistently demonstrated better performance: GMP and SET. The first, GMP, incrementally increases the net- work's s sparsity level during training by gradually remov- ing mog-magit weights over a predefined time window, This allows the network to adapt to the increasing sparsity and miligates the risk of destabilizing learning dynamics. For further information about the pruming schedule, we refer the reader to Appendix C.4. The second, SET, takes inspiration from evolutionary algorithms and maintains a fixed overall sparsity throughout training by continuously rewiring the network's connectivity. Unlike gradual prun- ing, which increases sparsity time, SET preserves a constant sparsity level but introduces dynamic plasticity through periodic topological updates. More details about the SET algorithm are presented in Appendix C.5. We note that both strategics are well aligned with the core objective of this work, which is to investigate the learning dynamics of sparse agents rather than to optimize inference-tine performance. As both SET and GMP rely on unstructured pruning, they provide greater representational flexibility than structured approaches, making them particularly suitable for analyzing adaptation and interference multi-task reinforcement lcarning (Hoefler al., 2021).
100
150 200
Epoch
Figure 1: Comparative performance of vari- ous sparsification strategies (GMP, SET, LTH) against a dense multi-task bascline and an aggro- gation of single-tasle agents trained individually each task the MT5 benchmark. Both GMP and SET outperform the dense multi-task base- line and the LTH-based models, while LTH-based models show limited improvement over single-task performance and struggle to adapt effectively in the multi-Lask selting.
Plasticity Measures We monitor three metrics during training, interpreted correlative indicators of plasticity based recent surveys and analyses (Berariu et al., 2023; Lyle et al., 2023; Klein et al., 2024; Falzari & Sabatelli, 2025), namely dormant neuron percentage, effective rank, and the trace of the Fisher Information Matrix. The computation of these metrics is detailed in Appendix C. Our analysis focuses on observing consistent patterns between pruning interventions, changes in these metrics, and MTRL performance, rather than claiming direct causality.
4 Core Effects of Sparse Methods
This section details our first set of empirical findings, beginning with the effect of sparsity on multi-task performance, followed by analysis of its effects on plasticity indicators. All performance comparisons refer
5

---

Published in Transactions on Machine Learning Research (07 (07/2025)
to the aggregated final outcomes presented in Table 1. Plasticity metric analyses are primarily illustrated using the MT5 benchmark data shown in Figure 2, Figure 4, and Figure 5, as trends were generally consistent across other benchmarks unless otherwise stated. Detailed learning curves and plasticity metrics for all benchmarks are available Appendix F and Appendix G, respectively.
Table 1: Final aggregate performance epoch 200 across architectures and agent treatments on MT3, MT5, and MT7 benchmarks. Gold marks the best performance within the respective multi-task architecture and benchmark, while blue marks the second-best, performance within each architect and benchmark. Full learning curves illustrating training progression are available in Appendix F.
MT3
MT5
MT7
Agent Treatment
IQM (t)
95% CI
IQM (t)
95% CI
IQM (t)
95% CI
Multi- Task PPO (MTPPO)
Dense
0.70
(0.60, 0.76)
0.65
(0.61, 0.71)
0.72
(0.69, 0.75)
Gradual Pruning
0.77
(0.73, 0.80)
0.81
(0.75, 0.86)
0.76
(0.70, 0.80)
SET
0.76
(0.74, 0.77)
0.80
(0.75, 0.84)
0.80
(0.77, 0.84)
ReDo
0.74
(0.68, 0.77)
0.83
(0.78, 0.84)
0.80
(0.76, 0.83)
Reset
0.70
(0.64, 0.72)
0.80
(0.74, 0.84)
0.80
(0.76, 0.83)
Weight Decay
0.71
(0.70, 0.78)
0.75
(0.66, 0.82)
0.74
(0.70, 0.77)
LayerNorm
0.28
(0.20, 0.37)
0.33
(0.25, 0.40)
0.38
(0.33, 0.45)
Micture of Experts (MoE)
Dense
0.74
(0.71, 0.76)
0.77
(0.70, 0.82)
0.80
(0.75, 0.84)
Gradual Pruning
0.77
(0.74, 0.79)
0.84
(0.78, 0.86)
0.87
(0.83, 0.88)
SET
0.76
(0.74, 0.78)
0.79
(0.72, 0.85)
0.82
(0.78, 0.85)
ReDo
0.77
(0.76, 0.80)
0.82
(0.81, 0.85)
0.85
(0.82, 0.88)
Reset
0.64
(0.54, 0.73)
0.78
(0.73, 0.83)
0.84
(0.80, 0.87)
Weight Decay
0.75
(0.71, 0.76)
0.78
(0.70, 0.85)
0.77
(0.71, 0.82)
LayerNorm
0.44
(0.38, 0.46)
0.34
(0.29, 0.40)
0.39
(0.32, 0.43)
Mizture of Orthogonal Experts (MOORE)
Dense
0.78
(0.71, 0.80)
0.84
(0.81, 0.85)
0.87
(0.84, 0.88)
Gradual Pruning
0.80
(0.77, 0.81)
0.85
(0.80, 0.87)
0.88
(0.86, 0.89)
SET
0.69
(0.60, 0.74)
0.78
(0.72, 0.83)
0.82
(0.79, 0.85)
ReDo
0.72
(0.68, 0.75)
0.82
(0.78, 0.84)
0.85
(0.84, 0.87)
Reset
0.67
(0.62, 0.72)
0.75
(0.71, 0.79)
0.84
(0.80, 0.87)
WD
0.75
(0.73, 0.76)
(0.76, 0.87)
0.87
(0.84, 0.88)
LayerNorm
0.54
(0.49, 0.62)
0.60
(0.53, 0.65)
0.66
(0.61, 0.70)
4.1 Sparse Methods Improve Task Performance
Our findings indicate that both GMP and SET generally lead to improvements in multi-task performance, an observation consistent with a significant body of research in supervised learning, where appropriately pruned sparse networks have been shown to match, outperform, and generalize better than their dense counterparts (Guo et al., 2019; Morcos al., 2019; Sabatelli et al., 2020; Hoefler et al., 2021). However, in our multi-task settings, the extent of these benefits from pruning varies with the underlying agent architecture and desired sparsity level. For MTPPO and MoE architectures, both Gradual Pruning and SET consistently resulted in improved final aggregate returns compared to their respective dense baselines across all tested benchmarks (MT3, MT5, MT7), as shown in Table 1. This suggests that these common MTRL architectures frequently contain considerable overparameterizatioon that sparse methods can effectively address, hinting at a direct. link between sparse intervention and improved MTRL outcomes. In contrast, the impact of sparse methods on MOORE was more muanced. While general, the effect of GMP on MOORE on performance was close that of the dense bascline, SET led a slight decline across all benchmarks. While substantial gains were not observed for MOORE with sparsification, the ability to prune high levels of sparsity (up to

---

Published in Transactions on Machine Learning Research (07 (07/2025)
95%) without significant performance degradation still indicates that even sophisticated architectures be overparameterized. Nonetheless, we note that very aggressive pruning (e-g., 99% sparsity with GMP) could lead to issues such as rank collapse or performance drops in MOORE (see Appendix H, Figure 23 and Figure 24).
4.2
Sparse Methods Mitigate Plasticity Loss
Dense
GMP
SET
Fisher Trace
Effective Rank
Actor Dormant [%]
Critic Dormant [%]
150
340
25
30
MTPPO
320
20
25
100
300
15
20
15
10
30
390
MoB
310
30
320
40
300
10
20
20
280
260
10
120
MOORE
100
16
360
20
144
340
12
320
15
300
50
200
200
100
150
Epoch
Figure 2: Evolution plasticity indicators for Dense, Gradual Magnitude Pruning (GMP), and Sparse Evolutionary Training (SET) across different MTRL architectures (MTPPO, MOE, MOORE) on the MT5 benchmark. Subplots display Fisher Trace, Effective Rank, and percentages of Actor and Critic Dormant neurons, illustrating the distinct effects of each sparsification strategy compared to the dense baseline.
The observed performance improvements, particularly within the MTPPO and MoE architectures, strongly correlate with the ability of sparsification methods to mitigate common indicators of plasticity loss, while displaying distinct learning dynamics for GMP and SET. Notably, the plasticity profiles (Figure 2) show that agents employing either GMP or SET generally exhibit more favorable plasticity metrics compared to their dense counterparts. Specifically, sparse agents typically maintain lower percentages of dormant neurons and higher or more stable mean effective rank in their representations. Furthermore, the trace of the Fisher Information Matrix (FIM) in sparse agents typically stabilizes at lower values post-initial learning, suggesting convergence to less sensitive parameter configurations, in contrast to the often persistently high values in dense networks. While both GMP and SET contribute to these general improvements over dense networks, they induce individually different plasticity dynamics. SET, with its continuous rewiring, proved particularly effective minimizing neuron dormancy very low levels in both actor and critic components throughout training, while also maintaining higher effective rank. In contrast, GMP's impact on dormancy was more pronounced in the actor network, with both actor and critic dormant percentages remaining higher than those under SET, though still improvement over dense networks. The FIM trace also differed: GMP often displayed a characteristic peak-and-decline pattern, whereas SET maintained low and stable FIM trace throughout training, suggesting continuous adaptation within a less volatile optimization regime. Collectively, these observations support the hypothesis that sparsification methods enhance the learning capability of MTRL agents, plausibly through the mitigation of processes associated with plasticity degradation in dense networks. Neverthcless, the influence sparsification on MOORE's plasticity did mirror the benefits seen in MTPPO and MoF agents, aligning with the varied overall performance outcomes discussed

---

Published in Transactions on Machine Learning Research (07/)025)
above. For MOORE, SET did reduce neuron dormancy, and its Fisher Trace showed slow growth and stabilization pattern. However, the effective rank for both SET and GMP remained similar to the dense baseline. GMP, in contrast to its effect in other architectures, sometimes even slightly increased dormancy compared to dense MOORE on certain metrics. Importantly, these specific plasticity modulations, such as SET's reduced dormancy in MOORE, generally did not translate into performance improvements for this architecture, with SET often resulting in a slight performance decline. This suggests that MOORE's inherent design, particularly its emphasis on representation orthogonalization (Hendawy et al., 2024), may interact with sparsification in various ways. Its sophisticated structure might be less responsive to the typical benefits derived from these plasticity changes, as it already exhibits relatively stable plasticity characteristics.
4.3 Generalization Continuous Control
To evaluate whether our plasticity-rclated findings in MiniGrid generalize to continuous con- trol, we extended our analysis to the MetaWorld Our experimental setup was guided by two insights from McLean et al. (2025): increasing the critic's capacity tends to yield greater benefits than increasing the actor's, making the critic the more capacity-sensitive component; and while overall plas-
MT10 benchmark (Yu et al., 2021).
GMP-Bal's
ticity loss (e.g., neuron dormancy) is relatively low in dense agents, it tends to be more pronounced in the actor than in the critic. These findings led us hypothesize that pruning only the actor, while preserving the full capacity of the critic, could enhance performance by improving network elficiency without compromising representational power. We evaluated this hypothesis by comparing three conditions: a dense MTMH- SAC baseline, GMP applied to both actor and critic, and GMP applied to the actor only. The actor-only pruning approach achieved the highest final success rate at 81% (95% Cl: 0.77, 0.83), outperforming both the dense bascline (73%; 95% CI: 0.67, 0.75) and the global pruning condition (75%; 95% CI: 0.73, 0.78); see Appendix F, Figure 13 for exact learning curves. This performance improvement was accompanied by a sustained reduction in actor neuron dormancy, as depicted in Figure 3, suggesting a more adaptive and efficient use of network capac- ity. Overall, these results extend our MiniGrid-based plasticity findings to the more complex Meta MetaWWrrd benchmark and offer
Aclor Documet [96]
10
200
Epuch
Figure 3: Reduction in actor neuron dor- mancy on MetaWorkd MT10. Selective prun- ing of the actor network (GMP-Actor) leads to a sustained decrease in dormant neu- rons compared to both the dense MTMH- SAC baseline and the globally pruned model (GMP-Bothh.
complementary perspective to McLean et al. (2025): while they emphasize scaling the critic, show that selectively pruning the actor can be equally beneficial. Together, these insights highlight the asymmetry in actor-critic dynamics and suggest that the benefits of sparsity are both role and context-dependent.
5 Interactions with Alternative Mechanisms
This section shifts focus to comparative analysis between sparsification and other alternative strategies for plasticity and multi-task learning. We first examine pruning in relation to explicit interventions such as ReDo and Reset, which dircetly manipulate network parameters to countcract plasticity loss. We then consider implicit mechanisms such as standard regularization techniques (Weight Decay) and architectural choices (LayerNorm) that influence plasticity without explicit intervention. All comparisons are conducted under the same multi-task training setup and are summarized in Table 1. Additionally, we present a final ablation study exploring potential synergies of combining GMP with other optimization techniques (Weight Decay and PCGrad (Yu et al., 2020)).
8

---

Published in Transactions on Machine Learning Research (07 (07/2025)
5.1 Sparsification versus Explicit Plasticity-Inducing Mechanisms
We compared GMP and SET against interventions that explicitly target symptoms of plasticity loss: ReDo (reinitializing dormant neurons) and Reset (layer reinitialization), using their best-performing configurations derived after hyperparameter tuning (scc Figure 11 of Appendix E). In terms of final task performance (Table 1), sparsification methods generally achieved returns competitive with, and occasionally better than, ReDo or Reset, especially for MTPPO and MoE architectures. While statistical significance for outperformance was not always established due to overlapping confidence intervals, sparse methods consistently presented a strong alternative without directly targeting specific plasticity symptoms. For MOORE agents, performance differences between ReDo and the sparse approaches were minimal, while Reset introduced substantial variability. Examining the plasticity profiles (Figure 4), SET was particularly effective for MTPPO and MoE, often maintaining lower percenlage of actor dormant neurons than even ReDo and consistently achieving the highest effective rank. In contrast, within MOORE, ReDo was more effective in reducing dormancy, while SET's effective rank advantage was less apparent. ReDo's impact on the Fisher Trace and mean effective rank often mirrored that of the dense baseline, indicating primarily addressed dormancy without broadly altering other representational characteristics. The Reset intervention, due its periodic reinitializations, frequently induced abrupt shifts and instability in markers like the FIM and effective rank, especially post-reset, consistent with prior work (Falzari & Sabatelli, 2025). Performance-wise, Reset rarcly outperformed sparse agents, whereas ReDo was more competitive; however, achieving a lower percentage of dormant neurons via ReDo did not always guarantee superior task performance (e.g., in MOORE), and SET sometimes achieved lower actor dormancy without this direct targeting.
GMP
SET
RoDo
Reset
Fisher Trace
Effective Rank
Actor Dormant [%]
Critic Dormant [%]
310
25
30
MTPPO
150
20
100
15
20
290
260
10
240
380
100
360
23
30
MoE
340
20
320
15
20
10
260
1220
MOORE
100
400
20
60
350
15
10
10
300
50
100
150
200
100
150
200
100
150
200
0
100
150
200
Epoch
Figure 4: Comparative plasticity dynamics of sparse methods (GMP, SET) versus explicit plasticity-inducing interventions (ReDo, Reset) and a dense baseline, across MTPPO, MoE, and MOORE architectures on the MT5 benchmark. Metrics include Fisher Trace, Effective Rank, and percentage of Actor and Critie Dormant Neurons.
5.2 Sparsification versus Implicit Plasticity-Inducing Mechanisms
To further characterize the role sparsification fostering plasticity, we contrast its effects with more implicit baattiity-induuiin mechanisms: weight regularization (Weight Decay, WD) and architectural normalization (TayerNorm). These techniques have been explored for mitigating plasticity loss by promoting parameter

---

Published in Transactions on Machine Learning Research (07/2025)
Deaze
GMP
SET
WD
LayerNorm
Fisher Trace
Effective Rank
Actor Dormant [%]]
Critic Dormant [%]
400
25
MTPPO
350
100
13
20
250
10
30
150
5
100
400
25
MoE
60
300
20
40
20
200
10
MOORE
80
20
60
15
12
40
300
20
250
100
150
200
0
50
100
150
200
50
100
150
200
50
100
150
200
Epoch
Figure 5: Comparative plasticity dynamics of sparse methods (GMP, SET) versus regularization techniques (Weight Decay, LayerNorm) and a dense baseline, across MTPPO, MoE, and MOORE archilectures the MT5 benchmark. Metrics include Fisher Trace, Effective Rank, and percentage of Actor and Critic Dormant Neurons.
stability or scale-invariant updates (Lyle et al., 2023). These experiments evaluate whether sparsification methods like GMP and SET offer distinct advantages over, or complementary benefits to, these common regularization approaches. In terms of task performance, both GMP and SET generally outperform agents trained with only Weight Decay or LayerNorm across all architectures and benchmarks. In Figure 5, Weight Decay exhibits plasticity dynamics remarkably similar to the dense baseline across all metrics and architectures. This suggests that while WD is common regularizer, in these MTRL contexts, it does not substantially alter plasticity characteristics beyond a standard dense network, nor does it typically lead to performance surpassing well-configured sparse agents. LayerNorm, conversely, induces more pronounced changes to plasticity. Even though can lead to very low levels of dormant ncurons, this accompanied by severe and persistent reduction the effective rank. This drop, visible across all architectures, signifies limited representational diversity, which also correlates with its lowered performance. We hypothesize that in the multi-task learning setting, LayerNorm, by normalizing activations across features within each layer and sample, might inadvertently introduce strong correlations in the gradients from different tasks smooth out task-specific feature distinctions excessively. This could lead to a less expressive representation space, hindering the network's ability to learn diverse task solutions despite the apparent reduction in neuron dormancy. The low effective rank, coupled with often the atypically low Fisher Traces for a dense agent, likely contributes to LayerNorm's 's consistently poor task performance. In contrast, sparsification methods like GMP and SET generally achieve better balance: they effectively mitigate dormancy (SET often being most effective) and maintain or improve effective rank compared the dense baseline (especially GMP initially), without the severe representational collapse with LayerNorm. This suggests that sparsification offers more muanced approach to capacity control and plasticity preservation than these standard implicit regularization techniques in the studied MTRL scenarios, leading to superior overall learning outcomes.
5.3 Interaction with Optimizers
We finally explored the potential synergies of combining GMP with other optimization techniques, specifically weight decay and PCGrad (Yu et al., 2020), popular method designed to mitigate
10

---

Published in Transactions on Machine Learning Research (07/2025)
gradient interference in multi-task settings, on the MTPPO architecture using the MT5 bench- mark. The performance results are shown in Figure 6 and plasticity dynamics in Figure 7. In terms of final task performance, GMP in isolation achieved the highest returns (Figure 6).
While PCGrad alone improved performance over the dense basc- line, its combination with GMP did not yield further gains be- yond GMP alone. Interestingly, the plasticity profiles also reveal that GMP alone maintained the most favorable characteristics (Figure 7). The GMP+WD combination showed similar plastic- ity dynamics to GMP alone, although with slightly worse values in some metrics, correlating with its slightly lower performance. This lack of synergy with Weight Decay might be anticipated, as WD encourages smaller weight magnitudes overall, potentially incrcasing the pool of weights that magnitudo-basee would larget, which could lead to less discerning pruning process even premature removal of weights that might have otherwise become important. PCGrad, when applied to a dense network, did not demonstrably improve plasticity indicators such as Fisher Trace or actor dormancy compared to the dense baseline, despite its performance uplift. This suggests that the primary benefits of GMP in this context may stem from its inherent regularization effects and capacity optimization, which are not necessarily enhanced by, or may even be slightly counteracted by, the addition of these particular optimizers.
Donee
POdrd
GMP
GMP+WD
GMP+POOnd
ST PPO
01.5
Normatioed IQM
0.6
pruning
200
Epoch
Figure 6: Performance comparison for Grad- ual Magnitude Pruning (GMP) interactions with Weight Decay (WD) and PCGrad on the MTPPO architecture with the MT5 benchmark.
Decm
PCGma
GXP
GMP-WW
GMPP-PCCRal
Fisher
Elfective Rank
Actor Dormant [56]
Critic Dormant [%]
341
30
322
20
00
200
10
240
18
(10)
150
200
341
100
150
200
541
100
150 250
38 100
150
200
Epoch
Figure 7: Comparative plasticity dynamics for Gradual Magnitude Pruning (GMP) and its interactions with Weight Decay (GMP+WD) and PCGrad (GMP+PCGrr) contrasted with PCGrad on dense network and a dense baseline. All experiments are on the MTPPO architecture with the MT5 benchmark. Subplots display (from left to right)): Fisher Trace, Effective Rank, Actor Dormant %, and Critic Dormant %.
6 Considerations of Sparse Methods
While this study is primarily empirical, the observed benefits of sparsification methods in MTRL can be interpreted through several established concepts from the sparsity, optimization, and multi-task learning literature. However, we also want to acknowledge their limitations and considerations for practical application.
Optimization, Sparsity, and Generalization The iterative removal (GMP) or rewiring of connections (SET) effectively guides the network towards sparse solutions. This mechanism can generally be viewed as Lo regularization, encouraging sparsity by penalizing the mumber of non-zero parameters (Louizos et al., 2018) and reducing the model's degrees of freedom. While this might confine optimization to lower-dimensional subspaccs (Gao & Jojic, 2016; Hocfler et al., 2021), such sparsc solutions are often associated with 'flatter' minima in the loss landscape (Pesle, 2023; Shah et al., 2024). Flatter minima are highly desirable due to lower sensitivity to parameter perturbations (Foret et al., 2021; Lee et al., 2025b), widely believed to result in better gencralization and robustness under distribution shifts (Hochreiter & Schmidhuber, 1997; Jiang et al., 2019; Kaddour et al., 2023; Li et al., 2024), the primary drivers of plasticity loss. The convergence such local flat minima is often indicated by specific dynamics in the curvature of the loss for instance, the
11

---

Published in Transactions on Machine Learning Research (07/2025)
maximal Hessian eigenvalue typically grows, peaks, and then declines during training (Fort & Ganguli, 2019). Our empirical results for MTPPO and MoF, where the Fisher Trace (a proxy for curvature (Tewandowski et al., 2024b)) exhibited a peak- peel-ind-decline pattern with GMP (Figure 2), align with convergence to such flatter minima. SET, with its contimuous rewiring, maintained a low, stable Fisher trace, suggesting a robust optimization process.
This aligns with findings in supervised learning where sparse networks are recognized for reduced overfitting and better generalization than dense counterparts (Gopalakrishnan et al., 2018; Cosentino et al., 2019; Guo et al., 2019; Liu et al, 2019; Liu, 2020). The iterative nature of GMP and the dynamic regrowing of SET are generally thought to help models evade suboptimal local minima (Jin et al., 2016; Gale et al., 2019; Hoefler et al., 2021; Jin et al., 2022; Graesser et al., 2022), a principle consistent with our findings. Furthermore, the success of pruning in MTRL models with shared backbones and task-specific heads mirrors similar effectiveness in multi-task supervised learning Xiang et al. (2024). Nonetheless, an excessive reduction in degroes of freedom via aggressive pruning can hinder the satisfaction of specific architectural demands, such as maintaining expert orthogonality in MOORE (see Appendix H, Figure 23 and Figure 24), especially if capacity becomes overly constrained.
Limitations and Practical Considerations Despite their benefits, these sparsification techniques have considerations. GMP, while conceptually simple, typically operates on dense weight matrices internally during training, applying masks to simulate sparsity. This means it does not inherently reduce the memory footprint or computational cost during training compared to dense models; true benefits often require specialized hardware or software for sparse operations at inference. While inference can be efficient, the training phase still bears the overhead of the original dense model, with the additional overhead of pruning at specified timesteps. SET, the other hand, can maintain true spansity throughout training and inference if implemented with sparse data structures. However, current widely available implementations are often optimized for fully connected layers, and extending their dynamic rewiring efficiently to convolutional or recurrent architectures can be more complex. Additionally, the random nature of SET's regrowth phase, while promoting exploration, might not always lead to the most optimal connectivity patterns without more guided heuristics. Both methods also require careful tuning of their own hyperparameters to achieve optimal results. While potentially less sensitive than some explicit plasticity interventions, this still constitutes tuning effort. The architecture-deeender nature of the benefits, shown by our experiments with MOORE, also indicates that these are not one-size-fits-ll solutions.
7 Conclusion
In this work, we examined dynamic sparsification, specifically Gradual Magnitude Pruming (GMP) and Sparse Evolutionary Training (SET), as a means to mitigate plasticity loss and improve performance in multi-task reinforcement learning (MTRL... Our results show that both methods can enhance adaptability and generalization across several architectures, with consistent gains observed in MTPPO and MoE agents. Similar benefits were also found in Multi-Headed SAC agents evaluated on the Morad MT1O benchmark, suggesting that the effectiveness of sparsification extends to continuous control tasks. While performance on MOORE was more variable, likely due to its built-in mechanisms for managing interference, our findings highlight the importance of aligning sparsification strategies with architectural design. In addition to perfor- mance improvements, sparsity can offer benefits such as reduced hyperparameter sensitivity, computational efficiency, and implicit regularization through structured parameter removal. These results support the view that general-purpose mechanisms that shape learning dynamics rather than task-specific interventions can yield robust benefits in MTRL. As future work, we plan on investigating the theoretical underpinnings and potential interpretability advantages of sparse MTRL models.
References
Zahoer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of Plasticity in Continual Deep Reinforcement Learning, March 2023. URL arXiv:2303.07507 [cs].
07500
12

---

Published in Transactions on Machine Learning Research (07 2225))
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep Reinforcement Tearning at the Edge of the Statistical Precipice. In Advances in Newral Information Pro- cessing Systems, volume 34, pp. 29304 29320. Curran Associates, Inc., 2021. URL https: /tttp::/prrceee /ttp////eccceee.../// Abstract.htum
httpa///eaaaaeeee////a/////aaaaa courips.cc/paper.
Jordan Ash and Ryan P' Adams. On Warm-Startin Neural Network Training. In Advances in Neural Informa- tion Processing Systems, volume 33, pp. 3884-3894. Curran Associates, Inc., 2020. URL https://papers.
neurips... heurips.cc/paper. Secaaperrfleeseee
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization, July 2016. URL http:
//arxiv. wwwpxii.rrg/bbs...o wwgg/bs//////////o/ 06450. arXiv:1607.06485 [stat]]
Mare G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodeep Moitra, Sameera S. Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforce-
ment learning. Nature, 788778))) December 2020. ISSN 1476-4687. doi: 10.1038/s4 (0444441000-2200000
URL httph::t////tt btttps://www..e www.w.ature..o cow www./////tttttteoeooooooooe 020-8866-22229-33020000 2939-8. Publisher: Nature Publishing Group.
Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks, October 2023. URL http://arxix: org/ 2b16 00042. arXiv:2106.000% [cs].
Aditya Bhatt, Daniel Palenicok, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Elficiency and Simplicity. October 2023. URL
Johan Bjorck, Carla P. Gomes, and Kilian Q. Weinberger. Is High Variance Unavoidable in RL? A Case Study in Continuous Control. October 2021. URL https: httpps://pperreviex. http:/reww....ee/ooo///dsds//aee
Matthew Botvinick, Sam Ritter, Jane X. Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hass- abis. Reinforcement Learning, Fast and Slow. Trends in Cognitive Sciences, 23(5):408-422, May 2019. ISSN 1364-6613, 1879-307X. doi: 10.1016/j..ics.2019.. 2019.02.006. URL https://www.cel) cell. com/trends/ oognlttieesccieeee///ttttt/////d (1000000-66111))0006----...... 30061-0. Publisher: Elsevier.
Johan Samir Obando Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nicolaus Foerster, Gintare Karolina Dziugaile, Doina Precup, and Pablo Samuel Castro. Mixtures of Experts Unlock Parameter Scaling for Deep RL. June 2024. URL https: https:///ppenrevee http::/r.....orrrr/////////////
Guangran Cheng, Lu Dong, Wenzhe Cai, and Changyin Sun. Multi-Task Reinforcement Learning With Attention- Basse Mixture of Experts. IEEE Robotics and Automation Letters, 8(6):3812-331)) June 2023. ISSN 2377-3766. doi: 1LL109/LLA.2223327144444 URL https://eeessplore.eeee ieee org//ocuett///t 10111062.
Maxime -hevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks, June 2023. URL http:://axtti http://wwwvvv.///aaaa/// abs/ 2306. 13831. arXiv:2306.138: 13831 [es].
Wesley Chung, Lynn Cherif, Doina Precup, and David Meger. Parseval Regularization for Continual Reinforcement Learning. November 2024. URL https://openrevieet wwt.:/oorumid--aIIPPPP
Justin Cosentino, Federico Zaiter, Dan Pei, and Jun Zhu. The Search for Sparse, Robust Neural Networks, December 2019. URL http://arxits: orgg/bs//1//...o.o http://aarxtvorrg//as//11122
02386. arXiv:1912.02386 [cs].
Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. MushroomRL: Simplifying Reinforcement Learning Rescarch. Journal of Machine Learning Research, 22(131) 2021. ISSN 1533-7928. URL http://jmlr... http::/jmml.oor//pppree..
arg/papers//22/115 11112///181566611 htmm...t....
Carlo D'Eramo, Davide Tatco, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing Knowledge in Multi- Task Deep Reinforcement Learning, January 2024. URL http://arxit: wwwpiiiooor//bba. www./o///////////o/// 09561. arXiv:2401.09m0 [cs).
13

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Coline Devin, Abhishek Gupta, Trevor Darrell, Picter Abbeel, and Sergey Levine. Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer, September 2016. URL http://arxiv..t abs/1609.07000 arXiv:1609.07000 [cs].
Shibhansh Dohare, J. Fernando Kernandez-Garcia, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. Maintaining Plasticity in Deep Continual Learning, April 2024. URL http://arxiv. org//bs/
2306. 13812. arXiv:2306. 13812 [cs].
Massimiliano Falzari and Matthia Sabatelli. Fisher-Guided Selective Forgetting: Mitigating The Pri- macy Bias in Deep Reinforcement Learning, February 2025. URL
arXiv:2502.00802 [cs].
Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and Regularization in DQN, January 2020. URL http://arree wwwp://g///.......//.0// 00123. arXiv:1810.00123 [es].
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sorgey Levine, Pablo Samuel Castro, Alcksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop Regressing: Training Value Functions via Classification for Scalable Deep RL, March 2024. URL http:
//arxiv. aaaxivv.rgg/bs//24000 0395000000000........
03950. arXiv:2403.03950 [cs].
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-Awar Aware Minimization for Effi 01412. arXiv:2010.04112
ciently Improving Generalization, April 2021. URL [cs].
014421
Stanislav Fort and Surya Ganguli. Emergent properties of the local geometry of neural loss landscapes, October 2019. URL http://arxiv. www.aaiwww...gabbaaaaaaaaeeeaaae wwtggbbb//990//// 05929. arXiv:1910.05929 [cs].
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, March 2019. URL httpp://axxvv..orrge 1a00b///33333355. arXiv:1803.0366 [os].
Trevor Gale, Erich Elsen, and Sara Hooker. The State of Sparsity in Deep Neural Networks, February 2019. URL http://arxiv> httt:://rxdv.orr/abbb/ 19021 09574. arXiv:1902.09574 [es].
Tianxiang Gao and Vladimir Jojic. Degrees of freedom in deep neural networks. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, UAI'16, pp. 232-241, Arlington, Virginia, USA, June 2016. AUAI Press. ISBN 978-0-9966431---5.
Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A Survey on Interpretable Reinforcement Learning, February 2022. URL arX1v:2112.18112 [cs).
13112.
Soorya Gopalakrishnan, Zhinus Marzi, Upamanyu Madhow, and Ramtin Pedarsani. Combating Adver- sarial Attacks Using Sparse Representations, July 2018. URL http://arxiv. wwgg/as/////////e/e/ae/////////e
arXiv:1803 1803.03880 [stat].
Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. The State of Sparse Training in Deep Reinforcement Learning, June 2022. URI. http:/ htttp:///xxivvv. hawpiaxivvvvvoo///a//aaaa wwt./org//////ee///oooeee 10369. arXiv:2206.108: [cs).
Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse DNNs with Improved Adversarial Robustness, November 2019. URL attp://arxiv htt.:/.rrr/ab///1/0. 09619. arXiv:1810.09619 [cs].
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Crite Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, August 2018. URL http://arxiv.or abs/1801 01290. 1rXiv:1801.01290 [cs].
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, Robust World Models for Continuous Control. October 2023. URL https://openreview/ http://wwww......eoooooe////oaaa
Ahmed Hendawy, Jan Peters, and Carlo D'Eramo. Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts, May 2024. URL http://axx..orrg/aa///1111.1111 11385. arXiv:2311.11385 (es).
14

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task Deep Reinforcement Learning with PopArt, September 2018. URL http://arxiv hwwphlvv.rrr//s....//////s/oo abs/ $809.044777 arXiv:1809.04774 [cs].
Sepp Hochreiter and Jürgen Schmidhuber. Long Shrrt-Term Memory. Neural Comput., 98)::1335-178)) November 1997. ISSN 0899-7667. doi: n0.1162/neco.1197.33 neco 1997.1197.00 998.779.8.. 9.8 8. 1735.
URL
Torsten Hoefter, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks, Jamuary 2021. URL http://arxiv. www..oog/aaba. harg/bbs/111 2a0s/ 00554. arXiv:2102.0055 [cs].
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient con-Stationarity and Generalisation in Deep Reinforcement Learning, September 2021. URL http:
//arxiv. //aaa//v.rrg/as////////ooooa www.ooorr//bbo.//oooo 058./200000000....222 arXiv:2006.0588 05826 [cs].
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic General- ization Measures and Where to Find Them, December 2019. URL arXiv:191..02178 [cs].
Tian Jin, Michael Carbin, Daniel M. Roy, Jonathan Frankle, and Gintare Karolina Dziugaite. Pruning's Effect on Genoralization Through the Lens of Training and Regularization, October 2022. URL http:
//arxiv.org/abst 1373/1138... arXiv:2210.13738 [cs].
Xiaojic Jin, Xiaotong Yuan, Jiashi Fong, and Shuicheng Yan. Training Skinny Deep Neural Networks with Iter-
ative Hard Thresholding Methods, July 2016. URL http::/arxiv.ce// www.aiiww./aaaaaaaaae.e /ww.aa////////////// 05423. s=Xiv:1607.05444
[cs).
Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt J. Kusner. When Do Flat Minima Optimizers Work?, January 2023. URL http://arxiv org/abs////.//ooooooooooooooo aaaaabb//2222 00661. arXiv:2202.00661 [es].
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017. URL http://arxixa org/ ab////a/....../////////////////a 1112.111211 6980. arXiv:1412.6980 [cs].
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A Survey of Zero-shot Generalisation in Deep Reinforcement Learning. Journal of Artificial Intelligence Reseorch, 76:201 264, January 2023. ISSN 1076-9757. doi: 10.1613//airr.11411 URL http://arxiv.. hwtp://rr/abbb....oooom 09794. arXiv:2111.09994 [cs].
Timo Klcin, Lukas Miklautz, Kevin Sidak, Claudia Plant, and Sebastian Tschiatschck. Plasticity Loss in Deep Reinforcement Learning: A Survey, November 2024. URL http://arxxv..or//aa///11.....0 24833. 04832. [cs).
arXiv:2411 104832
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit Under-Parrmmeteriaaion Inhibits Data-Ffficient Deep Reinforcement Learning, October 2021. URL http:///arddv..rr/////0000000 arXiv:2010.14499 [cs].
wwtp/os////aa///oo// 1449// 14498.
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Tyle. Slow and steady wins the race: maintaining plasticity with hare and tortoise networks. In Proceedings of the ss International Conference on Machine Learning, volume 235 of ICML 24, pp. 26416 26438, Vienna, Austria, July 2024. JMLR.org.
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jel Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. SimBa: Simplicity Bias for Scaling Up Parameters Deep Reinforcement Learning, May 2025a. URL http:://ttare wtwpr//..rrg/b////1111 09754. arXiv:2410.09754 [cs].
Tachwan Lee, Kycongkook Sco, Jacjun Yoo, and Sung Whan Yoon. Understanding Flatness in Generative
Models: Its Role and Benefits, March 2025b. URL http:///rxiv wwtp:/wwwwwr//b////e/eee 11078. arXiv:2508 (cs) version: 1.
15

---

Published in Transactions on Machine Learning Research (07/2025)
Sergey Levine, Chelsea Finn, Trevor Darrell, and Picter Abbeel. End-to-End Training of Deep Visuomotor Policies, April 2016. URL http://arxiv.ce http://awww...rr//b////........0 arXiv:1504.000 [cs].
Alex Lewandowski, Dale Schuurmans, and Marlos C. Machado. Plastic Learning with Deep Fourier Features, October 2024a. URL
http://.or//abab/////aae 20634. arXiv:2410.20334 [cs].
Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C. Machado. Directions of Curvature Explanation for Loss of Plasticity, October 2024b. URL http://arxiv. wwwg/bs////aeeeee 00246. 4.
ar arXiv:2312.0022 26022 [cs] version:
Aodi Li, Liansheng Zhuang, Xiao Long, Minghong Yao, and Shafei Wang. Seeking Consistent Flat. Minima for Better Domain Generalization via Refining Loss Landscapes, December 2024. URL http://arxiv.
org/abs htg/abs/2412 13573. arXiv:2412.13573 [cs] version: 1.
Shiwei Liu. Learning Sparse Neural Networks for Better Generalization, volume 5, pp. 5190-5191 July 2020.
doi: 10.24963 (009633/ccii2020)) 2020/33...7 URL
org/ ottppproceedings//200//333 ISSN: 1045-0823.
Shiwei Liu, Decebal Constantin Mocanu, and Mykola Pechenizkiy. On improving deep learning generalization with adaptive sparse connectivity, June 2019. URL http::/arrxii org/ abtaa/ss//119 11626. arXiv:1906.1162 [cs].
Vincent Liu, Han Wang, Ruo Yu Tao, Khurram Javed, Adam White, and Martha White. Mcasuring and Mitigaling Interference in Reinforcement Learning. In Proceedings of The 2nd Conference Lifelong Learning Agents, pp. 781-795. PMLR, November 2023. URL https: https:///coceeengs..mom .ml. www.ppees////............. / liu23a. html. ISSN: 2640-3499.
Dor Livne and Kobi Cohen. PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learning. IEEE Journal of Selected Topics in Signal Processing, 14(4):789 801, May 2020. ISSN 1932-4553, 1941-0484. doi: N0.1109/JSTSSP2000
URL httt:///arxiv. http:://arxd..org//bb/00000 wttp://www.aab//....o.o.o 05012. arXiv:2001.055) [cs].
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning Sparse Neural Networks through $L 0$
Regularization, June 2018. URL http:://rx:: http://arxiv.oorr//ba//111. 01312. arXiv:1712.01312 [stat].
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and Preventing Capacity Loss in Reinforcement Learning, May 2022. URL http://arxiv wwwp:/iwworr/bb////00000 095660. arXiv:2204.09560 [cs]]
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Un- derstanding plasticity in neural networks, November 2023. URL /ttp:///rxiv.orr wttp://///ab....oororoooo 01486. arXiv:2303.01 [cs].
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the Causes of Plasticity Loss Neural Networks, February 2024a. URL http://arxiv. org/ abaaaaaaaaaa /2///2422111 18762. arXiv:2402.18762 [cs].
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning, July 2024b. URL
httpp://axxivvoo//// http://arxiv. www. www..ogg///aa.e..ooooo..oo..eeaa haalaba/24440 //t.///107000...... 01800. arXiv:2407.01800 [cs].
Michaël Mathieu, Sherjil Ozair, Srivalsan Srinivasan, Caglar Gulcehre, Shangtong Zhang, Ray Jiang, Tom Le Paine, Richard Powell, Konrad Zoina, Julian Schrittwieser, David Choi, Petko Georgiev, Daniel Toyama, Aja Huang, Roman Ring, Igor Babuschkin, Timo Ewalds, Mahyar Bordbar, Sarah Henderson, Sergio Gómez Colmenarejo, Aäron van den Oord, Wojeiech Marian Czarnecki, Nando de Freitas, and Oriol Vinyals. AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning, August 2023. URL org/
http://arxiv. org/ abs baabs/2308. 03526. arXiv:2308.0355 [cs).
Reginald MeLean, Evangelos Chatzaroulas, Jordan Terry, Isaac Woungang, Nariman Farsad, and Pablo Samuel Castro. Multi-Tash Reinforcement Learning Enables Parameter Scaling, March 2025. URL http: / /arxiv.. org/abs/2503. 05126. arXiv:2503.05126 [cs).
16

---

Published in Transactions on Machine Learning Research (07/2025)
Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO, November 2024. URL http: //arxiv..rrr/..eeee.........o.oa wtwp://.rrg/bb///55.00000 00662. arXiv:2405.00662 [cs].
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network
science. Nature Communiaations, 911):2383, June 2018. ISSN 2041-1723. doi: 10. 11038//41467-018-0A6
URL https: :tttpa//www.rr nature, com/ cctmm//rracclees/11117011111---0
Publisher: Nature Publishing Group.
Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers, October 2019. URL http://aree org/abs/ 1976. arXiv:1906.02773 [stat].
Michal Nauman, Michal Bortkiewiez, Piotr Milos, Tomasz Trzciáski, Mateusz Ostaszewski, and Marek Cygan. Overestimation, Overfitting, and Plasticity in Actor-Critic, the Bitter Lesson of Reinforcement Learning, June 2024a. URL http:///aaxxv..or/bbbbs.se
arXiv:2403.005/ [es].
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mitoš, and Marek Cygan. Bigger, Regularized, Optimistic: scaling for compute and sample-efiicient continuous control, December 2024b. URL http://arxiv. htttp://arrdi..or/aas////// wtgg/bs////0...o./ 16158. arXiv:2405.1613 [cs].
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The Primacy Bias in Deep Reinforcement Learning, May 2022. URL http://arxiv. wrg//bs////////////// 07802. arXiv:2205.07802 [cs].
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascamu, Will Dabney, and André Barreto. Deep Reinforcement Learning with Plasticity Injection, October 2023. URL http://arxiv.on org/ abs/2305. 15555. arXiv:2305.1555) [cs].
Johan Obando-Ceron, Aaron Courville, and Pablo Samuel Castro. In value-based deep reinforcement learning, pruned work is good network, June 2024. URL arXiv:2402.12479 [cs].
12479.
Elena-Alexande Peste. Efficiency and generalization of sparse neural networks, thesis, 2023. URL https: //research-explorerr ww..iisstaaaata.................
ac act ISSN: 2663-337X.
Matthia Sabatelli and Pierre Geurts. On The Transferability of Deep-Q Networks, November 2021. URL httt:://arxiv.or//// org. aba///a/.../////////aa/aaaaaaa/a 02639. arXiv:2110.02639 [cs].
Matthia Sabatelli, Mike Kestemont, and Pierre Geurts. On the Transferability of Winning Tickets in Non-
Natural Image Datasets, November 2020. URL http://aaxivvorrg/bb////00 2000//////0000222.2...222 05232. arXiv:2005.05233
[cs].
Remo Sasso, Matthia Sabatelli, and Marco A. Wicring. Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learning, April 2023. URL
14410. 2XXiv22205114400 [es].
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, August 2017. URL
06347. arXiv:1707.06347 [cs].
Aditya Shah, Aditya Challa, Sravan Danda, Archana Mathur, and Snehanshu Saha. A Granger-Causal Perspective Gradient Descent with Application to Pruning, December 2024. URL http://arxiv.org/ abs/2412.03038 03035. arXiv:2412.03035
[cs].
David Silver, Julian Schritlwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Dricssche, Thore Gracpel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676):354-355, October 2017. ISSN 1476-4687. doi: 00.138/nature2/// Publisher: Nature Publishing Group.
URL https: //wwww
17

---

Published in Transactions on Machine Learning Research (07/2025)
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-Task Reinforcement Learning with Context-based Representations, June 2021. URL
arXiv:2102.06177 [es].
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The Dormant Neuron Phenomenon in
Deep Reinforcement Learning, June 2023. URL http://aaxxvv..rabbba/////s 12902. arXiv:2302.12902 [cs].
Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. PaCo: Sarameter-Composittee Multi-Tast Reinforcement Learning, October 2022. URL http://arxiv.orr/abeee org/abs/2210. 11653. arXiv:2210.1166 [es].
Yee Whye Teh, Victor Bapst, Wojciech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust Multitask Reinforcement Learning, July 2017. URL
http:///axiiv org//og///////ooooooo ab2/1707 04175. arXiv:1707.04173 [cs].
Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. The Computational Limils of Deep Learning, July 2022. URL httt:://aaxiv... http:/aawxvv...oraaaa/ www.oorg///aaooooooeoe 05558, arXiv:2007.0555: [cs].
Gido M. van de Ven. On the Computation of the Fisher Information in Continual Learning, February 2025.
URL httt:///axivv org//ogg//....ooooooo abs/aaa...222//ee/ 11756. arXiv:2502.11756 [cs].
Mingean Xiang, Steven Jiaxun Tang, Qizheng Yang, Hui Guan, and Tongping Liu. AdapMTL: Adaptive Pruning Framework for Multitask Learning Model, August 2024. URL http://arxiv. wrtpi////r//bb//////// 03913. 2Xiv:2408.03913 [cs].
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-Task Reinforcement Learning with Soft Modularization, December 2020. URL http://arxiv. attp://arxiv.or//abb///aa brgg/bb///00..o// 13661. arXiv:2003.1366 [cs].
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gra- dient Surgery for Multi-Task Learning, December 2020. URL
arXiv:2001.06785 [cs].
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning, June 2021. URL http:///axxt..rrr/////111100-
arXiv:1910.10897 [cs].
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression, November 2017. URL http://arxit: wrg///s/////..oo.. 01878. arXiv:1710.0187 [stat].
Zhuangdi Zhu, Kaixiang Lin, Anil K. Jain, and Jiayu Zhou. Transfer Learning in Deep Reinforcement Learning: A Survey, July 2023. URL
arXiv:2009.07888 [cs].
18

---

Published in Transactions on Machine Learning Research (07 (07/2025)
A Training Details and Hyperparameters
This appendix details the hyperparamcters used for the experimental evaluations presented in this study, For MiniGrid, we report the IQM and Cls of the episodic return over 30 random seeds, whereas for MetaWorld, we report the TQM and Cls the mean success rate across 10 seeds. Table 2 provides a list covering the general experimental settings, architecture of the used networks, and specific hyperparameters used for MoE and MOORE. All architectures are multi-headed, with task-spocific heads. Hyperparameters were largely adoptod from Hendawy al. (2024), with modifications in the number of training epochs, number of evaluation episodes, and evaluation frequency. Table 3 outlines the training details and parameters for Meta World MT10. The full implementation is available at https: / //gggtttubbb. com/ att.oroo288 www.ssttt.patt...a.eeeeaaaaa.tet tetrrsittddiiieettee wwwpdddvvieeeeeee sarvvee_plastictty. wwt.chuu..../a.oooo.oooooeoe
B Environment Details
This appendix provides details on the MiniGrid (Chevalier-ooiveer el al., 2023) environments used in our multi-task benchmarks. We use standard environments from the MiniGrid suite, which are designed to test various capabilities such as navigation, memory, and problem-solving in partially observable grid-world settings with sparse reward. For environmental details on World, we refer the reader to Yu et al. (2021).
B.1 Composition
Our experiments use three multi-lask benchmarks MT3, MT5, and MT7, as proposed by Hendawy et al. (2024), composed as follows:
MT3: LavaGapS7-vo RedBlueDoors-6x66-e MemoryS11-v0
5T5: MT3 + DoorKey-6x6-v0 + DistShift1-v0
MT7: MT5
SimpleCrossingS992 MultiRoom-N2--5
B.2 Descriptions
Below are descriptions for each unique environment used in the benchmarks, adapted from Chevalier-Boisvert et al. (2023). In all environments specifics the size of the map SxS.
DoorKey-666-v0: The agent must pick up a key, navigate to a locked door, and open it to reach a goal square.
The agent starts in the top-left corner and must reach the goal, which is in the top-right corner, but has to avoid stepping into lava on its way. Stepping into lava terminates the episode.
DistShifti-vo:
RedBlueDoors-6x6--0 The agent is in room with two doors, one red and one blue. The agent has to open the red door and then open the blue door, in that order.
MemoryS11-v0: The agent starts in small room where sees an object. then has to go through narrow hallway, which ends in split. At each end of the split, there is an object, one of which is the same the object in the starting room. The agent has to remember the initial object and go to the matching object at the split.
SimpleCrosssingggN220o The agent has to reach the green goal square on the other corner of the room while avoiding walls. Walls run across the room either horizontally or vertically, and have M crossing points which can be safely used; the path the goal is guaranteed to exist.
Nice:RRoo-N22S4--00 This environment has a series of connected rooms with doors that must be opened get the next room. The final room has the green goal square that the agent must get N specifies the number of rooms.
19

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 2: Core experimental setup, agent architecture, and algorithm hyperparameters on MiniGrid. The choice for hyperparameters largely borrowed from Hendawy et al. (2024), while following their exact training configuration (except number of evaluation episodes).
Hyperparamcter
Value
General:
Number of environments
[3, 5, 7] step per environment
Steps per enviroument
Number of epochs
200
Steps per epoch
2000
Total number of timesteps
400000
Train frequency
2000 timesteps
Evaluation episodes
25 per task
Evaluation frequency
10000 timestops
Shared Feature Extractor:
Type
Conv2D [16, 32, 64] (2.2). (2,2), (2,2)]
Channels per Layer
Kernel Size Activations
[ReLU, ReLU, Tanh]
PPO:
Optimizer
Adam (Kingma & Ba, 2017)
Critic Loss
MSE x 10-3
Actor Learning Rate
Critic Learning Rate
x 10-3
Critic Network Hidden Size
128
Actor Network Hidden Size
128
Number of Lincar Layers Number of Output Units
2 |T| (number of tasks) |A|| for actor, 1 for critic [Tanh, Linear]
Output Activations
GAE X
0.95
Entropy Term Coefficient Clipping
0.01
0.2
Epochs for Policy
8
Epochs for Critic
1
Batch Size for Policy
256
Batch Size for Critic Discount Factor (7)
2000
0.99
Task Encoder (Jor MoE/MOORE):
Experts
[2, 3, 4
Encoder Linear Layers
Encoder Output Units
(number of experts)
Encoder Use Bias
False
Encoder Activation
Lincar
20

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Table 3: The hyperparameters and training setup used for MTMH SAC on Meta World MT10.
Hyperparameter
Value
General Training:
Total Timesteps
20000000
Batch Size
1280
Replay Buffer Size
1000000
Warmstart Steps
40000
Evaluation Frequency
200000 steps
Number of Epochs
200
Number of Updates
2000000
Number of Tasks
10
Evaluation Episodes
50 per task 500
Max Episode Steps
SAC:
Discount Factor (7)
0.99
Target Smoothing Coeff. (T)
0.005
Number of Critics
Initial Temperature (a:)
1.0
Target Q- Value Clip
5000
Actor and Crilic
Optimizer
Adam
Layer Type
Linear
Learning Rate
x 10-4
Max Gradient Norm
1.0
Network Depth
Hidden Size
400
Aclivation
ReLU [-20, 2]
Log-Ste Std Bounds
Temperature Optimizer:
Optimizer
Adam x 10-4
Learning Rate
Max Gradient Norm
None
Gradual Magnitude Pruning:
Desired Sparsity PF
95%
Pruning Frequency fp
5000 timesteps
Pruning start interval tstart
0.05x number of timesteps
Pruning end interval fond
0.80x number of timesteps
Sparsity Pr at timestep t
PF 1 (1
t-tarass lend - becart
21

---

Published in Transactions on Machine Learning Research (07/2025)
LavaGapS7-v0:
The agent has to reach the green goal square at the opposite corner of the room, and must pass through narrow gap in vertical strip of deadly lava. Touching the lava terminates the episode with zero reward.
B.3 Reward Normalization
To ensure fair comparison across tasks with inherently different reward scales, raw episodic returns are normalized with respect to the maximum achievable reward in each environment. The standard MiniGrid reward for successful task completion calculated as
- 0.9 x pttpaeee/mmx__ppiddeeettsssss
while failure results in score of We further normalize this score by performing Min-Max scaling with respect the maximum performance obtainable in each environment. We use the default maximum timesteps of each environment and estimate how many sleps optimal agent solve the environment. This normalization procedure scales the performance such that score of 1.0 represents achieving the optimal (shortest path) solution, facilitating comparisons of learning efficacy across environments with varying complexities and step horizons and addressing reward scales. Table presents the optimal steps, maximum allowed steps, and the maximum achievable reward used for the score normalization of each environment.
Table 4: Environment-specffi parameters for reward normalization. This table lists the optimal number of steps to solve each task, the maximum default permissible steps per episode, and the resulting maximum achievable raw reward score (used as the max score for normalization).
Environment Name
Optimal Steps
Max Steps
Achievable Reward
DoorKey-626-v0
11
360
0.9725
DistShifti-v0
11
252
0.9607
RedBlueDoorrs-666-00
8
720
0.9900
LavaGaps7-v0
8
196
0.9633
MemoryS11-v0
15
605
0.9777
SimpleCrrossingg9W2--0
15
324
0.9583
Mult::Room-V2-S4-00
5
40
0.8875
C Implementation Details
This appendix details the methodology and interpretation of the plasticity metrics, pruning schedule, and the sparse evolutionary training used in this work. The plasticity measures serve as correlative indicators of an agent's learning capacity and adaptability. The computation of activations and gradients for these metrics relies sampling from a plasticity replay buffer of training observations approximate expected values via sample means. Hyperparameters specific to these calculations are detailed in Table 5.
C.1 Neuron Dormancy
We adapt the dormant neuron formalization from Sokar et al. (2023). A neuron's activity assessed relative to other (non-masked) neurons in the same layer. Given imput distribution D (approximated by the plasticity replay buffer) and an activation (x) of a neuron i in layer 1 with H' neurons under input x E D, the normalized activation is
(NED) s/ Hi Neuron i is called 7-dormant for some threshold if s S. T. If H² denotes the number of dormant neurons per layer, then the dormancy ralio B, is the ratio of dormant neurons and all neurons across all
E BzeD(M((>))
HT
22

---

Published in Transactions on Machine Learning Research (07 (075)
layers in the network Lall except the final Lout
(hown) Eieton's (howc) H
By = La Mhoul Hi
A high percentage of dormant neurons suggests significant underutilization of the network's capacity, potentially hindering its ability to learn complex functions adapt to new data, key aspect of plasticity. For the ReDo method, dormant neurons were reinitialized at specific time intervals fa and a threshold т, determined through the hyperparameter sweeps in Appendix E
C.2 Trace of the Fisher Information Matrix
The Fisher Information Matrix (FIM) F quantifies the sensitivity of a model's output (e.g., the policy) to changes in the parameters (Klein et al., 2024; Falzari & Sabatelli, 2025; Ven, 2025). For a policy æ, its Fisher trace is given by
Tr(F) = Be [[ [(a)(())
The trace of the FIM can be viewed as a measure of the policy's sensitivity to parameter perturbations. A very high or persistently increasing trace might indicate that the policy is in "sharp" region of the loss landscape, making it brittle to small changes and potentially indicative of overfitting or optimization instability. Conversely, a lower, stabilized trace, as observed in our pruned agents (see Section 4.2), can suggest convergence to 'flatter" minima, implying a more robust policy that is less sensitive to parameter variations and more capable of sustained learning or adaptation.
C.3 Effective Rank
For feature matrix & (e-g., a shared feature extractor) with d singular values as sorted descendingly, the effective rank at tolerance 8 is
E 2 21-
sranks min
Σd E
The effective rank measures the dimensionality of the space spanned by the features. A low effective rank suggests reprosentation collapse, where learned features highly correlated and less diverse, limiting network's is ability to represent various information. Conversely, high elfective rank implies richer, more diverse set of feature representations, implying greater capacity to learn and distinguish between inputs.
C.4 Gradual Magnitude Pruning (GMP)
We implement the pruming schedule proposed by Zhu & Gupta (2017), which progressively increases network sparsity during training. At regular pruning intervals (defined by a pruning frequency fp), connections (weights) with the smallest absolute magnitudes are masked (set to zero). This process continues until a target sparsity level Pt is achieved for the current training step The sparsity level Pt. follows a cubic growth schedule, gradually increasing from an initial sparsity at fatart to final target sparsity PF at tendi
0
if l < lsttar,
Pt
PF 1 -
(1
f-fatar Fand Satare
if Estart S t S fends
PF
if tend-
This schedule allows the network to adapt incrcasing levels of sparsity rather than undergoing abrupt structural changes. The specific values for PF, pruning frequency, fetart, and tend used in our experiments were determined through ablation studies (see Appendix D).
23

---

Published in Transactions on Machine Learning Research (07 (0/5))
C.5 Sparse Evolutionary Training (SET)
The Sparse Evolutionary Training (SET) mechanism, inspired by Mocanu al. (2018), maintains constant overall network sparsity p - [ throughout training, where | is the total number of non-zcro weights and Nostl is the total number of paramcters in the sparsified layers. At prodefined evolution intervals, a fraction ε of the existing connections with the smallest absolute magnitudes |eyy are pruned. To preserve the sparsity level 8, an equivalent number of new connections, Naew = e || simultaneously regrown. These new connections are typically introduced randomly at locations within the network that currently have zero weight, allowing exploration of novel sparse topologies. The initial sparse connectivity for SET is established using an Brdös-Rényi-Koe Ç. SET was applied to all linear layers in our models, with Ç. and evolution frequency determined via ablations (Appendix D), while the fixed sparsity was kept at 95% for consistency with the GMP sparsity.
Ilo [|gurrett are
(ERK) scheme, controlled by parameter
Table 5: Configuration details for gradual magnitude pruming, sparse evolutionary training, and the alternative plasticity-chhnccnng methods (ReDo, Reset, and Weight Decay) evaluated.
Hyperparameter
Value
Gradual Magnitude Pruning:
Desired Sparsity Pr
95% 500 timesteps
Pruning Frequency fp Pruning start interval Istart Pruning end interval fend
0.05x number of timesteps 0.80x mumber timesteps
Sparsity Ps timestep
PF
L-batai Lend- Cand-latact
Prune Bias
False
Pruned Layers
[Con2DD, Linear]
Sparse Evolutionary Truining:
Sparsity
95%
density
11
Sparsity Distribution Ç
0.3
Evolution Frequency
2000 timesteps
Plasticity:
Plasticity Buffer Max Size
100000
ReDo (Sokar al., 2023) Frequency fa
5000 timesteps
Dormant Neuron Threshold
0.001
Dormant Activation Batch Size
1024
Fisher Trace Batch Size
1024
Effective Rank Batch Size
1024
Effective Rank Target Reset Frequency fr
Shared Feature Extractor
100000
Number of Resets m
2
Reset Target Layers
Output
Weight Decay Coefficient À
1 10-6
24

---

Published in Transactions on Machine Learning Research (07 (075)
D Sparse Methods Hyperparameters
This appendix details the hyperparameter selection for GMP, with tuning experiments conducted on MTPPO with the MT5 benchmark. Figure illustrates GMP ablations: the left subplot shows that initiating pruming early (e-g., at 5% of training) and concluding by 80% is favorable for the schedule window [Istart, lena]; the center subplot indicates stable performance across moderate pruning frequencies fp: and the right subplot explores various final target sparsities p. Figure 9 presents SET tuning: the left subplot suggests moderate evolution connection (e.g., 11-15) perform well; the center explores sparsity distribution parameters ( and be beneficial. These ablations informed the hyperparameter
the right shows less frequent evolution choices used in our main experimental evaluations. All tuning experiments were run with the original training configuration, outlined in Table 2 for 30 seeds and 200 epochs. For visual purposes, we omit the confidence intervals of the less successful runs to avoid cluttering and only include them for the best hyperparameter run.
J005, 0.
[..00..8
8.100
f. = 500
4. 1000
- 0.56
2-0.00 0,50
p- 0.00
(02,00M)
D4) 1000,
Jn
0.07
jo.6, 0.7]
(0.06,
J.o5, L.j
10000
p
0.8
0.5
Nornulkosi 108
0.00
0.6
0.6
0.2
0.2
0.2
50
150
Epoch
Epoch
Egoch
Figure 8: Hyperparameter ablation for Gradual Magnitude Pruning (GMP) on MTPPO with the MT5 benchnarkk (Left) Varying pruning schedule window [Astart; bend]. (Center) Different pruning frequencies fp. (Right) Various final target sparsity levels p.
0=0.1 [
03
C-02
Evulos Froq.
0.4
Evalve Froq.
0.8
Normolted IQM
0.4
0.2
0.2
0.2
50
100
150
200
50
100
150
200
50
100
150
200
Epoch
Epech
Epoch
Figure 9: Hyperparameter ablation for Sparse Exolutionary Training (SET) on MTPPO with the MT5 benchmark. (Left) Varying evolution connection percentage s. (Center) Different sparsity distribution parameters Ç. (Right) Different evolution frequencics (episodes).
25

---

Published in Transactions on Machine Learning Research (07 (025)
E
Comparison Methods Hyperparameters
This appendix outlines hyperparameter considerations for the comparison methods evaluated in this work. All luning experiments were primarily conducted on the MTPPO architecture using the MT5 benchmark to establish robust configurations. All tuning experiments were run with the original training configuration, outlined Table for 30 seeds and 200 epochs. For visual purposes, we omit the confidence intervals of the less successful runs to avoid cluttering and only include them for the best hyperparameter run.
the most critical hyperparameter. Lower values, specifically 0.001 and 0.0001, demonstrated significantly better performance compared to higher thresholds or omitting the threshold entirely. The ReDo application frequency fa exhibited less sensitivity, with a moderate frequency (e.g., fa = 5000 steps) performing well.
For ReDo, tuning (Figure 10) highlighted the dormancy threshold
For the Reset mechanism, we investigated the impact of the reset frequency fr (how often resets occur) and the maximum number of times m specific layers are reset throughout training, as illustrated Figure Our ablations indicated that a reset frequency of fr = 100k timesteps with a maximum of m resets per targeted layer (black line in both subplots) often provided good balance between promoting plasticity and avoiding excessive training instability.
The Weight Decay (WD) coefficient À was selected from a standard range. As shown in Figure 12, very small coofficients (c.g., À = 10-6, black line) resulted overall the best performance.
= 0.001
7 0.01
fa 5000
Ja = 2500
0.0
== 0.0001
fa 500
fa = 10000
1000
fa 25000
Normalized IQM
0.6
0.6
0.4
0.4
0.2
0.2
50
100
150
200
50
100
150
200
Epoch
Epoch
Figure 10: Hyperparameter ablation for ReDo on MTPPO with the MT5 benchmark. (Left) Varying dormancy threshold т. (Right) Different ReDo application frequencies fa. The black line (r 0.001, fa 5000) shows the best performance.
26

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
f. 100A
f. 80k
2. 100%
f- 80A
fr 1204
f. 60k
IT1 = 3r 60k
m = 4, f. 406
Normalizael IQM
0.4
0.4
0.2
0.2
50
100
150
200
50
100
150
200
Epoch
Epoch
Figure 11: Hyperparameter ablations for the Reset mechanism on MTPPO with the MT5 benchmark. (Teft) Varying reset frequency fr (with m = 2). (Right) Varying maximum number of resets m. The black line (Sr 100k, = 2) indicates the best performance.
À 10-8
À 10-8
À 10-4 A 10-1
À = 10-3
A 10-2
0.8
Normalized IQM
0.6
0.4
0.2
0
50
100
150
200
Epoch
Figure 12: Hyperparameter ablation for Weight Decay (WD) on MTPPO with the MT5 benchmark, showing performance for different decay coefficients The black line (A 10-9) shows the best performance.
27

---

Published in Transactions on Machine Learning Research (07 (075)
F
Detailed Learning Curves
This appendix provides the complete learning curves for various agent configurations, complementing the aggregated final performance data presented in Table 1. These plots illustrate the training progression across all evaluated MTRL architectures (MTPPO, MoF, MOORE) on the MT3, MT5, and MT7 benchmarks. Additionally, Figure 13 presents the results obtained on the continuous control MetaWorld MT10 benchmark that complement the findings described in Section 4.1.
Figure 14 displays learning curves comparing our primary sparsification methods (Gradual Magnitude Pruming, GMP, at 95% sparsity, and Sparse Evolutionary Training, SET, at 95% sparsity) against dense baseline and explicit plasticity-idducin interventions (ReDo and Reset).
Figure 15 similarly presents learning curves, in this case contrasting the same sparsification methods (GMP 95% and SET 95%) against a dense baseline and common implicit regularization techniques (Weight Decay WD, and LayerNorm).
In all subplots within Figure 14 and Figure 15, the horizontal dashed line indicates the aggregated performance of single-task PPO (ST PPO) agents. Each ST PPO agent was trained separately on single environment from the respective benchmark for the full 400,000 timesteps (the same total duration ion as the multi-lask agents) across 30 runs. Consequently, this ST PPO performance should be viewed as a potentially near-maximal reference point from a single-task perspective, as multi-task agents faced the more challenging scenario of learning all lasks within a benchmark concurrently using the same total number timesteps.
Dense
GMP-Actor
GMP-Both
Mean Success Rate
0.6
0.4
0.2
50
100
150
200
Epoch
Figure 13: Final success rates on MetaWorld MT10 for different pruning configurations. Selective pruning of the actor network (GMP-actor) outperforms both the dense MTMH-SAC baseline and the globally pruned model (GMP-all), achieving a final success rate of 81%. This supports the hypothesis that actor-only pruming enhances performance by improving efficiency while preserving critical representational capacity in the critic.
28

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Dense
GMP
SET
ReDo
Reset
ST PPO
MT3
MT5
MT7
0.8
0.8
0.8
MTPPO
0.6
0.6
0.6
0.4
0.2
0
0.8
0.8
0.8
0.6
0.6
0.6
MoB
0.4
0.2
0
0.8
0.8
0.8
MOORE
0.6
0.6
0.6
0.4
0.4
0.2
0.2
50
100
150
200
50
100
150
200
50
100
150
200
Epoch
Figure 14: Learning curves (Normalized IQM) comparing sparsification methods (GMP 95%, SET 95%) with explicit plasticity-inducing interventions (ReDo, Reset) and a dense baseline. Results are shown for MTPPO, MoE, and MOORE architectures across MT3, MT5, and MT7 benchmarks. The dashed line represents single-task PPO performance.
29

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Dense
GMP
LayerNorm
PPO
MT3
MT5
MT7
0.8
0.8
0.8
MТРРO
0.6
0.6
0.6
0.4
0.2
0
0.8
0.8
0.8
0.6
0.6
0.6
MoB
0.4
0.2
0
0.8
0.8
0.8
MOORE
0.6
0.6
0.6
0.4
0.4
0.2
0.2
50
100
150
200
50
100
150
200
50
100
150
200
Epoch
Figure 15: Learning curves (Normalized IQM) comparing sparsification methods (GMP 95%, SET 95%) with implicit regularization techniques (Weight Decay, LayerNorm) and a dense baseline. Results are shown for MTPPO, MoE, and MOORE architectures across MT3, MT5, and MT7 benchmarks. The dashed line represents single-task PPO performance.
30

---

Published in Transactions on Machine Learning Research (07 (07/2025)
G Detailed Plasticity Metrics
This appendix provides a comprchensive view of the plasticity metric evolutions across different benchmarks, complementing the primary analysis presented in Section 4.2 (which predominantly features results from the MT5 benchmark, also shown here for completeness as Figure 17 and Figure 20). The following figures illustrate the dynamics of Fisher Trace, Effective Rank, Actor Dormant percentage, and Critic Dormant percentage for all evaluated MTRL architectures (MTPPO, MoE, MOORE).
Figures 16, 17, and 18 compare the effects of sparsification methods (GMP and SET) against. explicit plasticity-indueing interventions (ReDo and Reset) and dense baseline, the MT3, MT5, and MT7 benchmarks, respectively. While specific magnitudes vary, general trends such as SET's strong effect on reducing dormancy and GMP's characteristic Fisher Trace dynamics are often observable across benchmarks, though interactions with architecture (especially MOORE) can modulate these effects.
Similarly, Figures 19, 20, and 21 present a comparison of the same sparsification methods (GMP and SET) against implicit regularization techniques (Weight Decay- WD, and LayerNorm) and dense baseline, for the MT3, MT5, and MT7 benchmarks, respectively.
Figure 22 presents the plasticity profiles MetaWorld MT10, using the MTMH SAC architecture. The Critic MER and Critic Dormant neuron percentages are nearly identical across all three configurations, with dormancy already close to zero even for the dense bascline, leaving little room for improvement. The actor metrics, however, show clear distinctions. Selectively pruning the actor resulted in maintaining the lowest percentage of actor dormant neurons and an increased mean effective rank compared to both the dense baseline and the globally pruned agent. The Fisher Trace remained highly volatile for all methods and showed discernible pattern.
Dense
GMP
SET
ReDo
Reset
Fisher Trace
Effective Rank
Actor Dormant [%]]
Critic Dormant [%]
40
300
МТРPO
20
30
2800
15
240
10
20
220
150
320
125
300
30
MoE
280
260
15
20
50
10
0
150
340
30
MOORE
320
25
15
100
300
10
260
2:0
220
50
200
100
100
100
Epuch
Figure 16: Comparative plasticity dynamics of sparse methods (GMP, SET) versus explicit plasticity-inducing interventions (ReDo, Reset) and a dense baseline, across MTPPO, MoE, and MOORE architectures on the MT3 benchmark. Metrics include Fisher Trace, Effective Rank, and percentage of Actor and Critic Dormant Neurons.
31

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Deuse
GMP
SET
ReDo
Rewel
Fisher Trace
Effective Rank
Actor Dormant [%
Critic Dormant [%)
MTPPO
150
310
320
25
20
30
100
300
120
260
210
380
30
100
360
23
30
MoE
340
20
320
20
40
300
240
10
120
23
MOORE
100
400
15
20
350
40
10
20
50
100
150
200
50
100
150
200
50
100
150
200
0
50
100
150
200
Epoch
Figure 17: Comparative plasticity dynamics of sparse methods (GMP, SET) versus explicit plasticity-inducing interventions (ReDo, Reset) and dense baseline, across MTPPO, MoF, and MOORE architectures the MT5 benchmark. Metrics include Fisher Trace, Effective Rank, and percentage of Actor and Critic Dormant Neurons.
Dense
GMP
SET
ReDo
Reeet
Fisher Trace
Effective Rank
Actor Dormant [%]
Critie Dormant [%)
200
MTPPO
150
400
20
30
25
350
20
10
15
50
10
100
25
MoE
450
20
30
20
400
10
0
1350
120
25
MOORE
100
460
20
15
440
10
40
120
400
50
100
150
200
50
100
150
200
50
100
150
200
0
50
100
150
200
Epuch
Figure 18: Comparative plasticity dynamics of sparse methods (GMP, SET) versus explicit plasticity-inducing interventions (RcDo, Reset) and dense bascline, across MTPPO, MoE, and MOORE architectures the MT7 benchmark. Metrics include Fisher Trace, Effective Rank, and percentage of Actor and Critic Dormant
Neurons.
32

---

Published in Transactions on Machine Learning Research (07 (075)
Dense
GMP
SET
WD
LayerNorm
Fisher Trace
Effective Rank
Actor Dormant [%
Critic Dormant [%)
350
25
MTPPO
150
300
30
200
10
150
5
10
120
MoE
100
300
20
30
15
20
200
10
160
30
16
MOORE
300
23
14
100
20
12
15
10
250
10
50
100
150
200
50
100
150
200
50
100
150
200
0
50
100
150
200
Epoch
Figure 19: Comparative plasticity dynamiis of sparse methods (GMP, SET) versus regularization techniques (Weight Decay, LayerNorm) and dense baseline, across MTPPO, MoF, and MOORE architectures the MT3 benchmark. Metrics include Fisher Trace, Effective Rank, and percentage of Actor and Critic Dormant Neurons.
Demse
GMP
SRT
WD
LayerNorm
Fisher Trace
Effective Rank
Actor Dormant [%]
Critie Dormant [%)
400
23
30
150
MTPPO
350
25
100
300
15
20
250
10
15
150
100
30
400
23
350
MoE
20
300
40
120
10
2800
10
MOORE
100
25
18
350
15
12
40
300
10
10
250
50
100
150
200
50
100
150
200
50
100
150
200
0
50
100
150
200
Epuch
Figure 20: Comparative plasticity dynamics of sparse methods (GMP, SET) versus regularization techniques (Weight Decay, LayerNorm) and dense bascline, across MTPPO, MoE, and MOORE architectures the MT5 benchmark. Metrics include Fisher Trace, Effective Rank, and percentage of Actor and Critic Dormant
Neurons.
33

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Dense
GMP
SET
LayerNorm
Fisher Trace
Effective Rank
Actor Dormant [%]
Critic Dormant [%)
200
30
MTPPO
100
20
25
350
15
20
100
300
10
10
100
450
25
MoE
400
30
350
10
250
10
120
25
18
MOORE
16
20
14
60
400
12
10
50
200
100
100
100
Epuch
Figure 21: Comparative plasticity dynamics of sparse methods (GMP, SET) versus regularization techniques (Weight Decay, LayerNorm) and a dense baseline, across MTPPO, MoE, and MOORE architectures on the MT7 benchmark. Metrics include Fisher Trace, Effective Rank, and percentage of Actor and Critic Dormant Neurons.
Dense
GMP-Both
GMP-Actor
Fisher Trace
Actor MER
Critic MER
Actor Dormant [%]
Critic Dormant
[%]
-10
380
12
340
0.8
375
0.6
370
300
8
0.4
0.2
260
50
100
150
200
0
100
150
200
50
100
150
200
30
100
150
200
0
100
150
200
Epoch
Figure 22: Plasticity dynamics on MetaWorld MT10. Selective pruning of the actor (GMP-Actor) leads to a decrease in dormant neurons and an increase in the mean effective rank compared to both the dense MTMH-SAC bascline and the globally pruned model (GMP-Both). In contrast, the critic metries and Fisher Trace show minimal clear patterns.
34

---

Published in Transactions on Machine Learning Research (07 (07/2025)
H Gradual Magnitude Pruning Performance Across Sparsity Levels
This appendix presents learning curves for Gradual Magnitude Pruning (GMP) across various target sparsity levels (Dense, 80%, 95%, 99%) for all architectures and benchmarks (Figure 23). These results show that while 80% and 95% sparsity generally yield strong performance, often matching or exceeding dense baselines especially for MTPPO and MoE, the optimal sparsity level is architecture and benchmark-specifcc. Notably, for the MOORE architecture, 99% sparsity leads to performance degradation later in training (visible Figure 23, bottom row). This performance drop correlates with significant representational rank collapse, illustrated by the sharp decline in effective rank for MOORE (Figure 24), indicating loss of representational diversity under extreme pruning in this specific architecture.
Dense
95% Sparsity
80% Spansity
99% Sparsity
ST PPO
MT3
MT5
MT7
0.8
0.8
MTPPO
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.8
0.8
MoE
0.6
0.6
0.4
0.4
0.2
0.2
0.8
0.8
MOORE
0.6
0.6
0.4
0.4
0.2
0.2
0
50
100
150
200
50
100
150
200
50
100
150
200
Epoch
Epoch
Epoch
Figure 23: Normalized aggregate returns for agents under different GMP sparsity levels (Dense, 80%, 95%, 99%) across the MT3, MT5, and MT7 benchmarks for MTPPO (top row), MoE (middle row), and MOORE (bottom row) architectures. The dashed line represents single-task PPO performance.
35

---

Published in Transactions on Machine Learning Research (07/2025)
Dense
95% Sparsity
99% Sparsisy
Fisher Trace
Effective Rank
Actor Dormant [%
Critic Dormant [%]]
100
МТPPO
600
350
40
400
300
15
30
200
250
20
13
10
Epoch
Figure 24: Plasticity metrics for MOORE on the MT5 benchmark under different GMP sparsity levels, illustrating the rank collapse at 99% sparsity, characterized by sudden increase in the Fisher Trace and neuron dormancy, and a sharp drop in the effective rank.
36


Reviews:
[{"review_id": "hIkqy3fBvY", "reviewer": "Reviewer_BVba", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written, it is easy to follow and understand. The key idea presented in the paper, gradual pruning, is easy to implement and does not require too much additional computation. The experiments are clearly designed and directly show if the claims made in the paper are true or not. The results support the first claim.\n\nDespite these advantages, the paper has some fatal flaws.\n1. **Performance results are not statistically significant.** The second claimed contribution of the paper is that gradual pruning improves performance over dense networks and some plasticity-preserving. This claim is based on the results presented in Table 1. However, Table 1 shows no statistically significant difference between the performance of dense or ReDo or Resets or gradual pruning. This means that claim 2 is not supported by the evidence provided.\n2. **Baselines are not tuned.** A part of the second claim of the paper is that gradual pruning performs better than plasticity-preserving methods ReDo and resets. To show this, Redo and resets must be tuned for these environments, and a wide range of their hyper-parameters should be tested. Table 4 shows that only one value of their hyper-parameters is used. \n\nIn Addition, there are a few minor issues:\n1. The second line of the title only contains \"ing\"; that is not aesthetically pleasing; consider moving the words \"Reinforcement Learning\" to the second line.\n2. Chung et al. (2024) showed that maintaining orthogonality in weight matrices is a very effective technique for maintaining plasticity. I think the good performance of MOORE and the bad effect of ReDo and resets could be because MOORE encourages weights and representations to remain orthogonal. I suggest adding a discussion to contextualize the work of Chung et al. (2024).\n3. The second half of the third contribution stated in the introduction (\"revealing insights into when and why sparsity interventions can complement existing architectural configurations for multi-task learning\") is not well supported. After reading the paper, I don't clearly see where the insights for \"when and why sparsity interventions can complement existing architectural configurations for multi-task learning\" are presented. Perhaps removing this part of the third contribution would be a good idea.\n\nChung et al. Parseval Regularization for Continual Reinforcement Learning. NeurIPS 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper claims to make three main contributions, which are clearly stated in the abstract and the introduction. \n1. In multi-task RL, gradual pruning mitigates some of the correlates of plasticity loss, such as dormant neurons.\n2. Gradual pruning improves performance over dense networks and some plasticity-preserving methods like ReDo.\n3. The effect of gradual pruning on plasticity and performance is architecture-dependent. And provide insights on when and why sparsity interventions help.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Two major changes are needed to accept this paper:\n1. Make results statistically significant. The authors already do 30 runs, but to show the difference between algorithms, more runs are needed, perhaps 100. The enviroments are simple and the networks are small, I think it would not require too much additional compute. \n2. Tune baselines. More hyper-parameter setting for ReDo and resets need to be tested to claim that they perform worse than gradual pruning.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rR97JY4esX", "reviewer": "Reviewer_8sLK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- **Novel research direction:** The application of gradual magnitude pruning to multi-task reinforcement learning addresses a relatively unexplored intersection, providing new insights into how sparsity interventions can mitigate plasticity loss in MTRL settings where representational flexibility demands are particularly high.\n- **Comprehensive architectural analysis:** The paper provides thorough empirical evaluation across three distinct MTRL architectures (MTPPO, MoE, MOORE), revealing architecture-dependent effects and providing clear insights into when and why sparsity interventions are most beneficial for different network designs.\n\n**Weaknesses:**\n- **Insufficient methodological analysis:**\n    - The paper lacks justification for gradual pruning over alternative sparsity strategies. Critical questions remain unanswered: What happens if models are trained sparse from initialization? Are there benefits to different pruning schedules?\n    - No exploration of combining pruning with weight decay, despite weight decay being a standard regularization technique. Given the computational overhead of gradual pruning, stronger evidence is needed that it provides benefits beyond simpler alternatives like weight decay.\n    - Missing ablation studies on key hyperparameters and pruning design choices.\n- **Limited experimental scope:**\n    - MiniGrid environments are relatively simplistic and may not adequately validate the approach's effectiveness for complex real-world scenarios. The evaluation would benefit from at least one experiment on more challenging multi-task benchmarks (e.g., Multi-task Meta-World) to demonstrate broader applicability.\n- **Incomplete related work:** The paper would benefit from more comprehensive discussion of relevant literature in network pruning for RL and recent advances in plasticity preservation methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates gradual magnitude pruning as a method to mitigate plasticity loss in multi-task reinforcement learning (MTRL). The work addresses how deep RL agents lose adaptability over time due to gradient interference, representational collapse, and neuronal saturation. \n\nThe authors make three key contributions: \n- demonstrating that gradual magnitude pruning (up to 95% sparsity) effectively mitigates plasticity loss. \n- showing that pruning-induced plasticity benefits directly translate to better multi-task performance. \n- revealing that pruning benefits vary across MTRL architectures (MTPPO, MoE, MOORE). \n\nThe evaluation uses PPO agents on MiniGrid environments across three multi-task architectures, demonstrating that network sparsity can simultaneously achieve compression and plasticity enhancement for more efficient and adaptable RL agents.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested Changes:**\n\n• **Methodological justifications**: The authors need to provide clearer empirical support for their design choices:\n  - Compare gradual pruning versus training sparse from initialization to demonstrate why the gradual approach is necessary\n  - Analyze key hyperparameters such as initial pruning epochs, final sparsity targets, and pruning intervals.\n  - **Investigate combining gradual pruning with weight decay to determine if gradual pruning provides benefits beyond standard regularization.**\n  - Recent work has shown that architectures with normalization layers (e.g., layer normalization, batch normalization) can inherently reduce plasticity loss issues [1,2,3]. It would be valuable to test architectures with these normalization layers to determine whether gradual pruning still provides additional benefits when plasticity problems are already reduced by normalization.\n\n• **Enhanced experimental validation**: The authors should include at least one experiment on a more complex multi-task benchmark (e.g., Multi-task Meta-World or similar) to demonstrate that the findings generalize beyond the relatively simple MiniGrid environments. This is essential to validate the practical applicability of the approach for real-world scenarios.\n\n• **Comprehensive related work section**: Expand the literature review to include more thorough coverage. \n   - pruning / dynamic network related [4,5,6]\n   - plasticity-related [7,8,9]\n\n[1] CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity, Bhatt et al, ICLR'24.\n\n[2] Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control, Nauman et al, NeurIPS'24.\n\n[3] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning, Lee et al, ICLR'25.\n\n[4] RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch, Tan et al, ICLR'23.\n\n[5] The State of Sparse Training in Deep Reinforcement Learning, Graesser et al, ICML'22.\n\n[6] Neuroplastic Expansion in Deep Reinforcement Learning, Liu et al, ICLR'25\n\n[7] On warm-starting neural network training, Ash et al, NeurIPS'20. (original reset paper).\n\n[8] Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks, Lee et al, ICML'24.\n\n[9] Plastic Learning with Deep Fourier Features, Lewandowski et al, arXiv'24.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aEtUOc8Dgn", "reviewer": "Reviewer_GspR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- This paper is quite concise and clear and therefore easy to read\n- The set of comparisons being run across multiple model architectures is good practice\n- Comparisons against alternative regularizers such as Weight Decay, ReDo, and Reset are appreciated\n- The application of sparsity for MTRL is a nice, and seemingly promising, direction of research\n\n**Weaknesses**:\n- The utility and novelty of the proposed method is unclear. Not only does it appear relatively ineffective for MOORE models, even in other architectures the performance is often matched or exceeded by an alternative and simple regularizer. Furthermore, the gradual pruning method is precisely taken from an existing work.\n- The proposed method is not theoretically justified or explained. Therefore the method seems to suddenly appear without much reasoning, justification, or explanation of its particular form and whether this sparsification schedule is important or not.\n- The method's drawbacks are not discussed in sufficient detail. For example, this method requires prior setting of the number of epochs of training. Furthermore, the choice of a gradual sparsificaion vs a fixed sparsification is never discussed or explored. Wave-like dynamics appear in the plasticity dynamics (Figure 2) and it is unclear if these are due to training or the sparsification/pruning schedule.\n- The ineffectiveness of this method when applied to MOORE is discussed but never explicitly investigated and explained. Therefore, any application to alternative architectures is somewhat left as an uncertain exporation for practitioners.\n- The results, when examined more closely, seem to suggest that the hyperparameter search may have been far too restricted in order for these results to be fair for all methods and models. Training curves in particular seem to suggest that the sparsity levels chosen might be too large in some cases and perhaps too small in other. It is also unclear whether the hyperparameters of baselines were sufficiently tuned.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper demonstrates the impact of gradual magnitude pruning on multi-task reinforcement learning (MTRL) problems. This approach is applied to a range of architectures including networks with shared backbones, mixture of experts (MoE), and mixture of orthogonal experts (MOORE) models. Overall, it is shown that gradual plasticity can improve the achieved accuracy in MTRL problems. It can also be shown that, when measuring indicators of plasticity (such as neuron dormancy, effective rank, and the trace of the Fisher Matrix) gradual magnitude pruning can lead to more favourable regimes (lower dormancy, higher rank, lower Fisher trace). One major exception is the case of MOORE architectures which often showed reduced accuracies after training and unfavourable changes in Rank and the related Neuron Dormancy.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would request a number of significant changes to this paper. Note, the scale of changes is so large that I would advise that this paper be rejected and either completely overhauled or presented at a small workshop event. Nonetheless, I have my requested changes below. I hope that this review is clear and helpful in providing input and improving the work.\n\n- **Textual**: The text of this work, especially the abstract, introduction, and discussion, should be more clear on the limitations and drawbacks of this work. In particular, the abstract and introduction make no mention of the failure of this method when applied in some cases (e.g. MOORE), and the discussion only lightly touches on the drawbacks of the specifically tested sparsification method.\n- **Methods/Theory**: The method itself requires far more explanation and justification than is currently provided. At present the sparsification schedule is described without any mention of why this form is appropriate, and it is only rather indirect in stating that this specific sparsification schedule is exactly that of Zhu & Gupta (2017). Furthermore, the reasoning or theoretical basis for why one should expect that this sparsification schedule (or sparsification in general) should result in benefits for neuron dormancy, effective rank, or Fischer Trace are also missing.\n- **Experiment Hyperparameters**: It is unclear how the hyperparameters for the various methods were selected. For example, the weight sparsification values of 80/95/99 seem to be rather arbitrary. More importantly, the hyperparameters of the baseline methods do not show whether alternatives were tested. For example the application of ReDo and Reset are carried out at predetermined timesteps but this could have been further optimised. Weight decay also appears to have a single value used (10^(-3)) but how this was selected? This should have ideally been tested across a range and perhaps even tuned for each architecture.\n- **Performance Curves**: From examination of the performance curves (e.g. Figures 3 and 4) it appears that the training of the models was not carried out to convergence (they still appear to be training at the end of the 100 epochs). This makes it difficult to know which of the compared methods for MTRL ultimately converge to the highest accuracy. This means that the conclusions of outperformance of this method are also still somewhat open to change. Furthermore, Figure 3 appears to show that the sparsification levels chosen (80/95/99) might in fact be far too high in some cases. For example, it appears that for the MOORE models in the MT3 task that all of these sparsity levels are in fact ‘too sparse’. It is not clear how these sparsity values were chosen but it would be good to see the performance of networks over a much wider range of sparsity values.", "broader_impact_concerns": "- None -", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)