Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning
Authors: Guozheng Ma
Lu Li
Zilin Wang
Li Shen
Pierre-Luc Bacon
Dacheng Tao
Venue: ICML 2025 oral (2025)
Keywords: Deep Reinforcement Learning
Network Sparsity
Scaling
Plasticity Loss
Regularization
Primary area: reinforcement_learning->deep_rl
TL;DR: Integrating network sparsity into the most advanced architectures can further unlock the scaling potential of DRL models and meanwhile effectively mitigating optimization pathologies during scaling.

Abstract: Effectively scaling up deep reinforcement learning models has proven notoriously difficult due to network pathologies during training,  motivating various targeted interventions such as periodic reset and architectural advances such as layer normalization. Instead of pursuing more complex modifications, we show that introducing static network sparsity alone can unlock further scaling potential beyond their dense counterparts with state-of-the-art architectures. This is achieved through simple one-shot random pruning, where a predetermined percentage of network weights are randomly removed once before training. Our analysis reveals that, in contrast to naively scaling up dense DRL networks, such sparse networks achieve both higher parameter efficiency for network expressivity and stronger resistance to optimization challenges like plasticity loss and gradient interference. We further extend our evaluation to visual and streaming RL scenarios, demonstrating the consistent benefits of network sparsity.

Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning
Guozheng Ma
Lu Li
Zilin Wang"
Li Shen
Pierre-Luc Bacon 23
Dacheng Tao
Fumanoid Walk & Run
Dog Trot Run
Abstract
Effectively scaling up deep reinforcement learn- ing models has proven notoriously difficult due to network pathologies during training, motivat- ing various targeted interventions such as periodic reset and architectural advances such as layer nor- malization. Instead of pursuing more complex modifications, show that introducing static network sparsity alone can unlock further scal- ing potential beyond their dense counterparts with -fate-of-the-ar architectures. This is achieved through simple one-shot random pruning, where predetermined percentage of network weights are randomly removed once before training. Our analysis reveals that, in contrast to naively scaling up dense DRL. networks, such sparse networks achieve both higher parameter efficiency for net- work expressivity and stronger resistance to opti- mization challenges like plasticity loss and gradi- ent interference. We further extend our evaluation to visual and streaming RL scenarios, demonstrat- ing the consistent benefits of network sparsity. Our code is publicly available at GitHub ?
DDPG
0006
topands
132
Hodd Sox
Vodil Sox
Descs Network (Dely Scxling Depth:
Sparte Nettvers (Dal Scaling Dopel)
Desca Network (Dely Scaling Watrs
Sparte Nettvers (Day Sealing Whlth
Desss Neowork Dostay Wolh and Unpelt
Sparie Netwvea (Beaing Wiltt and Depilo
Figure I. Model scaling trends of * * * sparse versus eoedense
networks on four hardest DMC tasks using SimBa architecture with SAC and DDPG. Beyond a 17M baseline SimBa network, dense networks (dashed lines) exhibit degrading performance with increased scale. In contrast, introducing sparsity while increasing model size (solid lines) can unlock further scaling potential.
1. Introduction
Deep neural networks have demonstrated consistent im- provements with increased scale in supervised learning tasks, where larger models reliably yield better results. How- ever, this scaling pattern breaks down in deep reinforcement learning (DRL), where increasing model size often leads to deteriorating performance (Nauman et al., 2024a;b). This limited scalability of DRL models can be largely attributed severe optimization pathologies that emerge during train- ing (Nikishin, 2024; Nauman al., 2024a), with these chal- lenges becoming increasingly pronounced as model size grows (Ceron et al., 2024a;b). Specifically, notable patho- logical behaviors include plasticity loss (Nikishin et al.,
2022; Sokar et al., 2023), parameter under-utilization (Ku- mar ct al., 2021), capacity collapse (Lyle et al., 2022a), etc.
Recent work has proposed various dynamic approaches to address these pathologies, aiming break through the seal- ing barrier of DRL. models (Lyle al., 2022a; Klein et al., 2024). Among these advances, periodic Reset and its vari- ants stand out representative approach that enhances model scaling capabilities by mitigating plasticity loss and pathological behaviors through strategic re-initialization of the entire network or specific neurons (Nikishin et al., 2022: Sokar al., 2023; Xu et al., 2024; Dohare al., 2024). While effective, these methods require drastic in- terventions in optimization dynamics, inevitably disrupting training stability and introducing significant computational complexity (Nikishin, 2024; Klein et al., 2024).
Equal contribution 'Nayyang Technical University 'Mila- Quebee Al Institute "Université de Montréal Uniierssity of Oxford. Correspondence to: Shen mathshenli @gmmil.com>>, Dacheng Tao <dacheng.tar tao @ ntu weteuuuudu.... edu.sg>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Recent architectural advances particularly spectral normal- ization (Bjorck et al., 2021), layer normalization (Lyle et al.,

---

Network Sparsity Unlocks the Scaling Potential of DRL
2023: 2024a), and residual connections (Espeholt et al.,
fectively counteracts these negative effects by preventing capacity and plasticity loss (Klein al., 2024), constraining parameter growth (Lyle et al., 2024b), enhancing simplicity bias (Lee et al., 2024), and mitigating gradient interfer- ence (Lyle et al., 2023). Furthermore, in Section 5, we extend our empirical evaluation to visual RL and stream- ing RL, demonstrating that the benefits of network sparsity consistently generalize across diverse RL setups.
2018) - have shown considerable success in mitigating plas- ticity loss and enabling DRL network scaling. Building these advances, recent works such as BRO (Nauman et al., 2024b) and SimBa (Lee et al., 2024) have achieved signifi- cantly improved scalability without requiring Reset opera- tions or RL algorithm modifications. However, while SimBa represents the current saate-of-the-are architectural design, its scaling capabilities remain fundamentally limited. Our investigation into scaling SimBa beyond previously studied sizes reveals a consistent pattern: performance deteriorates when scaling the network in any dimension, as evidenced by the sharp drops shown in the dashed lines of Figure 1.
Contributions of this paper can be summarized as:
1. While the advanced SimBa architecture (Lee et al., 2024) has greatly improved DRL network scalabil- ity, we show that introducing static network sparsity through simple one-shot random pruning (Liu et al., 2022) at initialization can unlock further scaling potential beyond previous limitations.
Another line of research aims to leverage adaptive or modi- fied network topologies enhance the parameter efficiency and scalability of DRL models. In this direction, Ceron et al. (2024a) demonstrates that gradual magnitude pruning of large models leads to dramatic improvements in value-based agents' performance. Ceron et al. (2024b) shows that value- based RL networks equipped with Soft Mixture of Experts (MoEs) (Puigcerver et al., 2024) exhibit improved param- eter scalability. In addition, Neuroplastic Expansion (Liu et al., 2024) improves network plasticity by dynamically growing the network from sparse to dense architectures, thereby benefiting from larger model sizes. Although im- plemented differently, these successful approaches share a crucial insight: introducing network sparsity and topology dynamicity during training holds the potential to enable better parameter scaling in DRL models. However, these studies predominantly propose dynamic methods meant to directly act upon the update steps during optimization while ignoring the static sparsity properties of the network at ini- tialization. Moreover, since existing studies primarily focus on standard MLPs, it remains unclear whether these sparsity benefits extend to modern architectures equipped with resid- ual connections and layer normalization (Lee al., 2024). Motivated by these current advances in DRL scalability, this paper aims explore central question:
2.
Our extensive analysis reveals that appropriate net- work sparsity alone can prevent severe optimization pathologies that emerge as models scale up, such as capacity collapse, plasticity loss, unbounded pa- rameter growth and gradient interference.
3. We validate that the benefits of network sparsity generalize well across broader RL scenarios.
2. Preliminary
In this section, we introduce the development and current best practices in DRL network architecture design aimed at mitigating optimization pathologies. Additionally, as foundation for our subsequent investigation, detail our approach to implementing network sparsity. Due to space constraints, a detailed review of pathologies, network scal- ing, and sparse models in DRL is provided in Appendix A.
2.1. Network Architecture Design in Deep RL
Early DRL community primarily treated neural networks as funetion approximators, focusing research efforts on core RL challenges such as exploration (Ciosek et al., 2019) and value overestimation (Fujimoto et al., 2018) rather than net- work architecture design. Hence, for a long period, most of DRL works simply employed basic MLPs by default (Fuji- moto et al., 2023), adding only a few convolutional layers when processing visual observations (Yarats et al., 2022).
Can static network sparsity alone unlock further DRL model scaling potential beyond current advanced ar- chitectures while preventing optimization pathologies?
Through a series of ablation and scaling studies we reveal a clear and definitive answer: YES! As summarized in Fig- ure 1, sparse networks continue to show performance gains well beyond the point where their dense counterparts hit scal- ing limits, demonstrating superior parameter efficiency and enhanced scalability at larger model sizes. Subsequently, Section 4 delves into why introducing sparsity can break through current scaling barriers by leveraging a range of empirical metrics as diagnostic tools. Our analysis reveals that while larger model sizes tend to induce more severe optimization pathologies, appropriate network sparsity ef-
Moreover, the RL paradigm fundamentally differs from (un)supervised learning, with its trial-and-error nature of online interactions and non-stationarity of both data streams and optimization objectives. The interplay of overlooked RL-tailored deep learning mechanisms and inherent RL challenges leads to severe optimization pathologies, which recently have been recognized through several terms, in- cluding primacy bias (Nikishin et al., 2022), dormant neuron phenomenon (Sokar et al., 2023), implicit under- parameterization (Kumar et al., 2021), capacity loss (Lyle
2

---

Network Sparsity Unlocks the Scaling Potential of DRL
al., 2022a), and more broadly, plasticity loss (Klein al., 2024). More concerning, these pathologies intensify with increasing model scale (Ceron et al., 2024a; Lyle et al., 2023), hindering networks' ability to leverage the enhanced expressivity that larger models should provide, thereby fun- damentally limiting the scalability of DRL.
2020) or targeted topology optimization in pruning at ini- tialization (Pal) (Lee et al., 2019; Hoang et al., 2023).
Random Pruning. Static sparse training with one-shot random pruning generates binary masks for each layer at initialization. These masks, which determine the network's s sparse topology, remain fixed throughout training. For a network with L layers, each layer has a binary mask M' {0, 1 xn 11n"'xn' where n' denotes the number of units in layer 1. The effective weights during both training and inference are computed as W M' c W where O denotes element-wise multiplication.
Recent studies have begun to focus on architectural improve- ments to mitigate these pathologies, progressively pushing forward the effective scaling of DRL networks. Various nor- malization techniques have shown different degrees of ef- fectiveness, including Spectral Normalization (Bjorck et al., 2021), Batch Normalization (Bhatt et al., 2024), and the widely adopted Layer Normalization (Lee et al., 2023; Lyle et al., 2023; 2024a). Beyond merely using ResNet as vi- sual encoder (Espeholt al., 2018), BRO (Nauman al., 2024b) first demonstrated the effectiveness of incorporating residual blocks in both policy and value networks, signifi- cantly enhancing robustness and performance in challenging RL tasks. Drawing from these insights and detailed analy- sis, SimBa (Lee et al., 2024) further enhanced the training stability by introducing observation normalization layers to regulate input data distributions. Representing the current best practices in DRL architecture design, SimBa success- fully scaled DRL models beyond 10M parameters.
Layer- Sparsity Ratios. Random pruning represents remarkably simple random sampling process, requiring only layer-wise sparsity ratios to be pre-defined. There are two commonly adopted approaches for determining layer- wise sparsity ratios from the overall network sparsity:
The sparsity ratio s' of each individual layer I is equal to the overall network sparsity S.
Uniform:
Erdōs-Rényi (ER): This approach randomly generates the sparse masks so that the sparsity in each layer s' scales as 1 - z'''+n' " for fully-connected layer (Mocanu et al., 2018) and as 1 - n n -= +'''''''+h for a convolutional layer with kernel dimensions w' x h/' (Evci al., 2020).
However, as shown by Figure 13 in Lee et al. (2024) and our subsequent investigation, further increasing SimBa's model size not only fails to yield performance improvements but also leads to significant degradation. This motivates us to consider: Have we reached the fundamental scaling limits of DRL models on current benchmarks and tasks? Or is there simple yet effective approach that could push these bound- aries further? Drawing inspiration from previous scalability improvements achieved through non-standard architectures beyond vanilla MLPs that incorporate both network spar- sity and dynamicity (Graesser et al., 2022; Tan et al., 2023; Ceron et al., 2024b;a; Liu et al., 2024), this paper decouples sparsity standalone feature to examine its effectiveness on top of advanced architectural designs. Our thorough investigation in Section 3 reveals that sparsity alone can un- lock further scaling potential. Our empirical study not only establishes a simple yet strong baseline for DRL network scaling, but also suggests untapped opportunities for more specialized DRL architectural designs.
Extensive prior studies in both supervised learning (Liu et al., 2022) and RL. (Graesser et al., 2022; Tan et al., 2023; Liu et al., 2024) have shown that ER-based initialization yields superior performance over uniform sparsity, espe- cially at high sparsity levels. Thus, we adopt layer- wise sparsity ratios based on ER throughout our investigation unless specified otherwise.
3. Sparsity Promotes DRL Network Scaling
This section aims to investigate whether introducing network sparsity can unlock further scaling potential in DRL models beyond the effective scaling limits of dense SimBa networks. We conducted extensive experiments on several of the most challenging DeepMind Control (DMC) (Tassa et al., 2018 8) tasks using both Soft Actor-Critie (SAC) (Haarnoja et al., 2018) and Deep Deterministic Policy Gradient (DDPG) (Lil- licrap, 2015) with advanced SimBa architecture (Lee et al., 2024). In this section, we first show that appropriate network sparsity both enables further model scaling and enhances parameter efficiency. We then analyze how model size and sparsity ratios interact to identify optimal combinations.
2.2. Sparse Network with One-Shot Random Pruning
To isolate the independent role of network sparsity on DRL scaling, we conduct our investigation using static sparse training (SST) (Liu et al., 2022) with one-shot random prun- ing. This straightforward approach establishes a fixed sparse topology through random pruning before training, avoiding confounding factors such as topology dynamics in dynamic sparse training (DST) (Mocanu et al., 2018; Evei et al.,
Experimental Setup. Introducing network sparsity when scaling up model size requires careful control of multiple variables in our comparative experiments, including the width and depth of both actor and critic networks, as well as their respective sparsity levels. Since our primary goal
3

---

Network Sparsity Unlocks the Scaling Potential of DRL
DMC Hard (SAC)
DMC Hard (DDPG)
DMC Hard (SAC)
DMC Hard (DDPC)
131
-
Dosa Nekweth
Spars Network
60
Derise Networs
Spuca Notwazk
03:
Peptl
Figure 2. Network scaling experiments comparing dense and sparse SimBa architectures trained with SAC and DDPG on DMC Hard tasks. Results demonstrate that appropriate sparsity enables effective model scaling while preserving parameter efficiency.
to investigate whether sparsity can extend DRL scaling boundaries beyond current dense model limitations, we es- tablish the following experimental protocol to ensure fair and systematic evaluation: Following SimBa's 's validated configuration, we maintain its default actor-critic size ra- tio where the critic network is four times wider and twice deeper than the actor network. Taking the default SimBa model size as our baseline (actor hidden dimension of 128, critic hidden dimension of 512, one and two SimBa residual blocks for actor and critic respectively), we scale both net- works by integer multiples in width or depth. For instance, a model with Width Scale = 2 and Depth Scale 4 means the actor network uses two blocks with hidden dimension 256, while the critic network uses four blocks with hidden dimension 1024. We specify only the overall sparsity level for both networks, with specific layer sparsity determined by the ER initialization. Complete experimental details are provided in Appendix B.I.
scaling of both dimensions. Detailed results for individual tasks can be found in Appendix B.2. First, when maintaining the same parameter count, larger sparse networks achieve superior performance compared to their smaller dense coun- terparts, indicating higher parameter efficiency. Second, at equal model sizes, sparse networks outperform their dense counterparts with fewer parameters, demonstrating more effective utilization of network capacity. Furthermore, in Section 4, we will demonstrate that these benefits funda- mentally arise from appropriate network sparsity preventing DRL networks from falling into more severe optimization pathologies during scaling.
Takeaway: Weight-level sparsity, a simple architectural feature, can further unlock the scaling potential of DRL networks beyond the improved scalability achieved by ad- vanced architectures, enabling networks better harness both parameter efficiency and model capacity.
3.1. Appropriate Network Sparsity Both Enables Model Size Scaling and Enhances Parameter Efficiency
3.2. The Interplay between Model Size and Sparsity
Having established that appropriate network sparsity can enable better scaling, we next examine the practical im- plications of incorporating sparsity into DRL networks to derive effective implementation guidelines. Specifically, we treat model size and sparsity ratios as independent config- uration parameters and analyze their performance impacts systematically. Several critical questions merit investigation, including: How does varying network sparsity influence the scaling potential of larger models? What role does sparsity play in enhancing default-sized model performance? Fur- thermore, what are the optimal combinations of model size and sparsity that yield the best DRL performance?
Although SimBa has integrated various recently proven in- terventions for mitigating DRL network pathologies and significantly improved its scalability, excessive model scal- ing still leads to performance collapse. As illustrated in Figure 2, when scaling the network along a single dimen- sion beyond critical thresholds (exceeding 2x width or Ix depth of the baseline model), larger models yield worse per- formance. This degradation trend suggests that larger dense networks suffer from severely reduced parameter efficiency, preventing DRL agents from exploiting the theoretically increased representational capacity.
These findings motivate our first investigation: whether in- troducing appropriate sparsity during model scaling can break through current scaling limitations. Using the SimBa network with Width Scale 2 and Depth Scale 1 as the anchor point, we maintain the same learnable parameters count as the optimal dense model while increasing the to- tal model size, exploring if such sparse scaling can more effectively leverage the increased network capacity.
As shown in Figure 3, increasing network sparsity exhibits substantially different effects across model scales. For large networks, higher sparsity ratios consistently lead to im- proved performance across all tasks. This is particularly evident in challenging scenarios like Humanoid Walk and Dog Run, where large sparse networks achieve up to 40% higher returns compared to their dense counterparts. Such results strongly indicate that sparsity serves as an effective mechanism for unlocking the scaling potential of larger models. In contrast, default-sized networks show modest
The scaling trends for width and depth are presented in Fig- ure 2, while Figure 1 additionally illustrates the combined
4

---

Network Sparsity Unlocks the Scaling Potential of DRL
Dog Trot
Dog Run
Humanood ond Walk
Humanoid Run
SAC
SAC
Epoodde Beturn
410
178
DOPG
DDPG
B00G
DOIG
7.8
Epledd
140
130
046 8.5
8.5 96
8.5
3.5
Sparsity
Sparsity
Sparaity
Sparaity
Dedaolt Dencen Network
Denoo Network
Defeult Spane Network
Sparss Natwork
Figure 3. Scaling via network sparsity on four hardest DMC tasks using SAC and DDPG. For both default SimBa networks (~ 4.5M parameters, blue lines) and large networks (~ 109M parameters, orange lines), we systematically explore sparsity ratios from 0.1 to 0.9, with steps of 0.1. Results demonstrate that while default networks suffer from high sparsity, large networks consistently benefit from increased sparsity ratios, highlighting the crucial role of sparsity in enabling effective model scaling.
Final Srank
Srank Progression
performance gains with low sparsity ratios, validating the general benefits of network sparsity. However, their perfor- mance deteriorates significantly with higher sparsity, indi- cating that sufficient learnable parameters are essential for maintaining network expressivity.
234
Largo Dense Network Large Sparse
The best practice for scaling DRL, networks is to increase model size while maintaining high static sparsity, achieving both efficient parameter utilization and superior representational expressivity.
Takeaway:
124
Spuraity
Eastrorazent Stopa
1od
Figure 4. Analysis network representation capacity via Srank metric on Humanoid Run using SAC. Network configurations match Figure 3. (Left) Final Srank after 1M steps across sparsity ratios. (Right) Srank progression for three network variants.
4. Understanding the Barrier of Scaling and the Benefits of Network Sparsity
network. Srank measures the effective rank of learned rep- resentations, indicating the diversity and richness of the representations learned by the network. This is computed by performing eigenvalue decomposition of the feature matrix covariance and summing the indicators of singular values above a threshold т: Srank Σj= [o, > 7), where F E Rdxm is the feature matrix containing d samples of m-dimensional features, o, denotes the singular values, and I(.) is the indicator function.
As demonstrated in the previous analysis, network sparsity empirically enhances DRL model scaling capabilities. This distinct scaling behavior raises the question: how can sparse networks with fewer learnable parameters achieve superior performance compared to their dense counterparts? To gain deeper insights into this phenomenon, we explore there representation capacity, plasticity, regular- ization, and gradient interference. Our analysis reveals that while naively increasing DRL model size worsens optimiza- tion pathologies, introducing appropriate sparsity effectively mitigates these issues through multiple mechanisms.
critical factors:
As shown in Figure 4. our analysis reveals two key findings. First, increasing network sparsity leads to a consistent im- provement in the critic's Srank, gradually approaching the theoretical upper bound of 256. More surprisingly, scaling up the network from 4.5M to 109M parameters leads to an unexpected degradation in representational capacity, re- flected by a marked decline in Srank. This capacity collapse potentially explains the scaling barrier in DRL.
4.1. Representational Capacity
The primary motivation for scaling up neural networks is to gain enhanced expressivity, allowing them to capture more complex relationships and learn more effective representa- tions than their smaller counterparts. Therefore, we first investigate how sparsity impacts network capacity.
Takeaway: Unlocking greater expressivity through seal- ing in DRL requires appropriate sparsity, as larger dense networks tend to suffer from severe capacity collapse.
Srank. We characterize the representational capacity using the Stable-rank (Srank) (Kumar et al., 2021) of the critic
5

---

Network Sparsity Unlocks the Scaling Potential of DRL
Episode Retura
Actor Dormant Ratio
Critic Dorment Ratio
Actor Gradient Norm
Critic Gredient Norm
Humanoka Run Tumannii Run
100
BAC
10
04
100
23
1.40
BAC
Dag Ruon
8
200
DDPG
Dog Rum
1B
16
16
04
02
3.0
0.2
6.1
6.5
0.
0.2
0.4
0.2
3.4
3.5
Small Dence Network (Model Size 1750
Large Dease Network (Model Size 208M)
Large Syarse Network [Mode] Size 208M Spuniiy - 9293
Figure 5. Plasticity measurements of three representative network configurations in the two most challenging tasks with SAC and DDPG. Despite employing the advanced SimBa architecture, large dense critic networks still suffer from rising neuron dormancy and gradient collapse as training progresses. Introducing sparsity, however, proves be effective solution, preventing such pathological trends.
4.2. Plasticity
Humanod Run
Dog Run
Recent studies have identified plasticity loss as a key patho- logical symptom in DRL networks, where models progres- sively lose their ability to adapt to new experiences during training, eventually reaching learning stagnation (Nikishin et al.,, 2022; Lyle al., 2023). While directly measuring plasticity remains challenging, several indicators have been found to strongly correlate with its deterioration, particu- larly the emergence of dormant neurons and the decay of gradient signals (Klein et al., 2024; Lewandowski et al., 2024). Hence, we characterize network plasticity through two key metrics: the dormant ratio (Sokar al., 2023), which measures the proportion of inactive neurons, and gra- dient norm dynamics, which monitors the preservation of
Epoosts Rotura
8
109
1.4
Lange Pamoe Nettooek (wile Bosttl
Largs Sparse Natwookk (wel Itoee))
Lange Dusee Netiook (wl Bost))
Lurg: Sparse Netwock (ee Baset)
Figure 6. Reset diagnostic comparison for large dense networks and large sparse networks. Despite dense networks relying on Reset operations recover plasticity, sparse networks maintain learning capability naturally without such remedial interventions.
the strength of learning signals during training, where dimin- ishing gradient norms potentially signal a loss of plasticity.
learning capability throughout training (Abbas et al., 2023).
The distinct plasticity dynamics between sparse and dense networks are illustrated in Figure 5. Benefiting from the ad- vanced SimBa architecture, small dense networks maintain healthy plasticity throughout training without significant deterioration. However, as model size increases, large dense networks exhibit clear signs of plasticity loss, manifested through rising critic dormant ratios and collapsing gradient norms in later training stages. Notably, actor networks show minimal plasticity deterioration, aligning with the findings in Ma et al. (2024). In contrast, large sparse networks effec- tively mitigate the severe plasticity loss commonly observed in larger models. Moreover, using equivalent parameter counts, large sparse networks achieve comparable or lower dormant ratios than small dense networks while sustaining stronger gradient signals throughout training.
Dormant Ratio. The dormant ratio is the proportion of dormant neurons within the entire network. Given an input distribution D, neuron in layer& is considered dormant if its dormant score of across input data P(:;D) D) falls below threshold The dormant score of of individual neuron can be defined as:
P E स7 EkEn Be- P(: (x))
(1)
where Mz) denotes neuron activation and H' is the layer neuron count. A neuron in layer é is 7-dormant if of < T.
Gradient Norm. We monitor the L2 norm of network gradients over active (non-pruned) parameters to quantify
6

---

Network Sparsity Unlocks the Scaling Potential of DRL
Reset as a Diagnostic Tool. Given Reset serves direct approach to restore plasticity in DRL networks (Nikishin et al., 2022), we employ this operation to assess whether plasticity loss remains a critical issue in the large sparse SimBa network. As shown in Figure 6, Reset operations significantly boost the performance of large dense networks by breaking their learning stagnation yet provide no bene- fits to large sparse networks, and may even slightly harm performance by disrupting established training dynamics.
Actor Simplicit Biss Score
Simplicity Score
Performance Improvment
Samplectty Same Sccono
0.5
0.3-
0.3 2.5
0.7
0..5
0.9
Spansiyy
In past practices, scaling up vanilla MLP networks often required Reset operations. With the progressive introduc- tion of architectural advances, this dependence on Reset gradually diminished (Lee et al., 2024). Now, by incorporat- ing sparsity into SimBa and simultaneously increasing both model size and sparsity ratio, we have completely elimi- nated the need for Reset while achieving superior scalability.
Figure 8. Simplicity bias scores and performance improvements across different sparsity ratios, where performance gains are aver- aged over large networks in eight scenarios from Figure 5,
from Lee et al. (2024) that evaluates network complexity at initialization avoid confounding factors from the non- stationary RL training dynamics.
Figure 8 demonstrates that network sparsity consistently promotes higher simplicity bias scores, correlating with im- proved performance in scaled-up architectures. Remarkably, SimBa's s comprehensive architectural improvements yielded only 0.5 increase in simplicity bias scores (from 5.8 to 6.3) (Lee et al., 2024), while our simple one-shot pruning approach further raises these scores by 0.3-0.4, highlighting the effectiveness of sparsity as key network property.
Takeaway: Weight-level sparsity effectively preserves plasticity in large-scale networks, eliminating plasticity loss as a bottleneck for scaling up SimBa architectures.
4.3. Regularization
We then examine how network sparsity serves as an implicit regularization mechanism that simultaneously constrains weight magnitudes and induces beneficial inductive biases towards simpler solutions.
Takeaway: Sparsity is an effective regularizer that can control parameter growth and promote simpler solutions.
Parameter Norm. Unbounded parameter growth has been identified as critical pathological behavior in deep RL networks, leading to training instability and severely hindering the network's ability to effectively learn value functions and policies (Lyle et al., 2024b).
4.4. Gradient Interference
Finally, we investigate whether network sparsity affects the interactions between gradients from different data points a phenomenon called gradient interference that impacts learning dynamics (Bengio et al., 2020; Lyle et al., 2022b). Following the analytical approach in Lyle et al. (2023), we will estimate the interference level by gradient covariance matrices, which are computed by sampling A training points X1, Xk, and constructing C2 E BR with entries:
Actor Parameter Norm
Critic Parameter Norm
Ponaster Nom
Cali- (Vol(0,x;), Vol(O, xz)) IV x,, IIII (((0,x)))
Enverssttvct Strpe
Devdrvataess Stops
3e5
(2)
Small Denas Netbookk (Ccodd
1250)
Large Dasse Netwodk (44daa Sixe 23854)
Largs Sparse Nerwark DModal Sae 200N. Spenity - 52% Lauraable Farmmotars - 1750
Figure 7. Parameter norm evolution for actor and critic networks, corresponding to the Humanoid Run (SAC) scenario in Figure 5.
Large Sprrso Nwtwork (Ciitas)
Largs Densn Network (Crisc)
While network sparsification inherently reduces the total number of parameters compared to dense architectures, re-
markable finding emerges in Figure 7: large sparse networks exhibit similar or even lower parameter norms compared to small dense networks with equivalent learnable parameters. This suggests that sparsity serves as an effective implicit regularizer beyond mere parameter reduction.
batiiic
Fin 5330
Figure 9. Gradient covariance matrices of large sparse and dense networks before and after training. Darker blue indicates strongly aligned gradients, darker red indicates strongly conflicting gra- dients, and lighter colors indicate more independent gradients. Sparse networks maintain more independent (less interfering) gra- dients throughout training compared to dense networks.
Simplicity Bias. Neural networks inherently favor learning simpler patterns over complex ones, phenomenon known as simplicity bias (Shah et al., 2020; Berchenko, 2024).
Figure 9 shows that while sparse and dense networks start with similar gradient correlations, sparse networks maintain
To quantify this bias, we utilize the simplicity bias score
7

---

Network Sparsity Unlocks the Scaling Potential of DRL
Antovd
Walker2d.vv
significantly weaker correlations throughout training.
2300
Neterak Spanaeg
111+
2754,3
20012
4350
WAR Spanting
1992
2633.5
2300
Takcaway: Network sparsity naturally promotes gradient orthogonality, mitigating gradient interference.
"CA
3013.1
3964
4166
-2000
22115
2619
2029-3
2600
2360
3300
3475
3516.5
2012.3
8651.
2240.2
2451
2094.7
2072
2300
128
312
124
P12
5. Sparsity Boosts Scaling in Broader Setups
Figure 12. Streaming RL network scaling performance across spar- sity ratios and widths, averaged over seeds after 2M steps. Walker2d-ve 24 (Width-612)
To examine the broader applicability of our findings, we ex- tend our experiments to visual RI. and streaming RI. scenar- ios. Here we report core results, with complete experimental details provided in Appendix C.
Ant-v4 (Wstth-512)
Eiptect Berser
Visual RL. In our visual RL experiments, we evaluate image- based DMC with DrQ-v2 (Yarats al., 2022) our baseline. Building on the insights of Ma et al. (2024), we only scale the critic network while keeping the actor fixed.
28
Drcae Network wi Spareetat
Spere Netwark (5-0.2) wo Sperselic
Dress Network Spaneizi
Speroo Netwock (5-0.11 Serommeettt
Quadruped Run
Hopper Hop
Figure 13. Comparing static sparsity (fixed pruned weights) and SparseInit (trainable sero-initialized weights) in streaming RL.
CHtt: Sparaty
451.5
420/2
7375
61.54
7000
CHU Spanit
185.1 1
231.5
301.2
524.5
570.2
7084
754.5
-000
04
313.1
250.1
255.2
234.0
training, Sparselnit only zeroes weights at initialization, al- lowing gradient updates to gradually reduce sparsity during training. Beyond showing that static pruning enables ef- fective network scaling in streaming RL (Figure 12), we demonstrate that both one-shot pruning and SparseInit sig- nificantly improve sample efficiency (Figure 13), underlin- ing the broad benefits of network sparsity in RL training.
125
CAt:: Widds
Crtte Wasts
Figure 10. Scaling via network sparsity and critic width on two representative visual RI. tasks, reporting mean episode returns averaged over 5 random seeds after 2M environment steps.
Quadruped Run
Critic Plasticity
0.55
176
2.45
500
0.43
6. Conclusion
220
Widta Scdd Sparcity
This work demonstrates that static network sparsity, achieved through one-shot random pruning, is simple yet powerful tool for unlocking the scaling potential of DRL. By carefully studying its effects on representational capacity (via Srank), simplicity bias, and gradient interference, we show that sparsity not only mitigates optimization patholo- gies but also enables larger models to achieve superior per- formance across diverse RL settings, including challenging streaming RL scenarios. These findings are particularly significant because they address a long-standing challenge in RL: scaling models effectively, an area where RL has historically struggled compared to supervised learning.
0.25
2p
Esveromment Stops
Essiommentt Stepa
Figure/ 11. Learning curves and critic plasticity for Quadruped Run under varying sparsity levels with a 4x wider critic network.
Results in Figure 10 show that while increasing critic width leads to gradual improvements with dense networks, com- bining network sparsity (0.8) with larger widths can further boost performance significantly, highlighting the effective- ness of sparsity in visual RL scaling. We dive into the plasticity dynamics to understand these performance differ- ences. As shown in Figure 11, scaling up the critic width by 4x leads to severe plasticity deterioration in dense networks, where large fraction of neurons become inactive. However, introducing sparsity effectively mitigates this issue, aligning with our findings in Section 4.2 that weight-level sparsity helps preserve neuron-level plasticity throughout training.
Our approach is easy to implement and readily compatible with any RL algorithm, requiring only a one-time prepro- cessing step. Unlike dynamic methods which update spar- sity patterns during training, static sparsity maintains a fixed sparse structure throughout training, avoiding both training instability and additional computational overhead.
Streaming RL. Streaming RL agents process each sample immediately upon arrival without storing past experiences, exacerbating the non-stationarity of the learning process and resulting in significant sample inefficiency (Elsayed et al.,, 2024; Vasan et al., 2024). To overcome this stream barrier, Elsayed et al. (2024) propose Sparselnit by randomly initial- izing most weights to zeros. Unlike our one-shot pruning at initialization which maintains fixed sparsity throughout
Our results highlight the importance of architectural choices in DRL and suggest that network architecture and RL algo- rithms should not be studied in isolation. By establishing sparsity as a key enabler of scalability, this work opens new avenues for research into specialized sparsity structures, dy- namic sparsity methods, and theoretical frameworks to make RL more practical and deployable in real-worke settings.
8

---

Network Sparsity Unlocks the Scaling Potential of DRL
Acknowledgment
Castro, P. S., Moitra, S., Gelada, C., Kumar, S., and Belle-
mare, M. G. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http:
This project is supported by the National Research Foun- dation, Singapore, under its NRF Professorship Award No. NRF-P2024-001. Lu Li and Pierre-Luc Bacon are supported by CIFAR. This research is enabled in part by compute resources, software and technical help provided by Mila (mila.quebec).
Ceron, S. O., Courville, A., and Castro, S. In value- based deep reinforcement learning, pruned network is good network. In Forty-first International Con- ference Machine Learning, 2024a. URL https: / //openreview.oo net/ for.mmidd=eeov900C.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Ceron, J. S. O., Sokar, G., Willi, T., Lyle, C., Farebrother,
J., Foerster, J. N., Dziugaite, G. K., Precup, D., and Castro, P. S. Mixtures of experts unlock parameter scaling for deep RL. In Forty-first International Con- ference on Machine Learning, 2024b. URL https: / //openreview.
References
Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, pp- 620-636. PMLR, 2023.
Ciosek, K., Vuong, Q. Loftin, R., and Hofmann, K. Bet- ter exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
Dohare, S., Hernandez-Garcia, J. F., Lan, Q. Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plas- ticity in deep continual learning. Nature, 632(8026): 768-774, August 2024. ISSN 1476-4687. doi: 10310 s41 441586-024-007711-7 URL http: httpt:::/xx.d....oo.o doi.org/ 10. 10.1038/s41555 024-07711-7.
Arnob, Y., Ohib, R., Plis, S., and Precup. D. Single-shot pruning for offline reinforcement learning. arXiv preprint arXiv:22 12. 15579, 2021.
Arnob, S. Y., Ohib, R., Plis, S. M., Zhang, A., Sordoni, A., and Precup, D. Efficient reinforcement learning by discovering neural pathways. In The Thihtt- Annual Conference on Neural Information Processing Systems, 2024. URL https: www./rpeeeeevv.eeeeeeeeeee net/ forum? id-WEoOrePOnS.
Elsayed, M., Vasan, G., and Mahmood, A. R. Streaming deep reinforcement learning finally works, arXiv preprint arXiv:2410. 14606, 2024.
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V. Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In Interna- tional conference on machine learning, pp. 1407-1416. PMLR, 2018
Bengio, E., Pineau, J., and Precup, Interference and generalization in temporal difference learning. In Inter- national Conference on Machine Learning, pp. 767-777. PMLR. 2020.
Berchenko, Y. Simplicity bias in overparameterizeed ma- chine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1052-111000, 2024.
Evei, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pp. 2943- 2952. PMIR, 2020.
Bhatt, A., Palenicek, D., Belousov, B., Argus, M., Ami- ranashvili, A., Brox, T., and Peters, Crossq: Batch normalization deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth In- ternational Conference on Learning Representations, 2024. URL httttp::///peerevieieee net forum? id-Pc20:IsTIX.
Farebrother, J., Orbay, J., Vuong, Q., Taiga, A. A., Cheb- otar, Y., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., et al. Stop regressing: Training value func- tions via classification for scalable deep rl. arXiv preprint WPXh::2403033950 2024.
Bjorck, N., Gomes, C. P., and Weinberger, K. Q. Towards deeper deep reinforcement learning with spectral nor- malization, Advances in neural information processing systems, 34:8242-825 8255, 2021.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error actor-critic methods. In Interna- tional conference on machine learning, pp. 1587-1596 PMLR, 2018

---

Network Sparsity Unlocks the Scaling Potential of DRL
Fujimoto, S., Chang, W.-D., Smith, E. J., Gu, S. S., Precup, D., and Meger, D. For SALE: State-action representa- tion learning for deep reinforcement learning. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL httph://///c.. pttctts://openreeeee opwnreveee.ee.eeeeeee net./ X2
Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., Yun, S.-Y., and Yun, C. Plastic: Improving input and label plasticity for sample efficient reinforcement learn- ing. In seirth-seventh Conference on Neural Information Processing Systems, 2023.
Lee, H., Hwang. D., Kim, D., Kim, H., Tai, J., Subrama- nian, K., Wurman, P. R., Choo, J., Stone, P., and Seno, T. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning, arXiv preprint arXiv:2410.09754, 2024.
Goldie, A. D., Lu, Jackson, M. T., Whiteson, S., and Fo- erster, J. N. Can learned optimization make reinforcement learning less difficult? arXiv preprint arXiv:2407.07082 2024.
Graesser, L., Evci, U.. Elsen, E., and Castro, P. S. The state of sparse training in deep reinforcement learning. In International Conference on Machine Learning, pp. 7766-7792. PMLR, 2022.
Lee, N., Ajanthan, T., and Torr, P. SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learning Representations, 2019. URL https: petttss:/oppeeee openreview. net htt//forumiidee BV22222
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off- -oflcc maximum entropy deep reinforce- ment learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Lei Ba, J., Kiros, J. R., and Hinton, G. E. Layer normaliza- tion. ArXiv e-prints, pp. arXiv-1607,201
Lewandowski, A.. Tanaka, H., Schuurmans, D., and
Machado, M. C. Directions of curvature as expla- nation for loss of plasticity. Preprint at http::/aaxii. org/abs/23 12.0022222002026 2024.
Hansen, N., Su, H., and Wang, X. Td-mpe2: Scalable, ro- bust world models for continuous control. arXiv preprint arXiv:23 10. /6828, 2023.
Lillicrap. T. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (rXXvv:509.02971 2015.
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Os- trovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Liu, J., Obando-Ceron, J., Courville, A., and Pan, L. Neuro- plastic expansion deep reinforcement learning. arXiv preprint arXfv:2410.07994 2024.
Hoang, D. N., Liu, S.. Marculescu, R., and Wang. Z. REVIS- ITING PRUNING AT INITIALIZATION THROUGH THE LENS OF RAMANUJAN GRAPH. In The Eleventh International Conference on Learning Representations, hwtp:/rrntrrrrrrree
Liu, S., Chen, T. Chen, X., Shen, L., Mocanu, D. C., Wang, Z., and Pechenizkiy, M. The unreasonable effectiveness random pruning: Return of the most naive baseline for sparse training. In International Conference on Learning Representations, 2022. URL https: /ownnevveeeeeeeeeeeeeeeeeee net / foorummmd:/BZ3 1ttp:/2___3_ttt...
2023. URL https:t////// id=uVeDssOff_
Juliani, A. and Ash, J. T. A study of plasticity loss in on-policy deep reinforcement learning. arXiv preprint ar0iv:2405. 19/53, 2024.
Lyle, C., Rowland, M., and Dabney, W. Understanding and preventing capacity loss in reinforcement learning. arXiv preprint (txxx:2204.09560 2022a.
Kaiser, E.,, Babaeizzaeh, M., Mitos, P., Osiński, B., Camp- bell, R. H., Czechowski, K., Erhan, D.,, Finn, C., Koza- kowski, P., Levine, S., et al. Model based reinforcement learning for atari. In International Conference on Learn- ing Representations, 2020.
Lyle, C., Rowland, M., Dabney, W., Kwiatkowska, M., and Gal, Y. Learning dynamics and generalization in deep reinforcement learning. In International Conference on Machine Learning, pp. 14560 14581. PMLR, 2022b.
Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural net- works. International Conference on Machine Learning, pp. 23190-2222 222101 PMLR, 2023.
Klein, T., Miklautz, L., Sidak, K., Plant, C., and Tschi- atschek, S. Plasticity loss in deep reinforcement learning: A survey, arXiv preprint
2024.
Kumar, A., Agarwal, R., Ghosh, D., and Levine, S. Im- plicit under- under-parameterinn parameterization inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021.
Lyle, C., Zheng, Z., Khetarpal, K., Martens, J., van Hasselt, H., Pascanu, R., and Dabney, W. Normalization and effective learning rates in reinforcement learning. arXiv preprint arXiv:2407.01800 2024a.
10

---

Network Sparsity Unlocks the Scaling Potential of DRL
Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pas- canu, R., Martens, J., and Dabney, W. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18777 2024b.
Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netra- palli, P. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33: 1573-9588 2020.
Ma, G., Li, L., Zhang, S., Liu, Z., Wang, Z., Chen, Y., Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview.woemmm/ee net forum? 2id -addddddaaaaxxooooo
Sokar, G., Mocanu, E., Mocanu, D. C., Pechenizkiy, M., and Stone, P. Dynamic sparse training for deep reinforcement learning. arXiv preprint arXiv:2106.0422
2021.
Sokar, G., Agarwal, R., Castro, P. S., and Evei, U. The dor- mant neuron phenomenon in deep reinforeement learning. In International Conference on Machine Learning, pp. 32145-32168, PMLR, 2023.
Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., and Liotta, A. Scalable training of arti- ficial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9 (1):2383, 2018.
Tan, Y., Hu, P., Pan, L.,, Huang. J., and Huang. L. RIx2: Training a sparse deep reinforcement learning model from scratch. In International Conference on Learning Representations, 2023. URL https: oppnreviee... net/ / www.rforruddiiiioe diw.ummmiddddeEoggg.......
Nauman, M., Bortkiewicz, M., Mitos, P., Trzcinski, T., Os- taszewski, M., and Cygan, M. Overestimation, overfit- ting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In Forty-first International Con- ference on Machine Learning, 2024a. URL https: /toppnneevew.neee./ http://oooommiidddddeeeeeeeeee
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. I.., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv: arXiv:/801.00 mat:.0001000000 2018.
Todorov, H.,, Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ inter- national conference on intelligent robots and systems. pp. 5026 5033. IEEE, 2012.
Nauman, M. Ostaszewski, M., Jankowski, K., Miloš, P., and Cygan, M. Bigger, regularized, optimistic: scal- ing for compute and sample-efficient continuous control. In Advances in Neural Information Processing Systems, 2024b.
Van Hasselt, H. P., Hessel, M., and Aslanides, J. When to use parametric models in reinforcement learning? Ad- vances in Neural Information Processing Systems, 32, 2019.
Nikishin, E. Parameter, experience, and compute efficient deep reinforcement learning. PhD thesis, Université de Montréal, 2024.
Vasan, G., Elsayed, M., Azimi, A., He, J., Shariar, F., Bellinger, C., White, M., and Mahmood, A. R. Deep policy gradient methods without batch updates, target networks, or replay buffers. arXiv preprint 1rXhv:2411.153700 2024.
Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In International conference machine learn- ing. pp. 16828-1668 16847. PMLR, 2022.
Vischer, M. A., Lange, R. T., and Sprekeler, H. On lottery tickets and minimal task representations in deep rein- forcement learning, arXiv preprint MXXv:2105.0166 2021.
Puigcerver, J., Ruiz, C. R., Mustafa, B., Renggli, C., Pinto, A. S., Gelly, S., Keysers, D., and Houlsby, N. Scalable transfer learning with expert models. In International Conference on Learning Representations, 2020.
Xu, G., Zheng, R., Liang, Y., Wang, X., Yuan, Z., Ji, T., Luo, Y., Liu, X., Yuan, J., Hua, P., Li, S., Ze, Y., III. H. D., Huang, F., and Xu, H. Drm: Master- ing visual reinforcement learning through dormant ra- tio minimization. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //openeview. net
Puigcerver, J., Ruiz, C. R., Mustafa, B., and Houlsby, N. From sparse to soft mixtures of experts. In The Twelfth International Conference on Learning Representations, 2024. URL https:/ http://pnneeeeeeeeeeeeeeeeeeeeea forum? id id=ixpsAj71tE.
Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp- 00365-30388 PMLR, 2023.
Yarats, D.,, Fergus, R., Lazaric A., and Pinto, L. Master- ing visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022. URL https:// openreview net/ forum? id= Srrruumidd_S__yyeeeee SJ___yeee...

---

Network Sparsity Unlocks the Scaling Potential of DRL
The appendix is divided into several sections, each giving extra information and details.
A
Related Work
12
A.1 Optimization Pathologies in DRL
12
A.2 Network Scaling in DRL
13
A.3 Sparse Networks in DRL
13
Detailed Experimental Setup and Results of Main Experiments
14
B.1 Detailed Experimental Setup
14
B.2 Detailed DMC Results
15
Detailed Experimental Setup of broader setups
18
C.1 Visual RL
C.2 Streaming RL
C.3 Atari-100k
19
A. Related Work
In this section, we review several topics closely related to our work. We begin by discussing the optimization pathologies unique to deep reinforcement learning (DRL). We then examine the scaling barriers in DRL models and various attempts to enhance their scalability. Additionally, we review existing applications of sparse networks in DRL, whether aimed model compression, training acceleration, or performance enhancement.
A.1. Optimization Pathologies in DRL
Although deep neural networks have driven remarkable advances current deep reinforcement learning (DRL) applications, growing evidence indicates that DRL networks are prone severe optimization pathologies during training (Nikishin, 2024; Nauman al., 2024a; Goldie al., 2024). These pathologies emerge from the unique challenges of integrating DL. mechanisms with the RL paradigm, presenting distinctive issues not encountered in traditional RL settings. Such difficulties stem from fundamental characteristics of reinforcement learning that distinguish it from supervised learning: non-stationary data distributions and optimization objectives, as well as the inherent nature of learning through online interactions.
These DRL-specific pathologies have recently been identified and characterized through various phenomena: primacy bias (Nikishin al., 2022), the dormant neuron phenomenon (Sokar al., 2023). implicit under- pareratteeemiitiiionnn (Kumar al., 2021), capacity loss (Lyle et al., 2022a), and the broader issue of plasticity loss (Klein et al., 2024; Abbas et al., 2023; Juliani & Ash, 2024). Although these studies approach the problem from different angles, they converge finding: DRL networks routinely develop severe optimization pathologies during training that fundamentally impair their ability to learn from new experiences. The consequences manifest either as severe sample inefficiency or, in the worst cases, complete learning stagnation (Ma et al., 2024). These pathologies manifest through several observable symptoms: a high proportion of inactive neurons (Sokar et al., 2023), reduced effective rank of representational features (Kumar et al., 2021; Lyle et al., 2022a), unbounded growth parameter norms (Lyle al., 2023), and increased gradient interference across training samples (Lyle et al., 2024b). Each of these symptoms contributes to the network's diminishing ability effectively leam and optimize both the policy and value functions.
common
Moreover, such pathological behaviors intensify with increasing model size, fundamentally limiting the scaling capabilities of DRL models (Ceron et al., 2024a;b; Bjorck et al., 2021). Consequently, a critical bottleneck unique to DRL has emerged: how to effectively scale up neural networks for better representation capacity while avoiding falling into severe optimization pathologies. In this work. we demonstrate a surprisingly simple yet powerful alternative: static network pruning prior to trainigg. Through extensive experimental comparisons and empirical analysis, we show that this approach to network
12

---

Network Sparsity Unlocks the Scaling Potential of DRL
sparsity effectively unlocks the scaling potential of DRL networks while significantly mitigating optimization pathologies.
A.2. Network Scaling in DRL
Using deep neural networks is a key factor in successfully applying reinforcement learning to complex tasks. However, while some recent advances in supervised learning have been driven by scaling up the number of network parameters, phenomenon commonly referred to scaling laws, it remains challenging to increase the number of parameters in deep reinforcement learning without experiencing performance degradation. Several recent works in DRL have addressed this by scaling up network sizes through various strategies. Schwarzer et al. (2023) transitioned from the original CNN architecture to the ResNet-based Impala-CNN architecture (Espeholt et al., 2018) and scaled the network width by factor of 4. Both BRO (Nauman et al., 2024b) and SimBa (Lee et al., 2024) employed deeper networks that incorporate layer normalization (Lei Ba et al., 2016) and residual connections. Ceron et al. (2024b) incorporated a soft Mixture-of-Exper Experts module (Puigoerver al., 2020) into value-based networks, resulting in more parameter-scalatee models and improved performance. Farebrother et al. (2024) show that value functions trained using categorical cross-entropy substantially enhance performance and scalability in multiple domains. Ceron et al. (2024a) utilized magnitude pruning on value-based networks, progressively decreasing the number of parameters in dense architectures during training to achieve highly sparse models, leading to improved performance when scaling network width. Despite these advances, scaling up network sizes in DRL using random static sparsity remains underexplored.
A.3. Sparse Networks in DRL
Initial explorations of sparse networks in DRL were primarily motivated by the potential for model compression, aiming to accelerate training and facilitate efficient model deployment (Tan et al., 2023). Early explorations of network sparsification in DRL primarily focused on behavior cloning and offline RL settings (Arnob et al., 2021; Vischer et al., 2021). In the more challenging context of online RL, Sokar et al. (2021) explored the application of Sparse Evolutionary Training (SET) (Mocanu et al., 2018) and successfully achieved 50% sparsity. However, attempts to increase sparsity beyond this level resulted in significant training instability. Subsequently, Tan et al. (2023) enhanced the efficacy of dynamic sparse training through novel delayed multi-step temporal difference target mechanism and dynamic-capacity replay buffer, ultimately achieving sparsity levels of up to 95%. Graesser et al. (2022) conducted comprehensive investigation and demonstrated that pruning consistently outperforms standard dynamic sparse training methods, such as SET (Mocanu et al., 2018) and RigL (Evci et al., 2020). Data Adaptive Pathway Discovery (DAPD) (Arnob et al., 2024) dynamically adjusts network pathways in response to online RI. distribution shifts, maintaining effectiveness at high sparsity levels.
Beyond the initial goal of achieving parameter-efficient architectures through sparsity, recent studies have recognized that sparse and adaptive networks can enhance DRL model scalability while mitigating training pathologies such as plasticity loss. For instance, Ceron et al. (2024a) shows that applying gradual magnitude pruning to large models significantly enhances the performance of value-based agents. Similarly, Ceron et al. (2024b) demonstrates that incorporating Soft MoEs into value-based RL networks enables better parameter scaling. Furthermore, Neuroplastic Expansion (Liu et al., 2024) addresses plasticity challenges by progressively evolving networks from sparse to dense architectures, effectively leveraging increased model capacity. Although these approaches differ in implementation, they all fall under the broader category of dynamic sparse training, where network topology evolves during training. In contrast, this work isolates sparsity as a standalone feature, revealing that static sparse training through random pruning initialization alone substantially enhance DRL network scalability, addressing a significant gap in current research.
13

---

Network Sparsity Unlocks the Scaling Potential of DRL
B. Detailed Experimental Setup and Results of Main Experiments
This section details the experimental setup and the results our evaluation Section and Section The code available in the supplementary materials.
B.1. Detailed Experimental Setup
We evaluate SAC and DDPG with SimBa network with varying sizes and sparsity levels on hard tasks of DeepMind Control Suites (Tassa al., 2018), also known DMC Hard. The complete list for DMC Hard is provided Table 1. Note that we omit Dog Stand from this since the default SimBa architecture already demonstrates strong performance on this task, consistently achieving scores above 900 the normalized 1000-point scale.
Table 1. DMC Hard consists of 6 continuous control tasks.
Task
Observation dim
Action dim
Dog Run
223
38
Dog Trot
223
38
Dog Walk
223
38
Humanoid Run
67
24
Humanoid Stand
67
24
Humanoid Walk
67
24
The experimental settings for DMC in Section 3 and Section 4 are primarily adapted from those employed in SimBa. Most of the hyperparameters in our experiments are identical to those used in Lee et al. (2024), except for the network width (hidden dimension) and depth (number blocks), as detailed in Table and Table 3. Unless otherwise specified, all experiments are conducted using 8 random seeds.
Table SAC hyperparameters. The hyperparameters listed below used consistently all experiments in Section and Section 4, For the discount factor, we follow Lee et al. (2024) using heuristics used by TD-MPC2 (Hansen et al., 2023).
Hyperparameter
Value
Critic block type
SimBa
Critic num blocks
{2,4,6.8)
Critic hidden dim
(2665122.1044,5532244626666
Critic learning rate
le-4
Target critic momentum (r)
5e-3
Actor block type
SimBa
Actor num blocks
(1,2,3.4)
Actor hidden dim
(64.128.256.884.512..044
Actor learning rate
le-4
Initial temperature (ao)
lc-2
Temperature learning rate Target entropy (Hl*)
le-4
|A|/2
Batch size
256
Optimizer
AdamW
Optimizer momentum (81, 82)
(0.9, 0.999)
Weight decay (X)
1e-2
Discount (7)
Heuristic
Replay ratio
2
Clipped Double Q
False
14

---

Network Sparsity Unlocks the Scaling Potential of DRL
Table DDPG hyperparameters. The hyperparameters listed below are used consistently across all experiments in Section and
Section 4, For the discount factor, we follow Lee et al. (2024) using heuristics used by ID-MPC2 (Hansen et al., 2023).
Hyperparameter
Value
Critic block type
SimBa
Critic num blocks
{2,4.6.8)
Critic hidden dim
(266:12..024415622282.22))))
Critic learning rate
le-4
Target critic momentum (T)
5e-3
Actor block type
SimBa
Actor num blocks
(1,2.3.4)
Actor hidden dim
(64,128,256,38845122,6
Actor learning rate
le-4
Exploration noise
W((0, 0.13)
Batch size
256
Optimizer
Adam
Optimizer momentum (31, (2)
(0.9, 0.999)
Weight decay (A)
le-2
Discount (7)
Heuristic
Replay ratio
2
Clipped Double Q
False
B.2. Detailed DMC Results
Scaling Trends Visualization. Figure 14 presents an alternative visualization of the model scaling results shown in Figure 2, using a linear scale for model size instead of the logarithmic scale used in the main text. This alternative visualization emphasizes the widening performance gap between sparse and dense networks, which becomes particularly pronounced at larger model scales (> 100M parameters).
Humanoid
Dog Trot Run
Humanoid
Run
Dog Run
SAC
SAF
DDPG
DDPC
208
306100
Deptho
(ealnng
Figure Model scaling trends * * sparse versus oeedense networks on four hardest DMC tasks using SimBa architecture with SAC and DDPG. The data points in this figure are identical to those shown in Figure I: however, this figure employs linear scale for model size, providing an alternative view of the scaling relationships.
15

---

Network Sparsity Unlocks the Scaling Potential of DRL
Single Task Results. We provide detailed breakdown of the scaling trends for individual tasks that were aggregated in Figure 2. The width scaling results are presented in Figure 15, while the depth scaling results shown in Figure 16. For each task, we use the optimal dense network as a reference point and explore larger model sizes with appropriate sparsity levels to maintain constant parameter counts. The consistency between individual task trends (Figure 15 and Figure 16) and the aggregated results (Figure 2) reinforces the generality of our findings.
Dog Walk (SAC)
Dog Trot (SAC)
Dog Run (SAC)
Diiton Nitwork
100
Dacza Natwvrrs
Denas Network
Spoine Notwork
Spaa se Nidwwerk
Sparse Neiwork
132
324
770
200
624
006
Witth Sclle
Watt Scale
Witth Scale
Humanoid Stand (SAC)
Humanoid Walk (SAC)
Humanoid Run (SAC)
0c6
240
102
724
700
200
643
175
150
Deeaar Natwerk
12x
Sparia on Natwerkk
Milth
Dog Walk (DDPG)
Dog Trot (DDPG)
Dog Run (DDPG)
Dozar Nodkeotk
Deioe Nelweek
Spacse Xitwork
+
Spares Network
300
623
600
680
300
206
200
753
414
Notwork
Widta Scale
Walh Scaie
Malc Scaie
Humanoid Stand (DDPG)
Humanoid Walk (DDPG)
Humanoid Run (DDPG)
Distae Natwark
754
Decas Nerwora
Denoe Network
Spores
66.0
204
654
Figure 15. Width scaling experiments comparing dense and sparse networks across all DMC Hard tasks. Results show episode returns for both SAC (top two rows) and DDPG (bottom two rows) implementations on six challenging control tasks. Each data point represents the mean performance across random seeds, with emror bars indicating standard deviation.
16

---

Network Sparsity Unlocks the Scaling Potential of DRL
Dog Walk (SAC)
Dog Trot (SAC)
Dog Run (SAC)
024
$33
240
644
371
192
724
400
700
Esjande
643
53
100
54.
Dosso Kotwork Space Notwark
Deiso Nerwerss
Denon Network
204
Spur Network
Sparss Nenwock
Depth
Depth
Humanoid Stand (SAC)
Humanoid Walk (SAC)
Humanoid Run (SAC)
Deron Ninwerk Spurm Nebwerk
300
220
Denen Network Spars Neswock
692
200
740
Dease Netwock
223
Epitede
Spare Network
024
100
170
640
100
Dep'h Stale
Depth Scale
Dopth Scale
Dog Walk (DDPG)
Dog Trot (DDPG)
Dog Run (DDPG)
50
724
382
7E0
400
64>
100
233
783
204
Spazn Netwark
Dosso Kitwark
424
Deisoo Nerwars
Netwark
Sparaa Neswodk
Denin Network
Deph Scale
Depli Scale
Depth Scale
Humanoid Stand (DDPG)
Humanoid Walk (DDPG)
Humanoid Run (DDPG)
100
100
383
100
683
683
140 Deroe Stiweek
583
Duza: Notweck
Spars Ketwark
Spares Network
Dep'h Soale
Scale
Scale
Figure 16. Depth scaling experiments comparing dense and sparse networks across all DMC Hard tasks. Results show episode returns for both SAC (top two rows) and DDPG (bottom two rows) implementations on six challenging control tasks. Each data point represents the mean performance across random seeds, with error hars indicating standard deviation.
17

---

Network Sparsity Unlocks the Scaling Potential of DRL
C. Detailed Experimental Setup of broader setups
C.1. Visual RL
We conducted visual RL experiments on DMC using image input as the observation. All experiments were based on the DrQ- v2 (Yarats al., 2022), with all hyperparameters retained from the original DrQ-v2 implementation. The sole modification involved adjusting the width of the critic network accommodate specific experimental settings. The hyperparameters presented in Table 4.
In Figure II, we use the Fraction of Active Units (FAU) metric for measuring plasticity. The FAU for neurons located
in module M, denoted as 2 M, is formally defined as:
ΣnEm (an(()) 0)
4
(3)
N
where (x) represent the activation of neuron n given the input and N is the total number of neurons within module M.
Table 4. DrQ-v2 hyperparameters.
Hyperparameter
Value
Replay buffer capacity
106
Action repeat
Seed frames
4000
Exploration steps
2000
n-step returns
Mini-batch size
256
Discount y
0.99
Optimizer
Adam
Learning rate
10-4
Critic Q-function soft-update rate
0.01
Features dim.
50
Repr. dim.
32 x 35 x 35
Hidden dim.
1024
Exploration stddev. clip
0.3
Exploration stddev, schedule
linear (1.0, 0.1, 500000)
18

---

Network Sparsity Unlocks the Scaling Potential of DRL
C.2. Streaming RL
We conducted streaming RL experiments on MuJoCo robot locomotion tasks (Todorov al., 2012), Ant-v4 and Walker2d-v4. All experiments were based on the Stream AC(A) algorithm (Elsayed et al., 2024), with all hyperparameters retained from the original Stream AC(A) implementation. The sole modification involved adjusting the width of the actor network and the critic network to accommodate specific experimental settings. The learning curves for agents with different network widths and sparsity levels are presented in Figure 17.
Ant-vd (Width-64))
Ant-vd (Width-128)
Ant-vd (Width-256)
Ant-vd (Widhh-512)
0.5
2.0
0.5
0.5
24
X10'
X30"
X30
X10
Walloor2d.ve 19 (Width-64)
Walkerrdd vá (Wieth-128)
Walkerr2d v4.. (Wleth-236)
Walkerr2d (Width-511)
25
Iptoonde Neturm
86
303
2-4
2.8
25
Environment Stops
Enviremment Stapa
Eavironment Stapa
Esvironmant Stepa
Sparstyy600
SpanstynO:
Spenstyym..
Figure Learning curves of Stream AC(A) agent on Ant-v4 and Walker2d-v4, evaluated across varying sparsity levels and network widths for both the actor and critic networks.
C.3. Atari-100k
We conducted Atari experiments on the Atari- 100k benchmark (Kaiser et al., 2020), where the agent may perform only 100K environment steps, roughly equivalent to two hours of human gameplay. Our experiments were based Data Efficient Rainbow (DER) (Van Hasselt et al., 2019), variant of Rainbow (Hessel et al., 2018) tuned for sample efficiency. All experiments were based on Dopamine (Castro et al., 2018), except that we used an IMPALA CNN architecture (Espeholt et al., 2018) instead of NatureCNN. The byperparameters are presented in Table 5.
We increase the width of the IMPALA CNN by factor three and evaluate three static sparsity configurations: dense (0.0), moderate sparsity (0.4), and high sparsity (0.8). Figure shows the improvement relative to the default setting, which uses the default size and dense network. As in previous experiments, the results indicate that introducing static sparsity into DRL networks unlock their scaling potential and yield performance improvements. We note that the Atari-1000 low-data regime may not fully demonstrate the benefits of scaling, and more comprehensive studies with longer training would be valuable for future work.
19

---

Network Sparsity Unlocks the Scaling Potential of DRL
+107
07%
13MPS
0%
0%
Breakout
ChoperCommand
CrazyCllmber
DemonAttack
1007
+019
+00%%
-30%
-21%
-275
+276
+20%
.2%
Prosthite
Gopher
Hero
Jamssboond
Kangaroo
Krull
25%
+379
20%
-30%
13%
09
0%
0%
Kusnter
MaPaoman
Pong
PricateFye
Obart
Readrunner
+100%
1000
+100%
+302%
+277
+207%
+100%
07%
0%
Sencgoost
+100%%
Width Scale=3, Sparsity=0.0 0.0
02%
Width Scale=3, S0a 0.4
Width Scale=3, Sparsity=0.8
Figure Performance improvements on Atari-100k benchmark when scaling network width (3x default size) with different sparsity levels using Data Efficient Rainbow (DER). Each bar represents the percentage improvement relative the default dense network configuration. The results demonstrate that introducing sparsity (both at 0.4 and 0.8 levels) generally yields better performance than dense models (0.0, blue) when scaling model size, with optimal sparsity levels varying across different games. This extends findings from continuous control domains to discrete action spaces, suggesting that the benefits of network sparsity are robust across different reinforcement learning environments.
20

---

Network Sparsity Unlocks the Scaling Potential of DRL
Table 5. DER hyperparameters.
Hyperparameter
Value
Gray-scaling
True
Observation down-sampling
84x84
Frames stacked
Action repetitions
Reward clipping
[-1, 1]
Terminal loss of life
True
Update
Distributional
Q
Dueling
True
Support of Q-distribution
51
Discount factor
0.99
Minibatch size
32
Optimizer
Adam
Optimizer: learning rate
0.0001
Optimizer:
0.00015
Exploration
Noisy nets
Noisy nets parameter
0.5
Training steps
100K
Evaluation trajectories
100
Min replay size for sampling
1600
Updates per step
Multi-step return length
10
CNN network
IMPALA CNN (Espeholt et al., 2018)
Target network update period
2000
21


Reviews:
[{"review_id": "UZDM1bK1id", "reviewer": "Reviewer_yHy2", "summary": "This paper shows that current deep RL architectures result in performance decreases when scaling the network in width or depth. Introducing sparsity in the form of a fixed mask over the weights, is able to to resolve this issue. The source of the benefits of sparsity is investigated in terms of representational capacity, plasticity and gradient interference. O\nFurther experiments demonstrate the utility of the method in visual RL and streaming RL.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: Limited DMC environments affecting generalizability\n\nWe acknowledge the reviewer's concern that most experiments are conducted on DMC, which might impact the generalizability of our analyses. To address this, we've conducted new experiments on the Atari-100k benchmark using Data Efficient Rainbow DQN (DER) [1] as our baseline algorithm.\n\nThe experiments compare performance improvements when scaling network width to 3x the default size with varying sparsity levels (0.0, 0.4, and 0.8). Results are shown in [`figure (anonymous link)`](https://anonymous.4open.science/r/ICML_2025_3388/atari_sparsity_results.pdf).\n\nThese results show that introducing network sparsity while scaling up model size produces similar benefits in discrete action tasks as observed in our continuous control experiments. This further validates the general effectiveness of network sparsity for scaling DRL models across different domains and algorithms.\n\nWe note that the Atari-100k low-data regime may not fully demonstrate the benefits of scaling, and more comprehensive studies with longer training would be valuable for future work. Nevertheless, these preliminary results provide additional evidence supporting our main findings about network sparsity's role in unlocking DRL scaling potential.\n\n[1] When to use parametric models in reinforcement learning?, NeurIPS 2019\n\n> **Q2**: Fixed uniform sparsity ratios vs. Erdos-Renyi ratios\n\nBased on the reviewer's question, we conducted new experiments comparing the naive approach of fixed/uniform sparsity ratios across all layers with the Erdos-Renyi (ER) ratio. The results shown in the [`figure (anonymous link)`](https://anonymous.4open.science/r/ICML_2025_3388/ER_Uniform_Ratios.jpg) reveal that at lower sparsity levels (≤0.6), both initialization methods perform comparably, while at higher sparsities (≥0.8), uniform layer-wise ratios exhibit a dramatic performance drop.\n\nThe advantage of ER initialization stems from its ability to provide more balanced information flow across network layers. By scaling sparsity proportionally to the geometric mean of input/output dimensions, ER maintains approximately equal fan-in/fan-out ratios across layers, preventing bottlenecks that emerge with uniform sparsity at high sparsity levels where network connectivity becomes critical for information propagation.\n\nWe want to emphasize that our paper's focus is on demonstrating how network sparsity as a fundamental property enables DRL model scaling, rather than identifying the optimal sparse topology. The fact that uniform sparsity at appropriate levels also improves performance further supports our main claim that sparsity itself (regardless of specific implementation) is a key enabler for unlocking the scaling potential of DRL networks.\n\n> **Q3**: Random sparsity benefits vs. Lottery Ticket Hypothesis findings\n\nThe reviewer raises an important question about reconciling our findings with the Lottery Ticket Hypothesis (LTH). The key difference lies in the fundamentally different problem settings and objectives:\n\nFirst, we must distinguish between previous sparse training studies and our work. Earlier studies on sparse training (including LTH) were motivated by model compression to reduce computational costs while maintaining performance. Such approaches operate under the assumption that in supervised or unsupervised learning, larger dense networks generally yield better performance—the core premise behind modern scaling laws.\n\nHowever, this assumption is violated in online DRL settings. As demonstrated in Figure 1 of our manuscript, online DRL networks face severe scaling barriers where increasing model size not only fails to improve performance but often leads to catastrophic collapse. Our analysis in Section 4 further reveals that these scaling barriers emerge because larger dense networks are more susceptible to optimization pathologies during online RL training, preventing them from leveraging their theoretical capacity.\n\nOur work aims to demonstrate that network sparsity itself, as a fundamental network property, can mitigate these pathologies and unlock the scaling potential of DRL networks. This represents a fundamentally different objective than LTH's focus on finding efficient subnetworks within larger models.\n\nTherefore, our findings don't contradict the Lottery Ticket Hypothesis but rather highlight the unique utility of network sparsity in addressing online DRL's specific pathologies and inscalability. For a given sparsity level, we believe that better topologies (such as \"winning tickets\") could outperform one-shot random pruning. However, finding tickets requires significant computational resources, and discovering \"lottery tickets\" without training remains challenging. Efficiently identifying winning tickets as trainable networks represents valuable future work, but identifying optimal sparse topologies was not the focus of our current study.", "claims_and_evidence": "This paper has very comprehensive experiments and does an excellent job at demonstrating the usefulness of introducing sparsity. \nThe variety of ablations and analyses help support the main points and also give some nice insights into why this method may be working.", "methods_and_evaluation_criteria": "The benchmarks chosen are standard and appropriate for this setting.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Extensive analyses are done on various aspects of the method (sparsity ratio, effect of scale) with appropriate experimental choices.\nDeeper looks into the effect on metrics such as reprentational capacity, plasticity (dormant neurons, gradient norms), gradient interference and simplicity bias give a more comprehensive view of the effects of sparsity.", "relation_to_broader_scientific_literature": "This paper makes a surprising observation: static sparsity can result in large performance gains and scaling.\nThis finding could make RL researchers rethink how to design new network architectures and I think these results would be of great interest to the RL community.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The presentation is very clear and visually-pleasing, with nice use of \"callout blocks\" of different colors to highlight certain takeaways or important points.\n\nThe writing and organization are also excellent and the paper is information-dense but still easy to follow.\n\nA weakness of the paper is that most of the experiments are conducted on four environments from DMC, which may impact the generalizability of the analyses. There are other experiments with visual and streaming RL though, which can support the idea that sparsity would still be useful in other settings. Overall, I think the currente experiments are adequate.", "other_comments_or_suggestions": "None", "questions_for_authors": "Some clarification questions:\n- Did you try the more naive appraoch of a fixed sparsity ratio for every layer rather than the Erdos-Renyi ratio? Is this ineffective or only less effective?\n\n- In the Lottery Ticket Hypothesis paper, the authors do not observe any performance benefits when introducing sparsity with random weights (rather than the winning ticket). How would you reconcile their result with this paper's results, where large benefits seem to be observed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FwLo7qDl03", "reviewer": "Reviewer_W4Bt", "summary": "This paper uncovers an interesting finding: Instead of pursuing more complex modifications, introducing static Network sparsity alone can unlock further scaling potential beyond their dense counterparts with state-of-the-art architectures.  And in experiments, they show that only using one-step random pruning can achieve great performance in several commonly used benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive evaluation of our work.\n\n> Q: What would happen to the learning ability when you only prune the critic or actor? Does a sparse actor play a key role?\n\nThis is an excellent question about the relative importance of sparsity in different components of actor-critic architectures. We conducted additional experiments comparing four configurations: dense networks, sparse actor only, sparse critic only, and sparse for both actor and critic.\n\nOur results shown in [`figure (anonymous link)`](https://anonymous.4open.science/r/ICML_2025_3388/Actor%20vs%20Critic.jpg) clearly show that applying sparsity to both actor and critic networks yields substantially better performance than either partial approach. Interestingly, when sparsity is applied to only one component (either actor or critic), the scaling curves closely resemble those of the dense baseline.\n\nThis suggests that scaling benefits emerge from a balanced application of sparsity across the entire architecture rather than from any single component. We believe this occurs for several reasons:\n\n1. Critics require sparsity to avoid optimization pathologies during online TD learning. As demonstrated in Section 4, scaling up dense DRL models leads to severe plasticity loss in critics and diminished value representation capacity.\n2. The need for sparse actors aligns with previous findings [1,2] showing that actors are particularly sensitive to network scaling. In actor-critic methods, since actor learning depends on critic outputs, unnecessarily complex actor networks can impede performance. The specific mechanisms behind sparse actor benefits warrant further investigation.\n3. The balance between actor and critic parameters is crucial. Our experiments use the SimBa architecture, which carefully established optimal component sizing [1] - in the default 4.51M parameter configuration, the critic accounts for 4.34M parameters. When scaling our networks, we proportionally increased both actor and critic sizes, already accounting for their different representational capacity requirements. By maintaining this ratio during scaling, applying sparsity to both components becomes necessary to preserve the architectural balance that makes SimBa effective.\n\n[1] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning, ICLR 2025\n\n[2] Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control, NeurIPS 2024", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Their Method makes sense for the problem.", "theoretical_claims": "The theory in the paper is solid.", "experimental_designs_or_analyses": "I checked the experimental designs, and I think they are valid.", "relation_to_broader_scientific_literature": "I think this paper made a general contribution to the scaling law of the AGI.", "essential_references_not_discussed": "I think the related works analyzed in this paper are wide.", "other_strengths_and_weaknesses": "1) Well-written, clear summary\n2) completely experimental analysis\n3) rich and solid visualization metrics testing\n4) makes an important contribution to the understanding of the scaling law in Deep RL community", "other_comments_or_suggestions": "One thing I did not find in the paper is which part of the AC architecture should be more important -- actor or critic. What would happen to the learning ability when you only prune the critic or actor? Does a sparse actor play a key role?", "questions_for_authors": "Please see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iioYEqbBX8", "reviewer": "Reviewer_BkgJ", "summary": "This paper explores the scalability benefits of incorporating static network sparsity in deep reinforcement learning models. It introduces one-shot random pruning, where a fixed proportion of network weights are removed before training, leading to improved parameter efficiency compared to scaling up dense architectures. The analysis highlights that sparse networks enhance expressivity while alleviating optimization challenges such as plasticity loss and gradient interference. Furthermore, experiments on visual and streaming RL tasks demonstrate the robustness of sparsity, showcasing its consistent advantages across diverse reinforcement learning scenarios.\n\n## update after rebuttal\nThe authors provided additional results of DER on Atari, which also shows the sign of the proposed findings. I'll keep my positive evaluation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and positive evaluation of our work.\n\n> It would be interesting to see these observations extended to value-based methods in discrete action spaces, such as DQN on Atari. Nonetheless, the current findings are valuable and worth sharing with the community.\n\nWe agree that extending our findings to value-based methods with discrete action spaces is crucial for demonstrating the broader applicability of network sparsity benefits. To address this, we've conducted new experiments on the Atari-100k benchmark using Data Efficient Rainbow DQN (DER) [1] as our baseline algorithm.\n\nThe experiments compare performance improvements when scaling network width to 3x the default size with varying sparsity levels (0.0, 0.4, and 0.8). Results are shown in [`figure (anonymous link)`](https://anonymous.4open.science/r/ICML_2025_3388/atari_sparsity_results.pdf). *(Note: Due to time constraints before the first round response deadline, we've only completed experiments on 13/26 Atari tasks. We expect to update the figure with complete results within one more day.)*\n\nThese results show that introducing network sparsity while scaling up model size produces similar benefits in discrete action tasks as observed in our continuous control experiments. This further validates the general effectiveness of network sparsity for scaling DRL models across different domains and algorithms.\n\nWe note that the Atari-100k low-data regime may not fully demonstrate the benefits of scaling, and more comprehensive studies with longer training (e.g., 10M environment steps) would be valuable for future work [2]. Nevertheless, these preliminary results provide additional evidence supporting our main findings about network sparsity's role in unlocking DRL scaling potential.\n\n[1] When to use parametric models in reinforcement learning?, NeurIPS 2019\n\n[2] In value-based deep reinforcement learning, a pruned network is a good network, ICML 2024", "claims_and_evidence": "I'm not familiar with the analysis in Sections 4.3 and 4.4, but overall, I believe the corresponding experiments effectively support the claims presented in each section.", "methods_and_evaluation_criteria": "The primary experiments are conducted on the four most challenging DMC tasks using two RL algorithms, SAC and DDPG. Additionally, the authors extend their evaluation to visual and streaming RL. Overall, the evaluation appears solid and well-founded.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I appreciate that Appendix B provides detailed settings for the major experiments. While using eight random seeds per experiment is slightly lower than expected, it is still a reasonable choice.", "relation_to_broader_scientific_literature": "This paper presents insightful and valuable findings on scaling deep reinforcement learning networks. It demonstrates that incorporating static network sparsity through simple one-shot random pruning can enhance scalability and outperform dense counterparts. These findings are not only relevant to DRL but may also have broader implications for other learning paradigms, such as self-supervised learning and language model post-training.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "I appreciate the authors' detailed analysis of RL methods for continuous action control problems. It would be interesting to see these observations extended to value-based methods in discrete action spaces, such as DQN on Atari. Nonetheless, the current findings are valuable and worth sharing with the community.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aCWolVCAIN", "reviewer": "Reviewer_mWRx", "summary": "The paper builds upon previous work on scaling model size in RL [1], and extends its limit with simple layer-wise random pruning at initialization [2]. This is primarily verified in state-based RL (DMC), where a pruned large network greatly surpasses a dense network with the same number of trainable parameters. This is further extended to vision-based RL and streaming RL setting, which all have shown similar tendencies. Finally, they provide an extensive analysis on the benefits of one-shot pruning, specifically in the regime of representation capacity (s-rank), plasticity (dormant ratio, gradient norm, parameter norm), and gradient inference.\n\n[1] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning., ICLR'25.\n\n[2] The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training., ICLR'22.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. We address your questions below.\n\n> Q1: Related work on iterative pruning\n\nWe have already discussed relevant works on dynamic sparse training (DST) in RL within our Introduction and Related Work sections. In the next version, we plan to expand our coverage of more general topology evolution methods, including the papers you mentioned. Thank you for pointing out the valuable references.\n\n> Q2: The connection between gradient norm and plasticity\n\nYour question about distinguishing between 'large' and 'stable' gradient norms is insightful and helps us clarify an important relationship that wasn't fully explained in our manuscript.\n\nIn Figure 5, we observe that large dense networks experience a rapid collapse in gradient norm early in training while performance remains poor. This premature descent into a low gradient state reflects the agent's inability to effectively learn from new experiences, directly corresponding to the slow performance improvement in these networks. By contrast, sparse networks maintain more consistent gradient signals throughout training. Therefore, it's this early declining pattern, rather than absolute magnitude, that indicates plasticity loss.\n\nSeveral studies have explored the relationship between gradient norm collapse and plasticity loss, including:\n\n- Figure 7 in *The Primacy Bias in Deep Reinforcement Learning, ICML 2022*\n- Section 4.2 in *Loss of Plasticity in Continual Deep Reinforcement Learning, CoLLAs 2023*\n- Figure 10 in *Weight Clipping for Deep Continual and Reinforcement Learning, RLC 2024*\n- Figure 5 in *Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning, ICLR 2024*\n\nHow to precisely measure a network's plasticity loss level remains an open question, which is why current work tends to employ multiple different metrics simultaneously for a more comprehensive assessment. The lower dormant ratios, higher SRank values, and Reset diagnostic experiments in Section 4 collectively demonstrate how sparsity helps preserve network plasticity.\n\nYour question has prompted us to reconsider our presentation. Since gradient norm has confounding relationships with training stability, and our other metrics already establish the plasticity benefits of sparse networks, we plan to move this discussion to the appendix with more detailed explanations in our revised manuscript.\n\n> Q3: Further trend of gradient norms beyond 1M steps\n\nYour concern about continuously increasing gradient norms prompted us to extend our experiments to 2M steps on Dog Trot and Run tasks, shown in [`figure (anonymous link)`](https://anonymous.4open.science/r/ICML_2025_3388/GN.jpg), revealing two interesting patterns:\n\nIn Dog Trot, where performance plateaus around 1M steps, gradient norms in sparse networks eventually peak and then gradually decrease, demonstrating natural stabilization.\n\nIn Dog Run, where learning continues beyond 1M steps, gradient norms maintain moderate activity, corresponding directly with ongoing performance improvements.\n\nThese patterns suggest sparse networks achieve an effective balance: they maintain sufficient gradient activity for learning without collapsing (unlike dense networks), while also not becoming unstable over time. We'll include this extended analysis in our revised manuscript's appendix to better illustrate the long-term benefits of network sparsity.\n\n> Q4: Comparison between Erdos-Renyi ratio and other layer-wise ratios\n\nBased on sparse training research in the broader deep learning field, the advantage of ER initialization likely stems from providing more balanced information flow across network layers. By scaling sparsity proportionally to the geometric mean of input/output dimensions (rather than uniformly), ER maintains approximately equal fan-in/fan-out ratios across layers, preventing bottlenecks in both forward and backward passes.\n\nThe core purpose of our paper is to demonstrate how network sparsity as a fundamental property helps scale up DRL model size, rather than comparing different sparsification methods. We chose ER initialization because previous sparse training studies (in both DRL and supervised learning) have established its superiority over uniform initialization. To directly compare these approaches in our setting, we conducted additional experiments shown in the [`figure (anonymous link)`](https://anonymous.4open.science/r/ICML_2025_3388/ER_Uniform_Ratios.jpg).\n\nThe results reveal that at lower sparsity levels (≤0.6), both initialization methods perform comparably. However, at higher sparsities (≥0.8), uniform layer-wise ratios exhibit a dramatic performance drop. These findings suggest that:\n\n1. Sparsity itself benefits DRL scalability regardless of the specific layer-wise ratio configuration.\n2. ER initialization provides greater robustness, especially at high sparsity levels, by creating a more balanced network topology that better maintains information flow.", "claims_and_evidence": "The effectiveness of one-shot pruning has been shown by prior work: The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training (ICLR'22).", "methods_and_evaluation_criteria": "The proposed method (one-shot pruning) is well verified to 'unlock parameter scaling in RL', as shown by the consistent improvement in challenging DMC Hard tasks, which prior work (Simba) fails to follow.", "theoretical_claims": "No theoretical claims were made.", "experimental_designs_or_analyses": "Authors clearly show the benefits of one-shot pruning with metrics commonly used in diverse fields:\n\n1. Higher s-rank (larger representation capacity)\n2. Lower dormant ratio (better plasticity preservation [1])\n3. Larger gradient norm (better plasticity preservation)\n4. Smaller parameter norm (better plasticity preservation [2])\n5. Reset doesn’t help (proof of plasticity preservation). \n6. Higher simplicity bias score [3]\n7. Gradients closer to orthogonal (less gradient inference [4])\n\n[1] The Dormant Neuron Phenomenon in Deep Reinforcement Learning., ICML'23.\n\n[2] Normalization and effective learning rates in reinforcement learning., NeurIPS'24.\n\n[3] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning., ICLR'25.\n\n[4] Understanding Plasticity in Neural Networks., ICML'23.", "relation_to_broader_scientific_literature": "Unlocking parameter scaling in RL is a contribution towards a large foundational model in RL.", "essential_references_not_discussed": "I think most references in mind were present. However, the works on iterative pruning could be included:\n1. In deep reinforcement learning, a pruned network is a good network., ICML’24.\n2. Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective., NeurIPS’21.", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "Overall, I think this paper is well-written, well-analysed with interesting results.", "questions_for_authors": "1. It is unclear to me how ‘large’ gradient norm could be a proof of better plasticity. I think it’s more persuasive to claim that a ‘stable’ gradient norm rather than the norm itself. If there’s a literature around the connection between gradient norm and plasticity, it’d be nice it they’re included in the Gradient Norm section.\n2. On that note, what do you think about the gradient norms consistently increasing in Figure 5? It seems like they will keep increasing when trained beyond 1M steps, and I don’t think that’s something we’d want in terms of stability.\n3. Slightly off topic: are there any intuition on why Erdos-Renyi ratio is superior to uniform or other layer-wise ratios? Why is it better to have the ratios linear to the number of input/output neurons rather than quadratic?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper shows that scaling sparse neural networks trained with static network sparsity not only improves parameter efficiency but also enhances resistance to plasticity loss and gradient interference, compared to scaling dense neural networks. All reviews are very positive, recognize the novelty of the findings, and appreciate the presentation of the paper. Therefore, the AC suggests acceptance and encourages the authors to include the rebuttal discussion in the final version of the paper.
Decision: Accept (oral)