Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning
Authors: Andy Li
Aiden Durrant
Milan Markovic
Tianjin Huang
Souvik Kundu
Tianlong Chen
Lu Yin
Georgios Leontidis
Venue: Accepted by TMLR ()

Abstract: Pruning of deep neural networks has been an effective technique for reducing model size while preserving most of the performance of dense networks, crucial for deploying models on memory and power-constrained devices. While recent sparse learning methods have shown promising performance up to moderate sparsity levels such as 95% and 98%, accuracy quickly deteriorates when pushing sparsities to extreme levels due to unique challenges such as fragile gradient flow. In this work, we explore network performance beyond the commonly studied sparsities, and develop techniques that encourage stable training without accuracy collapse even at extreme sparsities, including 99.90%, 99.95\% and 99.99% on ResNet architectures. We propose three complementary techniques that enhance sparse training through different mechanisms: 1) Dynamic ReLU phasing, where DyReLU initially allows for richer parameter exploration before being gradually replaced by standard ReLU, 2) weight sharing which reuses parameters within a residual layer while maintaining the same number of learnable parameters, and 3) cyclic sparsity, where both sparsity levels and sparsity patterns evolve dynamically throughout training to better encourage parameter exploration. We evaluate our method, which we term Extreme Adaptive Sparse Training (EAST) at extreme sparsities using ResNet-34 and ResNet-50 on CIFAR-10, CIFAR-100, and ImageNet,achieving competitive or improved performance compared to existing methods, with notable gains at extreme sparsity levels.

Published in Transactions on Machine Learning Research (11/2025)
Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning
Andy Li
a.1..21 1..21@abddn.co uk
Deportment of Computing Science University of Aberdeen, UK
Aiden Durrant
aiden.duwranttlaaddn.cccc
Department of Compuling Science University of Aberdeen, UK
Milan Markovic
mllan.maarkovic@@adddd.cccc
Department of Computing Science & Interdisciplinary Institute University of Aberdeen, UK
Tianjin Huang
t.hunng@@eneter.cco ac.
Department of Computer Science University of Exeler, UK
Souvik Kundu
kouvikkkuunde Gintell com
Intel Labs, USA
Tianlong Chen
tianlong@es
unc. www.://wsscsmcccccmmcmm
Department of Compuler Science University of North Carolina at Chapel Hill, US
Lu Yin
1. Lyinntsurrey..cccoo
School of Computer Science and Electronic Engineering University of Surrey, UK
Georgios Leontidis
seriiiiss.oootiii@@aaaaa uk
Department of Computing Science & Interdisciplinary Institute University of Aberdeen, UK
Reviewed on OpenReview: https: // openreview. net/ forum? id- XX9JdOJDSR
Abstract
Pruning of deep neural networks has been elfective technique for reducing model size while preserving most of the performance of dense networks, crucial for deploying mod- els on memory and power-constrained devices. While recent sparse learning methods have shown promising performance up to moderate sparsity levels such as 95% and 98%, accuracy quickly deteriorates when pushing sparsities to extreme levels due unique challenges such fragile gradient flow. In this work, we explore network performance beyond the commonly studied sparsities, and develop techniques that encourage stable training without accuracy collapse at extreme sparsities, including 99.90%, 99.95% and 99.99% ResNet archi- tectures. We propose three complementary techniques that enhance sparse training through different mechanisms:: 1) Dynamic ReLU phasing, where DyReLU initially allows for richer parameter exploration before being gradually replaced by standard ReLU, 2) weight shar- ing which reuses parameters within a residual layer while maintaining the same number of learnable parameters, and 3) cyclic sparsity, where both sparsity levels and sparsity patterns evolve dynamically throughout training to better encourage parameter exploration. We eval-

---

Published in Transactions on Machine Learning Research (11/2025)
uate our method, which we term Extreme Adaptive Sparse Training (EAST) at extreme sparsities using ResNet-34 and ResNet-50 on CIFAR-10, CIFAR-100, and ImageNet, achiev- ing competitive or improved performance compared to existing methods, with notable gains at extreme sparsity levels. Code is available at http://hhh....oaoooooooott//////
1 Introduction
Network pruning (Han et al. 2015a bj LeCun et al. 1990: Liu et al. 2017 Li ct al. 2016 Kusupati et al., 2020) is a widely-used technique for reducing a network's parameters and compressing its size. Reducing model sizes is crucial for deploying models on edge devices with limited resources. Conventionally, pruning methods have focused reducing parameters from pre-trained models. However, it requires at least much computation as training dense model it must converge before pruning takes place. The Lottery Ticket. Hypothesis work (Frankle & Carbin 2018 Frankle et al. 2020a Malach et al. 2020) gives theoretical foundation that subnetworks have the potential to reach full performance even when trained from an initially sparse state. This insight has recently gained much traction in sparse training, a paradigm in which sparse networks are trained from scratch without the need for dense pre-training.
Sparse training methods can be broadly classified into two categories: static sparse training (SST), also sometimes referred to Pruning at Initialization (Pal), where the sparsity pattern is pre-determined
initialization and remains fixed throughout training [Lee et al. 2019 Wang el al. 2020 Tanaka et al. 2020a
De Jorge et al.) 2020), and dynamic sparse training (DST), where the sparsity pattern continuously evolves
during training [Mocanu et al. 2018 Dettmers & Zettlemoyer 2019 Evei et al. 2022 Yin et al.
2022). These
methods perform well at moderate sparsity commonly up to 98%, but experience rapid accuracy degradation beyond this due to limitations such as layer collapse and gradient issues (Tanaka et al.) 2020b Wang et al.,
2020). This raises a fundamental question what are the limits of sparse training? Understanding these boundaries is both scientifically valuable for characterizing sparse learning dynamics and polentially relevant for future deployment scenarios, where resource constraints continue to be problematic. However, extreme sparsities remain relatively understudied, with only a few methods attempting to maintain usable accuracy
at these extreme levels (Tanaka et al. 2020b; De Jorge el al. 2020
Price & TTamner 2021
Our work proposes a collection of adaptive methods that can achieve meaningful performance at extreme sparsitics. By leveraging three core strategics - 1) phased Dynamic ReLU activation, 2) weight sharing, 3) cyelic sparsity scheduling we present a flexible framework that is capable of optimizing performance at extreme sparsities. Fach component independently enhances the sparse model's capacity as seen in our ablation studies, where individual strategy improves performance on their own. These methods can also cohesively achieve an even stronger performance when combined. This framework, which we term Extreme Adaptive Sparse Training (EAST), offers a modular approach that can be adapted to existing DST frameworks, postponing their total performance collapsc. Through empirical analysis, we find that EAST prevents gradient vanishing and encourages parameter exploration, achieving notable performance gain over our baselines. Our primary focus on the final compressed model for deployment rather than training efficiency, as these represent distinct optimization objectives in resource-constrained scenarios. Our contribution and key achievements can be summarized as follows:
We introduce EAST method that combines phased Dynamic ReLU activation, weight sharing, and cyclic sparsity scheduling, aimed at retaining performance past common sparsity thresholds.
We perform extensive empirical evaluation of EAST on image classification tasks, and demonstrate competitive performance at sparsities beyond 99.90%.
We explore and demonstrate that DyReLU when used as a drop-in replacement for ReLU ini- tialization and then replaced completely can still improve the parameter exploration at extreme sparsities, leading to higher performance than sparse models trained using ReLU only.
2

---

Published in Transactions on Machine Learning Research (11/2025)
Sparse Initialization
Components
Training Progress
Weight Sharing
== (0p. M o)
(0 All
Middle
Final
Initial Sparse State (ERK)
DyReLU Phasing
DyRei
DyRaLUU
Output
Number of aammterss
Cyclic Sparsity
Trining
Topology Update
(orune
candidates)
Fegrown commections
Iniial State
(b) Grow Phass
(6) Prune Phase
DST Phase (14 connections)
Final Stste (14 connections)
connections
(14 connections)
(17 connections)
(14 connections)
Figure 1: Illustration EAST (top). Starting with ERK-initialized sparse network A (0o. Mo} at sparsity s, EAST employs three key components: DyReLU phasing, weight sharing, and cyclic sparsity to transform the network to final state A'' {0r, Mr], achieving meaningful performance at extreme sparsity levels. The topology update box (bottom) illustrates the connectivity change throughout training; connections are first grown, then pruned, and eventually maintained with fixed sparsity update schedule until completion.
Unlike existing DST methods that maintain fixed sparsity levels, EAST introduccs cyclic sparsity scheduling where both patterns and sparsity levels evolve dynamically during training. Our cyclic sparsity schedule allows models to explore parameters and stabilize at extreme sparsity, as evidenced by our ablation studies.
Unlike DCTpS, EAST achieves extreme sparsity without requiring dense matrix computations for inference, resulting a truly lightweight and efficient model for edge deployment. We also demon- strate the versatility of our method by combining with DCT layers on top of DST, consistently improving accuracy at the tested settings.
ResNet:34 34 on CIAR-110
ResNet-34 34 on CIFAR-100
ResNet--0 50 on CIFRR-10
ResNet-50 on CIFAR-100
Test (96)
80
60
20
99.0%
99.9%
99 9s
89 9g%
99.0%
99.0%
99 us
99 a9%
98
99.9
99 ss
99 3999
99,0%
99.5%
99. 93.95
99 99%
Sparsity
Sparsity
Sparsity
Sparsity
RigL
SET
EAST
SynFlow
Figure 2: Comparison of test accuracies across different sparsities. Each point represents median accuracy over three runs with different seeds, and the shaded regions highlight the variability across runs.
3

---

Published in Transactions on Machine Learning Research (11/2025)
2 Related Work
Dynamic Sparse Training. Early work by Mocanu et al. (2018) demonstrated the viability of DST through SET, which used magnitude pruning and random weight regrowth. Subsequent criteria for regrow- ing connections have been developed, such as using the momentum of parameters in SNFS Dettmers & Zettlemoyor (2019) or their gradient in RigL Evei et al. (2021) to determine the salience of deactivated connections. GraNet Liu et al. (2021a) adapled RigL's approach by starting with half the parameters and gradually increasing target sparsity instead of starting from the target sparsity. ITOP Liu et al. (2021b) provides deeper understanding of pre-existing DST methods and enhances parameter exploration during sparse training. Supticket Yin et al. (2022) further enhances the performance of DST by leveraging weight averaging at the late training stage. NeurRev Li et. al. (2024) addressed the issue of dormant. neurons by removing harmful negative weights. Recent advances include Top-KAST Jayakumar et al. (2020), which
maintains an auxiliary set of woights for enhanced exploration, and BiDST Ji et al. (2024) which formulates DST as a bi-level optimization problem for joint weight-mask optimization. MEST Yuan el al. (2021) int.ro
duces memory-efficiet sparse training framework for edge devices, while Chase Yin et al. (2024) adapts
dynamic unstructured sparsity into hardware-friendly thannel-wise sparsity, both optimizing performance on resource-cnnsrraine systems without sacrificing accuracy. SRigL [asby et al. (2024) extends the RigI-based pipeline to structured sparsity and imposes a constant fan-in N:M for better hardware efficiency. CHT [Zhang ct al. 2025) adopts regrowth strategy that does not require the use of gradient information in backpropaga- tion, and solely relies on topological information instead, allowing the regrowth computation to be cheaper. DSCR Wu et al. (2025) shows that, at moderate sparsity, dynamically sparse models can outperform dense ones robustness to common image corruptions. DynaDiag Wu et al. (2025) restricts parameters to a di- agonal sparsity pattern, which preserves full input output coverage while enabling GPU-friendly structured sparsity and dynamic mask updates within that pattern.
Extreme Sparsity Methods. Most current methods focus on sparsity levels at which pruned networks match or closely approximate the performance of their dense counterparts, concept referred to as matching sparsily (Frankle et al.) 2020b). However, there is growing interest in extreme sparsities, where accuracy trade-offss become inevitable but potentially worthwhile for substantial computational gains. Yet there re- main challenges, such as maintaining stable gradient flow and preventing layer collapse, which conventional pruning methods struggle to address. GraSP Wang al. (2020) and SynFlow Tanaka al. (2020b) specif- ically addressed the gradient preservation and layer collapse issues at high sparsities via careful network initialization. SNIP Lee et al. (2019) and SNIP-it, Verdenius et al. (2020) proposed Pal methods that showed promising results at higher sparsities, though primarily focused on static masks. [e Jorge et al. (2020)
demonstrated that methods like SNIP and GraSP can perform worse than random pruning at 99% sparsity and beyond, and proposed iterative pruning approaches (Iter-NNIP and FORCE) that maintained meaning- ful performance up to 99.5% sparsity through gradual parameter elimination. DCTpS Price & Tanner (2021) introduced "DCT plus Sparse' layers that combined fixed dense DCT offset matrix with trainable sparse parameters to maintain propagation at extreme sparsities up to 99.99%.
Price
Activation Functions. ictivation functions in sparse networks have traditionally relied on static activation functions such as ReLU (Nair & Hinton 2010), which applies fixed threshold to the input. Parametric variants such as PReLU He et al. (2015) introduced learnable paramcters to adapt the negative slope, offering slight improvements in gradient flow. Swish RRamachandran et al. (2017) and Mish Misra| (2019) aim to smooth gradients for better optimization but remain static throughout training. Dynamic ReLU (DyReLU) Chen et al. (2020) introduced an adaptive activation that dynamically adjusts the slopes of a ReLU based on the input.
3 Methodology
3.1 Activation Function
In this work, when initializing a network, we replace the standard ReLU activation with a non-ReLU acti- vation. We tested several activations and found that DyReLU performed best, specifically the DyReLU-B variant, which dynamically adapts and adjusts whannel-wise activation coefficients during training, enabling
4

---

Published in Transactions on Machine Learning Research (11/2025)
Overn Pocitive Fre sct.attoon 190
Poxtive Peeattivstion (96
Phasing Degna
10.0
40-
Phasing
30-
5.0
20
2.5
0.0
Ovent
Ins comy
uyer
layer.11
1ayer12
ayye2.0
laye22.
layo22
laye2.
layer2.
layer222
layer3.2
lave3.3
1aye2.4
tayora. 5
ayerd
layer
layerk.2
150
Epoch
Recu
DyReLL Rew
ReLU
DyReLU ReLU
Figure 3: Positive pre-activations analysis in ResNet-34 at 99.99% sparsity. The left figure shows layerwise- comparison of positive pre-activations after DyReLU is completely converted to ReLU. The right figure shows their overall amount before, during and after DyReLU phasing.
richer expressivity during early training. As training progresses, we gradually transition from DyReLU to standard ReLU, climinating the additional parameters. Preliminary details about DST, DyRcLU and details on phasing out DyReL.U can be found in the Supplementary Materials.
Understanding Gradient Flow Through Neuron Activity. With ReLU, non-positive pre-activations would result in zero gradient and prevent the learning signal from propagating. To better understand why DyReI.U phasing improves performance at extreme sparsity, we analyzed pre-activation values throughout training. As shown in Figure g the model trained solely with ReLU maintains around 1.3% positive pre- activations throughout training, with many deeper layers showing none, interrupting gradient propagation.
In contrast, our DyReLUU phasing approach naturally begins from high 56.8%, decreases during the tran- sition phase, and stabilizes at 4.5% even after DyReLU is completely converted to ReLU, three times higher than with ReLU alone. The deeper layers also retain some posilive pre-activations, preserving critical acti- vation pathways that would otherwise be lost. This allows extremely sparse networks to continue learning where conventional approaches would struggle due to interrupted gradient propagation. We detail the abla- tion studies on the effect of DyReLU phasing in Section 4.2
This mechanism of preserving activation pathways directly translates to sustained gradient flow during training. As demonstrated in Figure] 5 networks trained with DyReLU phasing maintain healthy gradient norms even after complete conversion to standard ReLU, whereas networks trained solely with ReLU exhibit near-zero gradient flow and complete performance collapse.
Phasing Out DyReLU. As training progresses, we phase out all DyReLU and gradually replace them with the standard ReLU. We introduce a hyperparameter that regulates the contribution of DyReLU during training. It makes full contribution when it is 1, and gradually decreases to 0 as training progresses, eventually reaching 0 before the first learning rate decay, which point DyReLU has been completely replaced by ReLU, We find that this helps the network settle into a more stable configuration, as DyReLU is no longer needed once the network has explored the parameter space adequately. This process removes all parameters.
The decuy factor for this transition is denoted as B(),, which decays linearly over the course of training:
B(t) = 1 - lstars
fond tstart
where tatart and tend denote the start and end epochs of DyReLU phasing. We choose tend to be the epoch before the first learning rate schedule update in all of our runs. We initialize with:
f(x; t 0) (Dy
During the transition, the activation function progressively shifts toward ReLU:
/ fxx;1) 1) =(()) SDyRaL (4) + +(1 B(()) (a)
5

---

Published in Transactions on Machine Learning Research (11/2025)
Outpul
X Cooooa
boooq yapss
Figure 4: A layer in ResNet-34 with 4 blocks. Block 3 and Block 4 share parameters (and the masks) with block 2. Specifically, convl layer (green) of block 3 reuses convl layer of block 2, and is multiplied by a learnable scaling factor, similarly, its conv2 layer (blue) reuses conv2 layer of block 2, and is multiplied by another scaling factor.
By the end of phasing Tend, we have:
fx;; Tena) (R)
This smooth transition from a dynamic to static activation function prevents the network from overfitting to complex activation pattern while still reaping the benefits of DyReLU the early stages of training.
3.2 Weight Sharing
Much rescarch sparse training has been about mask optimization determining the best subset of param- eters to keep. However, at extreme sparsities, gradient flow can collapse even with an optimal parameter subset. Therefore, ask different question: Can increase the number of parameter paths in the computational graph while maintaining the same number of learnable parometers? Our approach creates multiple references to the same parameter tensors, effectively allowing one learnable parameter to contribute to multiple locations in the borward/backwer
pass without additional storage cost.
Our weight sharing mechanism strategically reuses parameters within each layer's residual blocks, allowing the remaining active parameters to participate in multiple paths of the computational graph. If, certain sparsily, we have n remaining parameters in the original model, we keep exactly this many unique parameters to learn, but some of them now appear in multiple locations in the computational graph, multiplying their contributions without incrcasing storage requirements. During backpropagation, gradients from multiple blocks sharing the same parameters are automatically summed by PyTorrhh' 's autograd, strenghening the learning signal for these weights. Each sharing block includes learnable scaling factors that allow the blocks to specialize while maintaining parameter efficiency.
Specifically, we choose a block within a layer and share its parameters (along with their masks) with all blocks that follow it. These later blocks reference the single block tensor that they share with, and multiply by a learnable scaling factor for forward and backward passes. For example, in ResNet layer with 4 blocks (Figure if the second block shares parameters with block 3 and 4, a parameter in block 2 effectively appears three times in the computation graph while only being stored and optimized once.
Since multiple computational paths now reference the same learnable parameters, we adjust our sparsity calculation reflect the true number of unique learnable parameters rather than the apparent network size. Thus, sparsity = |IMofcarmbbllle where @learnbble represents only the unique learnable paramcter count ||o|o
being optimized. 0 is the theoretical parameter count the original network with a mask M E (0, 1)1 Formally, let be the number of residual blocks in a layer, and R be the block that shares parameters (1 < R S L)), where the remaining I. - R blocks share parameters with the R-th block. Let 0, represent the

---

Published in Transactions on Machine Learning Research (11/2025)
parameters of the i-th block. Our weight sharing scheme can be expressed as:
02 if On if
0.
3.3 Cyclic Sparsity
While classic DST methods such SET and RigL adjust the sparsity pattern, sparsity remains constant throughout training- pruning and regrowing of weights take place at the same rate. Let)M()) (rt)ne((())U Grow(VeL). In our work, we introduce cyclic sparsity schedule where the network's sparsity level itself evolves dynamically during training. Instead of maintaining constant sparsity, we cyclically adjust the sparsity s(t) between maximum value Smax and a minimum value Smin over period Tc. This creates phases of higher and lower connectivity, allowing the network explore parameters more effectively, preventing the gradient issues that plague static extreme sparsity training. Once the eyclic phase ends at Te, fix the sparsity at Smax and switch to standard DST updates. Details of the method are summarized in Algorithm
Algorithm Pseudocode for EAST
Input:
Sparse neural network with weights 0, maximum sparsity 8max; minimum sparsity Smin, end of cyclic sparsity Te, update frequency AT, prune rate Tp. DyReLU phase start T,, end Te
Output: Trained 0,
1: Initialize: 0g at Smax: + DyReLU
2: for t= 1 to Tend do T. St < Te then
3:
DyReLU to ReLU
o 3DyReLU (1- 3) ReLU
Update(3)
end if
if mod AT =- then
8:
if t == To then
9:
Starget + CyclicSchedule((, Smin, Smaxx Tc)
b Update target sparsity
10:
if Starget > Seurrent then
11:
O2 -- 0g ArgTopK( 0z1, (Starget Seurrent) 100 lo) else if Starget Seurrent then
D Magnitude pruning
12:
13:
0₂ t 0, ArgTopK([VoL] (Scurrent Starget 110xllo)
Gradient regrowth
14:
end if
15:
else
Switch to fixed update
16:
0g - 0. - ArgTopK(-100).1 1.-1
17:
0, + 0g + ArgTopK( VeC). (ell)
18:
end if
19: end if
20:
end for
21: return 0,
4 Experiments
Evaluation Protocol. Our experiments include image classification using ResNet-34 and ResNet-50 He ct al. (2016) on the benchmark datasets CIFAR-10 and CIFAR-100, as well as ResNet-50 on ImageNet, following the same evaluation prolocol as other SOTA methods in terms of datasets and backbones. We compare our method with DST methods SET Mocanu et al. (2018) and RigL Evei et al. (2021). and a competitive SST method SynFlow Tanaka et al. (2020b). which also experiments at very high sparsitics. For SynFlow, we use the official repository with its default hyperparameters to evaluate its performance. For SET and RigI,, we use the repository provided by ITOP Liu et al. (202bb). We keep all hyperparameters

---

Published in Transactions on Machine Learning Research (11/2025)
CIFAR-10
CIFAR- 100
Method
99%
99.90%
99.95%
99.99%
99%
99.90%
99.95%
99.90%
RosNet-34
Synllow
86.03220.71
61.61:10.76 76
56.3313311
10.00::0.00
57/4141.72
22,91:188
1.00=0.00
1.00=0:00
SET
93.00++0.15
72-+0-9-0911
70.84+11.11
10.00++0.00
71.97-0.72
47.64+00.45
32. 14+0..0
1.00=0000
Righ.
92. 02.92200.18
05.71440.28
81/17:40.32
10.03#40.19
70.7240.25
50.26+00. 16
38.41#0.95
1.14=0:21
EAST (owrs)
93.51=0.13
N6.99-0.32
83.83-0.02
62.121-0.90
71.14+0.51
56.32 +-0.3
45.81 +0060
18.84+0.555
DCTDS+RiEL
99.0110000
83. 99466.20022
80. 00:8001551
77.38:0.57
63. 03.12200.33
51 11.3000020 26
45.68#0.70
37 37.59#0.552 0.52
DCTpS+RRGT.4ASS (owss
89.72=0.00
84.19-0.64
81.96=0.22
77.64+0.17
64.333+0.44
63.21 0.228
47.80:+0.69
38.02-+0.26
ResNet-50
Synflow
77.061:7.40
48.6211.33
10:00:10.00
10.00:10.00
48.91111448
12:07:4:0.21
9.40.13.02
1.00:0.00
SET
91.99 11.9720.25
27.05415..1
16.01122.85
10:00:00000
70.35:0.11
8.59+2..61
1.26:20 28
1 1.00=0.00
Rigl.
92.7510..25
58.411442000
30.14.110.85
10.00:1..00
70.50.10.32
51.9010062
7.02
1. 1.00
EAST (ours)
93.48=0.13
86.16=0.12
81.18=0.59
19.55::1..9
71.01=0.15
56.76:0.22
45.55+0043
3.61=2.81
DCTDS+RigL
19.667
85.08100.55
85.03100.25
83.1310.53
64.85.10.23
35.9810008
52.71
49.61.1.11
DCTIS+RigL+EAST (ours)
89.67=0.12
87.10=0.35
85.87=0.39
83.88:0019
61.92:1.41
58.17=0.22
55.6010.35
51.33:1.57
Table 1: Accuracy comparison on CIFAR-10 and CIFAR-100 at different sparsity levels. Top section com-
pares sparse training methods at each sparsity level. Boltom section shows results with DCTpS, which achieves higher accuracies at 299.95% sparsity, but requires dense matrix computations. The best results for cach category are in bold.
Inference FLOP (M)
Inference
Network Size
Mothod
99.95%
90.99%
Comp. Cost
Theoretical
CPU-Supported
CPU-Supported Params
ResNet-50 (Dense)
1297.83
((man)
O(N)
O(N)
23.5M
SET
1.24 (0.001 x] 0.26 (0.0002x)
Oppnn)
((PN)
O(N)
23.5M
RigL
1.24 (0.001x)
0.26 (0.0002 x
Opmmn)
O(PN)
O(N)
23.5M
EAST (w/o WS)
1.24 (0.001x)
0.26 (0.0002 x
Oppmn)
O(PN)
O(N)
23.5M
EAST (w/ WS)
2.59 (0.002x)
0.52 (0.0004x) x
Oppmn)
O(PN)
O(N N.)
13.9M
DCTpS RigL
40.26 (0.031x)
39.09 (0.030x)
Olglog prim)
O(PN)
O(N)
23.5M
DCTPS RAST
EAST
40.26 (0.031x) 39.09 (0.030x)
O(g logg prmm)
O(PN)
O(N)
23.5M
Table 2: Comparison of computational complexity and network size of ResNet-50 on different methods. FLOPs are given alongside multiplicative change from the dense model (x). Let N denote the total pa- rameters in ResNet-50, N, the total shared parameters, P E (0, 1) the global density, and m x n the size of flattened weight tensors with density P, where q = max(m, n). "GPU-Supported Params" refers to the actual number of parameters on a commercial GPU without native support for irregular sparse patterns.
they are optimally configured the paper, including AT, ERK initialization, and the mask update interval of 1500 and 4000 for SET and RigI,, respectively.
In addition, benchmark DCT, the SOTA method specifically focused extreme sparsity, using the repository provided by the method in Price & Tanner 20221). While this method performs very competitively at extreme sparsity settings, it comes at cost due to the computation of dense matrix matching the full architecture size, which adds an extra layer of computational overhead and requires extra memory. This method achieves the best performance when paired with RigL, which we evaluate, and then we combine it with our approach to evaluate if our method can improve it further. In this integration, we use only DyReLU phasing and cyclic density, leaving weight sharing to prevent confficts with the DCT matrices embedded in the model architecture. Here, we use AT 100 to match results from the original paper, and we also use this value for integration to ensure fairness.
For all DST methods, we use SGD with momentum as our optimizer. The momentum cocfficient is set to 0.9, and L2 regularization coefficient is set to 0.0001. We set the training epoch to 250 and use a learning rate scheduler that decreases from 0.1 by a factor of 10X at halfway and three-quarters of way through training, For the DCT experiments, we train for 200 epochs with the Adam optimizer and a learning rate of 0.001. A more detailed description of the experimental setups and hyperparameters is included in the Supplementary Material. All experiments in work are conducted on A100 GPU.
8

---

Published in Transactions on Machine Learning Research (11/2025)
CIFAR-10 and CIFAR-100. We evaluate EAST with ResNet-34 and ResNet-50 backbones, with particular focus extreme sparsities (99%-99.99%). We repeat our experiments three times and report the average accuracy in Table ! which presents results across different configurations.
At 99%, all dynamic training methods achieve comparable performance, significantly outperforming Syn- Flow. This unsurprising as a static mask is expected to lead to worse performance than a dynamic mask [Evci et al., 2021 Mostafa & Wang 2019). EAST's advantages become increasingly pronounced at higher sparsitics.
At 99.90% and 99.95%, EAST consistently outperforms existing methods across both architectures and datasets, with particularly notable improvements on ResNet-50 where competing methods experience severe accuracy degradation. While other approaches show dramatic performance collapse beyond 99.90% sparsity, EAST demonstrates more gradual degradation patterns.
At 99.99%, we observe that all other methods result in complete collapse to random performance across both datasets and architectures. In contrast, EAST maintains non-random accuracies. ResNet-50 shows greater sensitivity at this extreme sparsity, likely due to vanishing gradient issues exacerbated by the deeper architecture and reduced parameter density per layer.
Our evaluation of DCTpS reveals complementary strengths. While DCTpS excels at the highest sparsity levels (99.95% and 99.999. 99),,%), classic DST methods including EAST maintain advantages at moderate extreme sparsities (99% and 99.9%) without requiring additional computational overhead. When combined with DCTpS, EAST consistently improves across all tested configurations, demonstrating the modular nature of our approach and its ability to enhance existing extreme sparsity methods.
ImageNet. Prior methods, such as SynFlow and DCTpS, were primarily evaluated on smaller datasets, such as CIFAR and Tiny-ImageNet. We extend these baselines to ImageNet by implementing and adapting their published code and evaluate the scalability of EAST on ImageNet using ResNet-50 at sparsity 99.5% and 99.9%. Given the much shortened training epochs for ImageNet, we implement only the DyReLU phasing and weight. sharing components, excluding cyclic sparsity. As presented in Table 3 EAST has remenstrraed scalability to large-scale datasets compared to existing methods. While SynFlow and SET collapse to random accuracies, EAST maintains stable performance across both tested sparsity levels and outperforms RigL by notable margin.
Interestingly, we observe different dynamics on ImageNet compared to CIFAR experiments. While DCTpS showed competitive advantages on smaller datasots, it does not exhibit the same competitive advantages as seen before, even with the extra overhead. DCTpS alone results in performance collapse and performs worse than RigL. When combined with RigL, the accuracy improvements are modest and may be attributed primarily to RigL's robust dynamic sparse exploration policy, rather than the DCT offset. On the inference efficiency front, EAST demonstrates practical advantages. Unlike DCTPS, which relies matrix, EAST does not have such a drawback, as all DyReLUU activations have been replaced with regular ReLU and thus runs just like the original ResNet model with ReLU. Therefore, EAST should thcoretically have similar inference speed as the other DST methods, but faster than DCTpS for inference. To verify this, we ran both models for inference 100 times, taking their average and finding that DCTpS has 1.5x higher latency and 1.33x lower throughput than ours.
dense DCT
4.1 Complexity Analysis
The operations in neural networks can be framed in terms of matrix multiplication, allowing us to analyze the complexity of these methods. We report the FLOP following RigL's computation method in Table If examine sparse matrix with pmn non-zeros, then theoretically the inference cost of EAST without weight sharing and other standard DST methods is just the cost of the sparse matrix, Oppm)), and their storage requirement is just O(PN). With weight sharing, EAST maintains the theoretical network size O(PN) while physically storing O(N - -N,) parameters, resulting in built-in compression on GPU's without native support for irregular sparsity patterns. It reduces the parameter count including zeros from 23.5M to 13.9M, but slightly increases the theoretical inference FLOP count due to the reuse of learnable parameters. This trade-off potentially offers practical benefits in memory-constrained devices.

---

Published in Transactions on Machine Learning Research (11/2025)
99.50% Spansity
99.90% Sparsity
Method
Top-1 (%)
Top-5 (%)
Top-1 (%)
Top-5 (%)
SynFlow
0.10
0.50
0.10
0.50
SET
0.10
0.50
0.10
0.50
RigL
32,50
58.58
8.53
24.18
DCT
0.10
0.50
0.10
0.50
DCT+RiggL
19.99
43.34
6.62
18.84
EAST (ours)
34.45
62.44
11.23
29.47
Inference Performance
Throughput (imgs/sec)
Latency (ms)
DCT-RigL EAST (ours)
1229.25
10.18
1638.41
6.80
Table 3: Performance comparison ImageNet at extreme sparsity levels using ResNet-50. For throughput and latency measurements, we use batch sizes of 2 and run 100 times and report the average.
Model
Method
99.95% Sparsity 99.99% Sparsity
RigL w/ ReLU
81.47
10.03
ResNet-31
RigL w/ DyReLU
81.80
54.58
RigL w/ WS
81.96
62.94
RigL w/ ReLU
30.14
10.00
ResNet-50
RigL w/ DyReLU
69.57
53.32
RigL w/ WS
82.02
62.71
Table 4: Ablating DyReI.U phasing on DST with evaluation at different sparsities. All seitings are kept the same except the use of DyReLU phasing vs standard ReLU, Experiments were run on CIFAR-10. Results show improved performance with DyReLU Phasing, especially at higher sparsity levels.
While DCTpS achieves higher accuracy than standard DST methods at extreme sparsity levels, it incurs substantial computational overhead, limiting its practical deployment benefits. DCTpS uses a dense DCT matrix, which theoretically could have a complexity of (q1loy)- We found that its theoretical inference FLOPs are about 32X higher than the standard DST at 99.95% and 150X higher at 99.99%. This is because the dense DCT computation dominates at extreme sparsities, becoming an increasingly significant bottleneck as sparsity increases. This additional computational overhead directly explains the inference throughput and latency differences observed in our experiments.
4.2 Ablation Studies
Isolating DyReLU and Weight Sharing. We first augment the standard RigT. pipeline with only DyRcLU phasing, replacing standard ReLU with DyReLU initialization and gradually transitioning back to ReLU before the first. learning rate schedule update. Weight sharing and cyclic sparsity are excluded in this isolated experiment. The results, summarized in Table demonstrate clear performance boost when utilizing DyReLU phasing, particularly at higher sparsity levels. At 99.99%, RigL. completely loses its expressive capacity. However, as soon as DyReLU Phasing is used as drop-in replacement for ReLU, it immediately starts converging. We observe that not only is training improved while DyReLU in effect, but the performance continues to improve even after DyReLU has been completely phased out.
Improving gradient flow during early training has been shown to be critical for successfully training sparse networks. Evei et al. (022)) demonstrated that methods with better gradient flow, partieularly during
10

---

Published in Transactions on Machine Learning Research (11/2025)
ResNet34
ResNet50
3.5
Gradient Norm (Log)
2.5
3-
2.0
2-
s Mww
1.0
0.5
0.0
100
150
250
100
150
200
250
ResNet34
ResNet50
3.5
Gradient Norm (Log)
3.0
2.0
1.5
1.0
1.0
0.5
0.0
0.0
50
100
150
200
250
50
100
150
200
250
Epoch
Epoch
RigL
RigL + DyReLU Phasing
RigL + Weight Sharing
Figure 5: Gradient flow analysis. The top row compares gradient with and without DyReLU. The bottom row compares gradient with and without weight sharing.
initialization and early training, consistently achieved superior performance. Building on this insight, we investigated the effect of DyReLU phasing on gradient flow in addition to measuring accuracy.
To analyze this, we computed the sum of the gradient norm all layers, and compared the gradient flow during training between models using DyReLU phasing against those with standard ReLU, shown in the top row of Figure 5 models using only standard ReLU have zero gradient norms 99.99% sparsity and this is reflected in their collapsed accuracy. On the other hand, the networks with DyReLU phasing receive high gradient flow in the beginning where = during phasing where < B < 1, and maintains healthy gradient flow when 3 = O. Results suggest. that the network retains some of the gradient-boosting benefits initially provided by DyReLU, We conjecture that this is because DyReLU amplifies important features in the beginning and helps the network settle into configuration that preserves gradients even after switching to standard ReLU, Gradient flow especially during early stages is a strong indicator of the DST's success, and helps explain the performance gap between the methods.
Similarly, we isolate the effect of weight sharing by adding only this mechanism to the RigL pipeline, keeping all other settings the same. Once again, gradient flow has been restored, and gradient flow, as shown in the bottom row of Figure This supports our hypothesis that creating additional gradient paths through weight sharing can help maintain stable training at extreme sparsities.
dramatic increase in
Effect Cyclic Sparsity, We evaluate the impact of dynamically varying sparsity levels throughout training compared to maintaining static sparsity. Using ResNet-34 and ResNet-50 on CIFAR-100 99.95% target sparsity, we test different cyclic schedules against static sparsity basclines. As shown in Figure b cyclic sparsity consistently outperforms static sparsity across both architectures to varying degrees. The
11

---

Published in Transactions on Machine Learning Research (11/2025)
Cyclic Schedule
ResNet- CIFAR-100
ResNet:50 CIFAR-100
Validation Accuracy (%6
50-
Validation Accuracy (%)
0.004
Dersity
0.003
20-
0.002
20
0.001
50
100
150
200
250
50
100
150
200
250
0
50
100
150
200
250
Epoch
Epoch
Epoch
Cyclic
Figure 6:
The cyclic sparsity schedule (left) and its effect on training a ResNet-34 (middle) and ResNet-50 (right) on CIFAR-100.
cyclic schodule allows the sparsity to temporarily relax before gradually returning the target extreme level. This periodic increase in connectivity encourages parameter exploration during the denser phases while maintaining the desired final sparsity level. While ResNet-34 shows incremental improvement with cyelic scheduling, ResNet-50 demonstrates more substantial gains, suggesting that deeper networks may benefit more from this cyclic approach.
We provide more test results using different eyclic hyperparameters Appendix C. Based on systematic evalustion across multiple configurations (Supplementary Tables C.2-C.5), we observe several trends. 10x density multipliers generally outperform 3x, but 30x not necessarily better than 10x, suggesting there may be an elbow point where further increases no longer improves performance, Having two cycles generally performs better than having one, with longer subsequent cycle providing additional gains. While optimal configurations may vary by architecture and dataset, recommend starting with 10x multiplier and a length multiplier of 2x if two cycles are used when tuning cyelic sparsity hyperparameters.
5 Conclusion
In this paper, we introduced EAST, new DST approach that focuses on extreme sparsity and incorporates DyReLU phasing, weight sharing, and cyclic sparsity scheduling. Through extensive empirical evaluation CIFAR-10, CIFAR-10, and ImageNet using ResNet-34 and ResNet-50, demonstrated that EAST achieves competitive performance at extreme sparsitys, surpassing existing methods by large margin. Fach component of EAST individually contributes to maintaining robust gradient flow and optimizing parameter exploration at extreme sparsity, while their combined effect can further enhance performance stability. Our results highlight the potential for practical applicability in cesstrce-onntrrained environments, establishing as a scalable and effective method for highly compressed networks. This work opens up future rescarch optimize models at extreme sparsities, an area often overlooked in previous studies.
Acknowledgments: Our work was supported by the UKRI EPSRC funding council (EP /D/042220///1 University of Aberdeen PhD studentship, the HPC facility "Maxwell", and the Interdisciplinary Institute.
References
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic relu. In European conference on computer vision, pp. 351-367. Springer, 2020.
Pau De Jorge, Amartya Sanyal, Harkirat S Behl, Philip HS Torr, Gregory Rogez, and Puncct K Doka- nia. Progressive skeletonization: Trimming more fat from a network at initialization. arXiv preprint arXiv:2006.09081, 2020.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing perfor- mance, 2019. URL. httttp:://arxxi..rrr/bbbs
048800.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners, 2021. URL htttp:://aaxx..ooo///a/////////p
Utku Evei, Yani Toannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 6577-6586, 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. aiviv preprinl Xiv: 1803. 803. 03635, 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. PMLR, 2020a.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.085 2020b.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXie preprint urXiv:1510.00149, 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In Advances in neural information processing systems, pp. 1145-1143, 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance imagenet classification. In Proceedings of the IEEE international conference computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Siddhant Jayakumar, Razvan Pascamu, Jack Rae, Simon Osindero, and Erich Elsen. Tap-kast: Top-k always 2020.
sparse training. Advances in Neural Information Processing Systems,
Jie Ji, Gen Li, Lu Yin, Minghai Qin, Geng Yuan, Linke Guo, Shiwei Liu, and Xiaolong Ma. Advancing dynamic sparse training by exploring optimization opportunities. In Forty-first International Conference on Machine Learning, 2024.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning, pp. 5544-5555. PMLR, 2020.
Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, and Yani Toannou. Dynamic sparse training with structured sparsity. In International Conference on Learning Representations (ICLR), 2024. URL https:
//openreview.net// http://wwww...toommmeeeooooooooa
Yann LeCun, John Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598 598-605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection scnsitivity. In International Conference on Learning Representations, 2019.
Gen Li, Lu Yin, Jie Ji, Wei Niu, Minghai Qin, Bin Ren, Linke Guo, Shiwei Liu, and Xiaolong Ma. Neurrev: Train better sparse neural network practically via neuron revitalization. In The Twelfth International Conference on Learning Representations, 2024.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference Learning Representations, 2016.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pech- enizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908 9922, 2021a.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy, Do actually need dense ? in-time var-parametteriztion in sparse training. In International Conference on Machine Learning, pp. 6989 7000. PMLR, 2021b.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang, Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736 2744, 2017.
Eran Malach, Gilad Yehudai, Shai Shalev-Schwarta. and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning, pp. 6682-660) PMTRR 2020.
Diganta Misra. Mish: A self regularized non-monotonic activation function. Xi preprint arXiv: 08681, 2019.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and An- tonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9():23383, 2018.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp. 4646 4655. PMLR, 2019.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICMI-10), pp. 807-814, 2010.
Maan Price and Jared Tanner. Dense for the price of sparse: Improved performance of sparsely initialized networks via subspace offset. In International Conference on Machine Learning, pp. 8620 8029. PMLR, 2021.
Prajit. Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. orXiv preprint (t4i::1710.09441.. 2017.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6377-6389, Cur- Associates, Inc., 2020a. URL 66624777888336330000777766666633
http://wwcedddn....eerr........e .dff
Hidenori Tanaka, Daniel Kunin, Daniel Yamins, and Surya Ganguli. Pruning neural networks without any data, by iteratively conserving synaptic flow. Advances in neural information processing systems, 33: 6377-6389, 2020b.
Stijn Verdenius, Maarten Stol, and Patrick Forré. Pruming via iterative ranking of scnsitivity statistics. arXiv preprinl ar arXiv:2006.000 00896, 2020.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:200..0797% 2020.
Boqian Wu, Qiao Xiao, Shunxin Wang, Nicola Strisciuglio, Mykola Pechenizkiy, Maurice van Kenlen, Dece- bal Constantin Mocanu, and Flena Mocanu. Dynamic sparse training versus dense training: The unex- pected winner in image corruption robustness. In The Thirteenth International Conference on Learning Representations (ICLR 2025), Singapore, 2025. Published a conference paper at ICLR 2025.
Lu Yin, Vlado Menkovski, Meng Fang, Tianjin Huang, Yulong Pei, and Mykola Pechenizkiy. Superposing many tickets into one: A performance booster for sparse neural network training. In Uncertainty in Artificial Intelligence, pp. 2267-2277 PMI.R, 2022.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pecheniskiy, Shiwei Liu, et al. Dynamic sparsity is channel-level sparsity learner. Advances in Newral Information Processing Systems, 36, 2024.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang Hc, Qing Jin, et al. Most: Accurate and fast memory-cconomic sparse training framework the edge. Advonces in Newral Information Processing Systems, 34:20838 20850, 2021.
Yingtao Zhang, Diego Cerretti, Jialin Zhao, Wenjing Wu, Ziheng Liao, Umberto Michicli, and Carlo Vittorio Cannistraci. Brain network science modelling of sparse neural networks enables transformers and 1lms to perform fully connected. Xiv preprint arX:::201.11107, 2025.
15


Reviews:
[{"review_id": "1tXOHvCwMc", "reviewer": "Reviewer_QHqX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. **Good integration of complementary mechanisms**: The paper shows how three orthogonal strategies - dynamic ReLU, structural parameter reuse, and sparsity scheduling - can be effectively combined, each addressing distinct failure modes in extreme sparse regimes.\n2. **Consistent improvements over baselines methods**: Experiments across multiple architectures (ResNet-34/50) and datasets (CIFAR-10/100, ImageNet) with multiple baselines (SynFlow, SET, RigL, DCTpS) show consistent improvements, particularly beyond 99.9% sparsity.\n3. **Extensive ablations**: Each component of EAST is analyzed in isolation and in combination. The supplementary material includes detailed hyperparameters and cyclic sparsity sensitivity analysis, increasing reproducibility.\n4. **Practical efficiency**: Unlike DCTpS, EAST avoids dense-matrix computations during inference. FLOP and latency comparisons highlight usefulness for real-world deployability.\n\n# Weaknesses\n1. **Lack of theoretical grounding**: The paper is almost entirely empirical. The mechanisms (especially DyReLU phasing and cyclic sparsity) are motivated heuristically, without formal analysis of why or when they should succeed or fail.\n2. **Limited exploration of general architectures**: All experiments are on image classification with ResNets. It is unclear if EAST generalizes to **transformers** or NLP models, where sparse training dynamics differ. This would be very useful to have.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a collection of complementary techniques designed to stabilize and improve **dynamic sparse training (DST)** at extreme sparsity levels (e.g. >99.9%), where prior methods collapse. The authors present **EAST (Extreme Adaptive Sparse Training)**, a modular framework combining three main innovations:\n1. **Dynamic ReLU (DyReLU) Phasing** - replacing standard ReLU with Dynamic ReLU early in training to enhance gradient flow and parameter exploration, then linearly phasing it out to standard ReLU before the first learning-rate decay. This preserves gradient pathways while removing DyReLU parameters for inference.\n\n2. **Weight Sharing (WS)** - intra-layer parameter reuse across residual blocks in ResNet architectures, multiplying each learnable parameter’s participation in forward and backward paths while maintaining identical learnable parameter count. This improves gradient propagation and utilization without increasing model size.\n\n3. **Cyclic Sparsity Scheduling** - varying the network’s sparsity level cyclically between minimum and maximum densities during training, alternating between dense and sparse phases. This encourages parameter exploration and prevents layer collapse before converging to the final target sparsity.\n\nThey integrate these techniques within standard DST frameworks (e.g., **RigL, SET**) and achieves consistent accuracy improvements at extreme sparsities on CIFAR-10/100 and ImageNet, using ResNet-34/50 backbones. The framework also complements **DCTpS** by improving its accuracy without additional inference cost. Ablation studies demonstrate the benefits of DyReLU, WS, and cyclic sparsity on gradient flow, measured through gradient norm analysis and positive pre-activation statistics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the including following items would make the paper stronger:\n1. Provide stronger theoretical or empirical justification for different components of the proposed method -- e.g. for why DyReLU phasing preserves gradients post-transition. In this case, a controlled analysis (e.g., gradient flow visualization or distributional change before and after phasing) would substantiate this mechanism.\n2. Some clarifications about weight-sharing -- how is the accumulation of gradients on shared tensors done, i.e. are the gradients are averaged, summed, or rescaled?\n3. **IMPORTANT** -- I would really like to see some experiments on non-vision or non-ResNet architectures.", "broader_impact_concerns": "There are no explicit ethical concerns beyond those common to general-purpose model-compression research.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4CSYcvaYR2", "reviewer": "Reviewer_QjVn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The paper tackles the challenging problem of maintaining network performance at extreme sparsity levels (99.90%-99.99%), which represents a significant technical challenge even if the practical motivation is unclear. The proposed EAST framework demonstrates that non-random accuracy can be maintained at these extreme sparsities where existing methods completely collapse, showing clear improvements over baselines like RigL and SET.\n\n2. The gradient flow analysis in Figure 5 provides valuable insights into why DyReLU phasing and weight sharing improve training stability. The systematic ablation studies isolate the contribution of each component, demonstrating that DyReLU phasing alone can restore gradient flow and prevent collapse at 99.99% sparsity. The experimental methodology is generally sound, with multiple runs and error bars reported.\n\n3. The modular nature of the approach is noteworthy, as demonstrated by successfully integrating EAST components with DCTpS to achieve consistent improvements. This suggests the techniques could potentially be adapted to other sparse training frameworks.\n\n## Weaknesses\n\n1. The related work section is severely outdated, relying almost entirely on papers from 2018-2021 with virtually no citations from 2023-2024. Given the rapid pace of development in sparse training, pruning, and efficient neural networks, this is a critical flaw. It is hard to address the impact of the work within the current state of the field when that \"current state\" is represented by 3-4 year old papers. Recent advances in hardware-aware sparsity, modern dynamic sparse training methods, and contemporary activation function research are completely absent.\n\n2. The central motivation for pursuing extreme sparsity lacks any supporting evidence. The authors claim there is \"increasing demand\" to push toward extreme sparsities in \"resource-constrained settings,\" yet provide no citations, no real-world applications, no deployment scenarios, and no evidence that accuracy trade-offs of 30+ percentage points would ever be acceptable. At 99.99% sparsity, EAST achieves 62% on CIFAR-10 compared to 93% at 99% sparsity—a catastrophic accuracy loss for marginal additional compression. The paper reads as solving a problem that may not meaningfully exist beyond academic curiosity.\n\n3. The central motivation for pursuing extreme sparsity lacks any supporting evidence. The authors claim there is \"increasing demand\" to push toward extreme sparsities in \"resource-constrained settings,\" yet provide no citations, no real-world applications, and no deployment scenarios; additionally,  no evidence that accuracy trade-offs would ever be acceptable in a real-world scenario. At 99.99% sparsity, EAST achieves 62% on CIFAR-10 compared to 93% at 99% sparsity, which looks like a catastrophic accuracy loss for marginal additional compression. \n\n4. The complete absence of training cost analysis is inexcusable. The paper introduces multiple complex mechanisms: DyReLU adds computational overhead during phasing, cyclic sparsity requires dynamic mask updates, and weight sharing introduces gradient accumulation complexity. Yet there is no comparison of training time, memory usage, or training FLOPs versus baselines. The authors dismiss this by stating their \"primary focus is on the deployable model,\" but this omission fundamentally limits the paper's contribution.\n\n5. The experimental scope is far too limited. Testing only ResNet-34 and ResNet-50 ignores the reality that Vision Transformers have become standard in computer vision, and modern CNN architectures like EfficientNet and ConvNeXt are widely used. The reliance on CIFAR-10/100 (32×32 toy datasets) for primary results is insufficient by modern standards. While ImageNet is included, the results are limited and reveal concerning inconsistencies, contradicting the paper's narrative. No evaluation on more difficult datasets, or any domain-specific datasets where extreme compression might actually matter, undermines claims of practical relevance.\n\n6. The paper acknowledges that cyclic sparsity has a \"large number of possible configurations makes systematic experimentation and hyperparameter tuning challenging,\" but provides minimal guidance or a principled approach to setting these hyperparameters. The weight sharing mechanism increases inference FLOPs while reducing parameters, yet there is no analysis of actual memory savings on real hardware or discussion of how shared parameters affect convergence properties. Table 2 shows that \"GPU-Supported Params\" remains O(N) for most methods, suggesting the practical deployment advantage is limited even at extreme sparsities where commercial accelerators typically support 90-98% sparsity efficiently anyway.\n\n7. The paper lacks comparison with alternative compression strategies, including structured pruning, quantization (which often achieves better compression-accuracy trade-offs), or knowledge distillation to smaller dense models. Without these comparisons, it is impossible to assess whether extreme unstructured sparsity is even the right approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes EAST (Extreme Adaptive Sparse Training), combining three techniques—DyReLU phasing, weight sharing, and cyclic sparsity—to train neural networks at extreme sparsity levels (99.90%-99.99%). While the work addresses an interesting problem, the paper suffers from significant methodological limitations, outdated literature review, insufficient experimental scope, and missing practical considerations that undermine its contributions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Update the related work section to include recent literature from 2023-2024, covering advances in sparse training, modern pruning techniques, hardware-aware sparsity methods, and recent activation function research. The current literature review is unacceptably outdated for a submission to a 2025 venue.\n\n2. Provide concrete evidence and citations supporting the claim of \"increasing demand\" for extreme sparsity. \n\n3. Include comprehensive training cost analysis comparing EAST against all baselines in terms of wall-clock training time, memory usage, and training FLOPs. \n\n4. Expand experimental evaluation to include modern architectures, particularly Vision Transformers (ViT, DeiT, Swin) and recent CNNs (EfficientNet, ConvNeXt). Evaluate on larger-scale and more diverse datasets beyond CIFAR.  The current scope (ResNets on CIFAR) is too narrow to support generalizability claims.\n\n5. Provide principled guidance for setting cyclic sparsity hyperparameters rather than acknowledging the difficulty and leaving it to future work\n\n6. Compare EAST against alternative compression strategies including structured pruning, quantization, and knowledge distillation to smaller dense models.\n\n7. Analyze actual memory footprint and inference characteristics on real hardware (not just theoretical FLOPs) to substantiate deployment claims.", "broader_impact_concerns": "No particular concers", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7U9ThsV77Q", "reviewer": "Reviewer_DADF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Authors propose an extensive empirical section that shows their approach is competitive with previous work.\n- Authors validate the effect of weight-sharing/DyReLU on potential gradient flows\n\n\nWeaknesses:\n- Clarity of the paper could be improved: \n1) some crucial descriptions regarding contribution are DyReLU in appendix A. This content should probably be moved in the main paper as it's one of the main paper contributions. \n2) Even after reading appendix A, some details regarding DyReLU are still not clear. How do you define the 'hyper-function' to compute a^c_k(x) and b^c_k(x)? 3) I am not sure I understand line 3-5 in algorithm 1? Should it be an if-statement instead of a for line 3 and what is \\phi line 4?\n\n- It's not clear what is the impact of each of the three proposed components. For instance DCTpS+EAST does not use weight-sharing yet achieves the best performance at sparsity level, which raises the question of importance of weight-sharing. On the other hand, ImageNet experiment does not use cyclic sparsity. Overall, it would be nice to ablate the effect of the different components (DyReLU, cyclic sparsity, weight-sharing) in the different setup and for different sparsity level.\n\n- Authors use previous hyperparameters for the different baseline but it is not clear that those hyperparameters are optimal in their setups (i.e. high-sparsity ratio). Is the empirical setup similar to those of previous work? If not, what is the effect of hyperparameter tuning of the baseline?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates training sparse neural networks with extreme sparsity ratios, including 99.90%, 99.95%, and 99.99%. The paper introduces and explores three components to train sparse networks: 1) a new non-linearity called DyReLU that aims to preserve gradient flow, 2) a cyclic sparsity-level schedule where the amount of sparsity evolves during training, and 3) a weight-sharing component that reuses the same parameters across different layers.\n\nThe authors empirically explore various variants on the CIFAR-10, CIFAR-100, and ImageNet datasets, showing that their approach outperforms previous baselines such as RigL and SET in the high-sparsity regime (> 99.9%).\n\nThe authors also provide empirical ablations showing that 1) DyReLU and weight sharing improve gradient flow in the high-sparsity regime, and 2) the cyclic sparsity schedule outperforms a fixed sparsity schedule.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition to improving the manuscript's clarity (see weaknesses), I believe it is important to add an ablation study demonstrating the effect of each contribution (weight sharing, DyReLU, cyclic sparsity) on accuracy after pruning, across different datasets and sparsity ratios. Additionally, ensuring that the baselines use properly tuned hyperparameters for the high-sparsity regime would help validate the correctness of the empirical setup.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)